We evaluated the complexity of the web-based ACP program “Explore Your Preferences for Treatment and Care” [2] using the NASSS framework, following the approach of Abimbola et al [11] who also conducted an ex post (retrospective) thematic analysis.

First, we developed a multimodal dataset consisting of different existing, relevant data sources (Table 1). The dataset includes peer-reviewed and published studies describing the development and evaluation of the program [4,6,7], notes of stakeholder group meetings, and a social return on investment analysis in which the social costs (input) and benefits for stakeholders (outcomes) of the program were mapped.

Next, 2 researchers (DvdS an MAS) developed a data extraction form based on the nonadoption, abandonment, scale-up, spread, and sustainability complexity assessment tool (NASSS-CAT; IRIHS group); the NASSS CAT-LONG interview guide [13]. They used this form to extract relevant pieces of text from the data sources per NASSS domain, that is, (1) condition, (2) technology, (3) value proposition, (4) adopters, (5; previously 6) external context, (6; previously 7) embedding and adaptation over time. We considered the original domain 5, “organization,” not applicable because the web-based ACP program is not implemented in a specific health care organization and the hosting organization, the Dutch College of General Practitioners, did not have to change their routines to embed the program. Next, DvdS and MAS together with the wider research group conducted an initial thematic analysis of the extracted information to identify whether sufficient information per domain was available. They conducted 3 additional semistructured interviews to complete the dataset (1) with a representative of the Dutch college of general practitioners (the program host), (2) a general practitioner, and (3) a representative of the organization that funded the development of the web-based ACP program. The interview questions were based on the NASSS CAT-LONG interview guide [13]. Interviewees provided written informed consent for participating in an online interview in which questions were asked about attitudes toward the web-based ACP program related to the domains of the NASSS framework. Interviews were audio-recorded and transcribed verbatim.

Subsequently, DvdS and MAS together with the wider research group conducted a thematic analysis of the final dataset, using the NASSS framework as a lens, leading to a thematic description of complexities per domain. We used relevant ACP literature to further underpin the findings. The conclusion whether a domain is simple, complicated, or complex was based on discussion and eventually consensus among the members of the research group, where we weighed the potential impact and relevance of the different complexities.

Complexities in each of the 6 domains were classified as “simple,” “complicated,” or “complex.” Per domain, all relevant aspects were taken into consideration. An overall classification as “simple” does not require a total absence of room for improvement for a domain; however, most of its elements should be considered as simple. Based on the multimodal dataset and the classification in Multimedia Appendix 1 [8,9], DvdS and MAS prepared a classification per domain, which was discussed during meetings with all researchers (MAS, DvdS, JACR, IJK, and AvdH), until consensus was reached.

This study was approved by the Medical Research Ethics Committee of the Erasmus MC, University Medical Center Rotterdam on October 21, 2019 (MEC-2019-0590), confirming that the rules laid down in the Medical Research Involving Human Subjects Act do not apply to the interviews as conducted in this study. The study conforms with the International Committee of Medical Journal Editors’ recommendations for the conduct, reporting, editing, and publication and for the protection of research participants. The interview participants were provided with information about the aim and content of the interview. They provided written informed consent. Interview participants did not receive compensation. The authors confirm that all patient or personal identifiers have been removed or disguised so the persons described are not identifiable and cannot be identified through the details of the story.

The domain “condition or illness” considers to what extent the condition or illness is well-characterized, well-understood, and predictable and to what extent the condition or illness is influenced by comorbidities or sociocultural factors.

The domain “the technology” encompasses a description of the web-based ACP program and the Thuisarts website, as well as user experiences with the technology.

The “value proposition” domain concerns whether a new technology is worth developing and for whom it may generate value. It includes demand-side value (value for the users) as well as supply-side value (value for the developers).

The domain “adopters” focuses on the adoption and continued use of the technology by the groups of people who are intended to actually use the program or to refer potential users to the program, and what they need to use the program or to refer to it. The potential adopters of the web-based ACP program include patients, relatives, health care professionals, and the public in general.

The “external context” domain encompasses (1) the policy and political climate, (2) professional organizations and patient organizations, (3) the regulatory context, and (4) the economic context.

To analyze the implementation of the web-based ACP program “Explore Your Preferences for Treatment and Care,” the NASSS framework was used to identify complexities within 6 domains. The analysis revealed no or little complexity in the domains “technology,” “value proposition,” “external context,” and “embedding and adaptation over time.” The program is evidence-based, freestanding, and well-maintained, with straightforward, well-understood technology. The program is expected to generate a positive value for different stakeholders as well as for the Thuisarts website. The analyses revealed some complexities in the “condition” domain, including the broad and rather undemarcated target population of ACP and the web-based ACP program and sociocultural factors that may limit users’ engagement. In the “adopters” domain, complexity entails that the program requires substantial commitment and effort by users with respect to reflecting on their preferences for future treatment and care. Some people with limited digital literacy may need support to use the program. Furthermore, it is yet unstudied to what extent health care professionals adopt the web-based ACP program. The program is embedded in the general practitioners’ platform “Thuisarts,” which is frequently used by general practitioners and the general public. The fact that the web-based ACP program is not embedded in a specific health care organization simplified its implementation, since no alignment regarding content or layout was required. On the other hand, embedding the program in the routine practice of a health care organization, could have boosted its use. Finally, we found some complexity in the “external context” domain. While the political and policy climate, regulatory context, professional organizations, and patient organizations are generally supportive regarding ACP, a national guideline for ACP and a treatment code to reimburse ACP could further increase willingness and ability of health care providers to engage in ACP. Overall, the analysis showed that the program has good potential for sustainable implementation, as it is expected to be continuously used in practice [8,38,39], will be updated if necessary, and will be maintained long-term.

The results of this study showed that the web-based ACP program “Explore Your Preferences for Treatment and Care” has relatively few complexities regarding the implementation. This may have several reasons. First, the user-centered design of the program included an extensive preparatory phase, including a scoping review to learn from existing web-based ACP programs [6], identification of the needs of patients with chronic disease and their relatives for web-based ACP in an interview study [7], and close collaboration with stakeholders including patients and relatives. Second, during the development of the program, we paid due attention to sustainable implementation in an existing, well-used platform—“Thuisarts.” The team of the Thuisarts website has expertise in developing information for the public with different levels of health literacy. It is remarkable that other web-based ACP programs often seem to have not paid due attention to implementation. Three recent reviews about web-based ACP programs [6,40,41] showed that studies have mainly focused on the development of web-based ACP programs and evaluation of their feasibility, usability, acceptability, and effectiveness, but most did not address their implementation. This is in line with reviews about implementing eHealth in general showing that research has mainly focused on the content and evaluation of eHealth tools, but less so took into account the external context and sustainable adoption in health care systems [8,10].

The results of the current study also indicated that health care professionals are an important group of adopters for the web-based ACP program. The web-based ACP program, following existing guidelines, recommends an active role of health care professionals in the ACP process of patients, for instance, to discuss diagnosis, prognosis, and relevant treatment and care options with the patient, and to document the ACP process in the medical file [3,4]. In the literature, several barriers are described that health care professionals may experience in the initiation of ACP, such as lack of time, lack of skills in conducting ACP conversations, and fear of taking away patients’ hope [25,27]. In the development of the web-based ACP program, the role of health care professionals was to a limited extent taken into account. First, the program is embedded in the general practitioners’ platform “Thuisarts,” which is frequently used by general practitioners [26] and the general public [17,18]. Furthermore, health care professionals were included in the stakeholder group and provided input during the development of the web-based ACP program. In the program, the user is encouraged to discuss and share their preferences with their health care professionals at several points [2]. However, the actual use of the web-based ACP program among health care professionals and their views and experiences regarding the program, are unstudied. Given the barriers that health care professionals may experience in the initiation of ACP [25,27], a patient-centered and community approach of ACP is important, in which patients can also take the initiative to engage in ACP themselves. This is central to the program, as the program supports the initiation of ACP, self-management of users, and encourages conversation with health care professionals about values, goals, and preferences.

The results indicated complexities considering the broad target population of ACP in the web-based ACP program. This complexity is a reflection of the evolving concept of ACP in the literature. Initially, ACP was conceptualized to be used for patients in the palliative phase and at the end of life [3,14,15]. More recently, an international consensus study on ACP recommended that individuals can engage in ACP in any stage of life, but that ACP can be more targeted when individuals’ health condition worsens or when their age increases [3]. This evolving concept of ACP aligns with the evolving concept of palliative care, whose definition has widened from a sole focus on the end of life to also including chronic illnesses [42] and a shift from in-hospital to community-based care [43]. Reaching the community with high-quality information and guidance about ACP through the Thuisarts website may therefore fit these developments.

A strength of this study is our systematic approach of evaluating the web-based ACP program using the NASSS framework. It provided insight in its complexities that might hamper its implementation. Furthermore, we had a large dataset including publications about the development and evaluation of the web-based ACP program, notes of stakeholder meetings, and interviews with the program host, a general practitioner, and a funding agency representative. Insight into the complexities may be used to improve future sustainable implementation of the program, and may be used to improve other web-based programs as well.

A limitation is the start of the NASSS evaluation after the tool was developed and embedded in the Thuisarts website, which has limited the opportunity to adapt the program during its development. Some of the authors (DvdS, JACR, AvdH, and IJK) were the developers of the web-based ACP program, and therefore they could provide in-depth insight in the developmental process. However, although the authors aimed to analyze the implementation of the web-based ACP program objectively, there can be a bias in the evaluation.

To enhance sustainable implementation, we provide several recommendations to improve the identified complexities of the web-based ACP program.

First, to make the program available in other languages than Dutch. This may further enhance the accessibility of the program to people with different languages. For instance, an English version of the program could be placed on the GPinfo website, which is the English version of the Thuisarts website that is still in development. The web-based ACP program may be translated to other languages as well, to enhance its accessibility to people who are not proficient in Dutch. To improve cultural inclusiveness, future research should explore the needs of people from various cultural backgrounds with regard to ACP and if desired, the program could be culturally adapted or different versions of the program could be developed.

Second, to enhance accessibility of the web-based ACP program to persons with very limited health or digital literacy. The program was developed to assist individuals with limited health or digital literacy, by offering readable texts, text-to-speech options, and videos. However, individuals with very limited health or digital literacy may still experience barriers to use the program. To enhance accessibility of the program for these users, recommendations to complete the program together with a relative could be added to the program. Furthermore, a PDF version of the web-based ACP program could be developed, that could be printed by a relative or health care professional, to make the program also available “offline.”

Third, to inform health care professionals about the variance in readiness for ACP among patients. Not everyone may want to engage in ACP [44], for instance, due to different cultural backgrounds and different needs to consider treatment and care preferences. In addition, we recommend to inform and support health care professionals in how to conduct ACP with their patients. The web-based ACP program may help to overcome lack of time, since health care professionals can refer to the program, patients can prepare themselves for an ACP discussion on beforehand, and then the discussion with the health care professional can be scheduled.

Fourth, the concept of ACP as an ongoing process of considering preferences (also including healthy individuals) [3], could be embedded in national guidelines, and be routinely integrated in health care practice to reach a broader target group for ACP. National guidelines and availability of a treatment code may lower barriers in ACP for health care professionals, and may consequently lead to increased engagement with ACP for patients. Websites aimed at the general public could refer to the web-based ACP program, for instance, pages about shared decision-making and pages within the Thuisarts website about chronic diseases. Furthermore, we recommend to conduct public awareness campaigns about ACP (including web-based ACP programs), as these have been found to enhance awareness of ACP [45]. These may also help to enhance accessibility of ACP for people beyond the medical health care setting.

Finally, to ensure adaptation of the program to evolving data protection requirements. During critical review of the program, that will take place every 3 years by a stakeholder group of patients, relatives, health care professionals, and patient organizations, compliance with (evolving) data security regulations, privacy requirements, and policies should be carefully reviewed. If necessary, the program should be adapted. In addition, the strategy of the Thuisarts website, the host of the web-based ACP program, is aimed at adapting the information on the Thuisarts website to the evidence-based guidelines for general practitioners as developed by the Dutch College of General Practitioners. When guidelines change, the information at the Thuisarts website is adapted accordingly by their editorial team. Furthermore, the strategy of the Thuisarts website focuses on collaborating with health care professionals and policy makers, and on including patients’ perceptions in the information, to ensure the content is adapted to significant changes in health care landscapes or policies.

Since the domains of the NASSS framework may be interdependent, reducing complexity in a particular domain may consequently also reduce the complexity in other domains. For instance, in case of national guidelines on ACP, health care professionals may be more willing to initiate ACP with their patients and patients may be more ready to initiate ACP.

The following recommendations, based on the results and conclusions of the current study, can aid future implementation of comparable web-based ACP programs.

In order to improve the sustainable implementation of eHealth tools, in particular web-based ACP programs, first, we recommend to systematically and throughout the process (from idea to implementation) assess their complexities, with careful attention for all potential adopters, their external context, and their embedment and adaptation over time. The NASSS framework is suitable to conduct such evaluation, as well as other models that can be used for implementation or sustainability evaluation, such as the Normalization Process Theory (a framework to understand how interventions become embedded or “normalized” in health care settings) [46] and the Centre for eHealth Research Roadmap (CeHRes) [47,48]. These frameworks consider similar domains as the NASSS framework, sometimes ordered slightly differently, and can provide guidance for the sustainable development and implementation of eHealth technologies [46-48]. We expect that others who aim to implement similar web-based ACP programs and who apply comparable implementation frameworks, will benefit from the findings of this study and the recommendations. Yet, these should be contextualized for each application (eg, aligned with local policy frameworks and targeted to the target groups central in that program).

Second, we recommend identifying all relevant target groups of the eHealth tools, and explore their needs and preferences to ensure the tool meets their needs. For the web-based ACP program in particular, we recommend exploring health care professionals’ perspectives as well as their actual use and awareness of the program.

Third, we recommend assessing as well as improving the attractiveness of eHealth tools to potential users, including a culturally diverse population and to groups with low digital literacy.

Fourth, we recommend that research includes a long-term perspective regarding the implementation of eHealth tools. This may include evaluating the long-term effects as well as the identification of new scientific evidence that necessitates an update of the content of eHealth tools.

Relatively few complexities were identified considering the implementation of the web-based ACP program “Explore Your Preferences for Treatment and Care.” The program is evidence-based, freestanding, and well-maintained, with straightforward, well-understood technology. The program is expected to generate a positive value for different stakeholders. Complexities include the broad target population of the program and sociocultural factors. People with limited digital literacy may need support to use the program. Its uptake might be improved by increasing awareness of ACP and the program among a wider population of potential users and among health care professionals. Addressing these issues may guide future use and sustainability of the program.