Between April 2022 and May 2023, we engaged in an extensive process involving literature reviews, examining existing ACP decision aids, and consultations with a study team comprising diverse experts including health services researchers, cardiologists, palliative care physicians, social workers, communication coaches, and information technology professionals. We also established a study advisory board consisting of patient representatives and HCPs trained to conduct ACP conversations and gathered their inputs regarding content and structure for the interactive web-based application “My Voice.” Guided by the COM-B model that focuses on capability (C), opportunity (O), motivation (M) to enhance behaviors (B) [36], “My Voice” aimed to improve patients’ capability for engaging in ACP conversations, creating opportunities for them to have these conversations and revisit them periodically, and motivating them to do so.

A professional production house produced narration-style videos featuring HCPs from this study’s team. The website content and videos were initially developed in English and subsequently professionally translated into two local languages (Mandarin and Malay) to ensure inclusivity and accessibility. Upon completion of the content development phase, we developed the initial prototype of “My Voice.” To ensure the security of participants’ identifiable information, we incorporated password protection and a two-factor authorization process. The research team obtained relevant institutional approvals at all stages of web application development.

“My Voice” includes patient and caregiver modules. The patient module consists of a series of educational videos lasting 1-2 minutes each, organized into five steps: (1) learn about heart failure, (2) think about what is important to you, (3) why and how to choose a spokesperson, (4) speak to your doctor about what is important to you, and (5) revisit “My Voice.”

In addition to the videos, step 1 includes a knowledge quiz. Step 2 incorporates questions to elicit patient values and goals (referred to as value clarification questions), which were based on the Serious Illness Conversation Guide [37]. Step 3 provides fields for nominating up to two surrogate decision makers, known locally as the nominated health care spokespersons. Upon completion of step 5, the patients’ responses to the value clarification questions and the details of their spokespersons are automatically populated into a summary document called the “My Voice” document. Patients can view and edit the document before it is saved on the website. The website then triggers automated emails and phone text messages containing the document to both patients and their designated surrogate decision makers. Patients and their designated surrogate decision makers can also view and print the document anytime through the website. The website also sends phone reminders to patients to revisit “My Voice.” Multimedia Appendix 1 highlights select pages from the “My Voice” website.

The caregiver module includes educational videos lasting 1-2 minutes structured into three steps: (1) learn about heart failure, (2) talk to your loved one, and (3) support your loved one. As with the patient module, step 1 incorporates a knowledge quiz. At the end of the module, caregivers have the option to view the patient’s “My Voice” document.

Between May and August 2023, we recruited a convenience sample of patients with heart failure, their caregivers, and HCPs, from four public hospitals in Singapore. These groups of participants are the key stakeholders in ACP conversations, and hence best positioned to provide feedback on the website. We recruited patients from outpatient clinics and wards, based on the following inclusion criteria: (1) adults aged 21 years or older; (2) diagnosed with heart failure; (3) Singaporean or permanent resident; (4) able to understand either English, Mandarin, or Malay; and (5) willing to use a web-based intervention. Caregivers of the above eligible patients were approached independently and included if they consented. Caregivers were: (1) adults aged 21 years or older; (2) providing informal care or ensuring the provision of care, or serving as the main decision maker for the patients with heart failure with no expectation of financial compensation; (3) able to understand either English, Mandarin or Malay; and (4) willing to engage with a web-based intervention. HCPs from department of cardiology namely cardiologists, cardiology nurses, and medical social workers trained in having ACP conversations were included.

We used a mixed methods design incorporating both qualitative interviews that provided feedback to iteratively revise the website’s content and design to improve the overall user experience, and quantitative surveys to estimate the website functionalities including its usability and acceptability. The results from qualitative and quantitative sections were integrated during interpretation as a narrative discussion.

Trained research staff (IC and SNK) conducted usability testing on a tablet device provided by the research team. Patients and caregivers viewed their respective modules while HCPs viewed both modules. On viewing the relevant modules, participants responded to open-ended questions during a qualitative interview and answered a brief questionnaire. Research staff facilitated navigation and ensured participants viewed all steps. All study procedures were completed in one sitting.

A topic guide developed by the research team based on concepts from the user-experience model [38]. The guide comprised of open-ended questions eliciting participants’ perceptions and satisfaction with the design and content of the website, and feedback for enhancement. Questions were tailored for each participant group (Multimedia Appendix 2). Participants were given time to explore and navigate each page of the website. Subsequently, they were prompted to provide feedback on the website’s navigational ease, and clarity of on-page explanations for each step. Research staff facilitated website navigation and ensured the participants completed all steps. Specifically, patients and HCPs were probed on the comprehensibility of the value-clarification questions and whether they could easily select their preferred answers from a potential list of response options. Their suggestions for refining the questions and the response options were elicited. Lastly, participants were asked about their overall impressions of the website, its perceived usefulness, aspects they liked most or least, and recommendations for improvement. Participant responses were audio-recorded and transcribed (to English if conducted in Mandarin or Malay) for analysis. The interviews were conducted in a private room within the health care facility and lasted between 30‐80 minutes. We analyzed qualitative data concurrently with data collection. Data saturation was achieved by the 27th interview, when no new feedback about the website emerged.

After viewing the respective modules, participants answered a survey in the language participants viewed the website (English, Mandarin, or Malay). The survey collected demographic information, and questions assessing usability and acceptability. Usability was evaluated by the 10-item SUS; each item was rated on a 5-point Likert scale (strongly disagree to strongly agree) [39,40]. We assessed acceptability using the acceptability rating scale developed by the Ottawa Hospital Research Institute adapting it to specific decision-making aspects of our study [41]. Patients and caregivers, rated items on a 4-point Likert scale ranging from poor to excellent (score: 1‐4), and for HCPs used a 5-point Likert scale ranging from strongly disagree to strongly agree (score: 1‐5). Example items included, “it will be easy for me to use ‘My Voice’ for introducing ACP to my patients” and “this ‘My Voice’ website is better than how I usually go about conducting ACP” [41]. Lastly, patients and caregivers rated the length (too long, too short, or just right) and amount of information (too much, too little, or just right) for “My Voice.”

The surveys were first developed in English and then translated by native speakers into Mandarin and Malay and verified by a second team member fluent in the language. Before fielding, the surveys were tested with volunteers from a nonclinical setting.

Previous literature suggests that a sample of 20 participants can identify 95% of usability problems [42]. Thus, a sample size of 30 was deemed to be sufficient for usability testing. Previous studies assessing usability of digital interventions also had similar sample sizes [43].

Two authors CR and IC analyzed the open-ended interview transcripts using qualitative description methodology [44]. We categorized the interview feedback from participants into broad concepts (eg, language or layout) paying close attention to positive and negative views toward the website to derive themes and subthemes inductively. The two authors verified each other’s coding carried out in Excel (Microsoft Corp), discussed and agreed upon the final themes derived. From the survey data we described participants’ demographic and health status characteristics. We also calculated the total SUS score as a sum of each item score and rescaled it within the range of 0 to 100. A higher score signified greater usability, and a score greater than 68 indicated good usability [39,40]. We then present the total scores for each participant group and overall sample. For each item on the acceptability scale, we calculated the proportion of patients and caregivers responding as good or excellent and HCPs rating as agree or strongly agree.

This study was approved by the SingHealth Centralized Institutional Review Board in Singapore (2022/2482) and was conducted in compliance to the institutional guidelines. Participants were briefed on this study’s purpose and had the option to withdraw at any time. We obtained written informed consent and provided US $20 cash as compensation upon completion of usability testing procedures. All data was deidentified to protect participants' privacy and confidentiality.

A total of 30 participants out of 44 (response rate 68%) consented including 11 patients, 9 caregivers, and 10 HCPs. Among them 21 (10 HCPs, 6 patients, and 5 caregivers) reviewed the English version, 7 (4 patients and 3 carergivers) reviewed the Mandarin version, and 2 (1 patient and 1 caregvier) reviewed the Malay version of the website, respectively, and completed the surveys in the respective languages. Participants’ mean age was 49 (SD 14.9) years and 60% (18/30) were females. Participant characteristics are described in Table 1.

The overall mean SUS score of 74 (SD 14.8; range: 42.5 to 95), with mean scores in each of the three groups of participants exceeding the minimum cutoff score of 68, indicated good usability. Specifically, 70% of participants had scores above 68, which included 64% (7/11) of patients, 78% (7/9) of caregivers, and 70% (7/10) of HCPs (Table 4).

Patients highly rated the way information was presented in the 5 steps, with over 80% of patients rating each of the 12 items on acceptability as good or excellent. Likewise, over 88% of caregivers rated each of the 3 items about acceptability as good or excellent (Table 5). Except for one patient who found the length of the website to be “too long,” 19 of 20 patients and caregivers (95%) rated the website length to be “just right,” and 18 of 20 patients and caregivers (90%) rated the amount of information presented to be “just right.” Seven of 10 HCPs (70%) rated 11 of the 15 items as strongly agree or agree. Items that received lowest levels of agreement (agree or strongly agree) included—“My Voice website is better than how I usually go about conducting ACP” (30%), “using My Voice website does not involve making major changes to the way I usually do things” (50%), “My Voice website is compatible with the way I think things should be done” (60%), and “the use of My Voice website is more cost-effective than my usual approach to conducting ACP” (60%; Table 5)

Study results show that “My Voice” ACP website was well received by patients with heart failure, their caregivers, and HCPs. Most participants provided positive feedback regarding their experience using “My Voice.” They found the website easy to navigate, its duration and content acceptable, and valuable in conveying and comprehending patient values and goals. The mean usability score of 74 (SD 14.8) exceeded the a priori threshold (68 and above) for both overall and within each participant group. Additionally, patients and caregivers’ acceptability ratings were high for all items, and 80% (8/11) of patients and 78% (7/9) of caregivers rated the website as good or excellent, while 70% (7/10) of HCPs rated 11 of the 15 items as strongly agree or agree. Most patients and caregivers (19/20, 95%) and 90% (18/20) of patients and caregivers found the length and amount of information in “My Voice” just right. These findings confirm the usability and acceptability of “My Voice.”

The findings on usability and acceptability are consistent with those of other web-based ACP decision aids [45-47]. Our findings regarding HCPs’ views of the decision aid as a tool for initiating ACP discussions also align with previous research [48]. However, in contrast to our findings, one study reported that participants had difficulty understanding and engaging with some website content [49].

Given that a significant proportion of patients with heart failure are older with lower literacy levels [50], feedback primarily focused on enhancing the website accessibility for this demographic. Suggestions included simplifying the login process, improving the layout, reducing wordiness, increasing font size, and simplifying terminology. Moreover, some patients expressed surprise upon learning about the incurable nature of their illness, indicating that they had not engaged in serious illness conversations with their HCPs. Our previous studies have also shown similar findings [51,52].

Patients and caregivers rated “My Voice” to be both acceptable and usable. However, while, HCPs rated its usability highly, they provided lower ratings to certain aspects of acceptability. These included suggestions that “My Voice” could potentially be more effective and replace the current ACP facilitation method. It is important to note that “My Voice” is primarily a patient preparation tool intended to complement, rather than replace, the patient-HCP conversations. Therefore, the concerns raised are not unexpected given its supplementary role in the process.

While web-based ACP interventions are increasingly prevalent in the literature, our intervention is innovative in several respects. First, it incorporates structured mechanisms to encourage frequent revisits to “My Voice,” through educating patients and caregivers about its importance and sending reminders to them via phone. Our previous research revealed that ACP conversations typically occur as one-time events despite evolving patient preferences [22-25]. “My Voice” thus addresses this current gap in ACP implementation by facilitating ongoing reflection on values and goals and fostering periodic ACP conversations with HCPs. Second, it is tailored specifically to patients with heart failure, featuring educational videos and a quiz regarding their illness. This targeted education not only imparts urgency but also provides the context for patients to reflect on their values and goals. Third, “My Voice” seamlessly integrates active caregiver involvement into the ACP process. Not only do we coach patients to choose a surrogate decision maker and engage with them, but our dedicated caregiver module educates the chosen surrogate about the patient’s illness and guides them on how to communicate with the patient and the HCPs. This caregiver module also facilitates the sharing of the patient’s “My Voice document” with the surrogate. This approach enhances the caregiver’s understanding of patient values and goals, preparing them to make end-of-life decisions for their loved ones. Importantly, caregivers in our usability study appreciated the “My Voice” website’s unique features, particularly its capability for patients to periodically update their goals and share them with their surrogates. Lastly, the inclusion of content in three different languages enhances the accessibility of “My Voice” to a broader range of ethnic and language groups, promoting inclusivity and ensuring that individuals from diverse backgrounds can effectively engage in the ACP process.

“My Voice” is one of the first web-based ACP interventions for patients with heart failure in the Asian context. This study’s strength lies in its use of a mixed methods design to gather participant feedback, agile methodology implementation to enhance user satisfaction, and inclusion of different ethnic and language groups across multiple sites. However, there are some limitations. The response rate from patients was low (46%), partly attributed to the older age of many participants approached and their reluctance to engage in web-interventions. Future studies could improve response rates by involving health and social care providers, community organizations, or adult children to facilitate initial contact with older individuals. Additionally, although the sample size was adequate to achieve thematic saturation, it was too small to discern variations across age, gender, and other sociodemographics.

“My Voice” is a usable and acceptable tool for empowering patients to engage in ACP conversations with their caregivers and HCPs. The efficacy of the “My Voice” website in improving patient and caregiver outcomes is being assessed through a randomized controlled trial.

The findings support the usability and acceptability of the web-based ACP intervention, “My Voice,” among patients with heart failure and their caregivers. Participants largely endorsed the interactive website as a valuable tool for communication and understanding patients’ values and goals, offering constructive feedback to enhance its user-friendliness for older patients.