Free-Text Responses in a Nationally Representative Experimental Survey about End-of-Life Care Choices: ChatGPT-4o-Assisted Qualitative Analytical Study

Introduction

Background

Many chronically ill older adults recognize the importance of discussing their end-of-life desires with their surrogates, family, friends, and clinicians, in the event they are unable to communicate during an emergency. Having advanced care directives reduces surrogate stress by avoiding the urgent decision-making process during emergencies and potential undesired treatment. It also may increase the likelihood that patients receive the care they desire, particularly in end-of-life situations where they may be unable to express their wishes [1]. Despite the increased use of advanced care directives, there are still incongruent care preferences between patients and their surrogates [2].

Gaps and Preliminary Studies

This discordance in end-of-life decision-making suggests that better facilitation and understanding of end-of-life decision-making are needed [3]. Previous studies have focused on the effectiveness of advanced care directives, caregiver communication, or physician influence only [1,4-6]. However, the end-of-life decision-making process is complex, and individuals must consider multiple factors when deciding. Our team created a vignette-based cross-randomized survey to better understand surrogate factors in making end-of-life decisions [7]. Quantitative results suggest that surrogates were less likely to recommend life-sustaining treatment for persons living with dementia, that surrogates were more likely to ignore patient preferences for life-extending treatments when the person had dementia, and that surrogates were more likely to choose treatments that matched their own preferences when patients’ wishes were unclear.

Objectives

In this study, our objective was to analyze free-text responses to the survey question “What were the main considerations that led you to give the answers you provided in the module above (which captured your end-of-life care choice for your care recipient)?” Given this survey included 6109 individuals, we decided to leverage artificial intelligence (AI) to assist our team in performing a qualitative analysis. Alternative existing approaches to analyzing such a large volume of data, including qualitative thematic analysis, have drawbacks including the need to select and analyze a smaller subset of responses to avoid the extensive resource use needed to perform traditional qualitative analysis [8]. Here, we share the findings of our qualitative analysis of free-text responses relating to end-of-life care decision making and detail how we used an AI-generative chatbot, ChatGPT-4o, to facilitate the analysis.

Methods

Data Sources

Overview

We analyzed free-text responses from a nationally representative, web-based experimental survey that included 6109 adults aged ≥18 years [7]. Thirty-five percent of survey respondents did not provide a free-text response to the question soliciting a rationale for the end-of-life care choice made in the vignette and thus were not included in this analysis. This survey was developed by co-authors LHN and MM. Both are PhD health economists and professors with experience studying the interaction between health care use and economic outcomes through combining survey, administrative, and clinical data. No relationship was established between the research team and participants before study commencement. A professional survey company, Ipsos Knowledge Network survey platform, recruited participants between December 8, 2022, and December 19, 2022, and ensured complete anonymity to the researchers, reducing social-desirability bias and other experimenter-demand concerns [9]. Surveys were available in English and Spanish, and participants were provided with devices to complete the survey if they lacked home computers or internet. This vignette-style survey was intended to systematically investigate surrogate decision-making in the context of dementia-related cognitive impairment and end-of-life care for older adults [7]. Further details on the type of data collected, timeline of data collection, and procedures are detailed in our previously published paper [7].

The objective of this paper was to analyze all the free-text responses qualitatively that were provided to the question “What were the main considerations that led you to give the answers you provided (Comfort Care or Life-extending Care) in the module above (which captured your end-of-life care choice for your care recipient)?” We followed the Standards for Reporting Qualitative Research (SRQR) guidelines (Checklist 1) in reporting this work [10]. Our dataset did not include any personal health information as this was an anonymous survey to a nationally representative group. Thus, we did not provide any personal health information to ChatGPT. While ChatGPT was used as a tool to perform efficient coding and theme generation, it was not used for writing assistance.

Summary

We performed several steps to conduct this qualitative analysis of free-text responses including (1) following traditional thematic analysis procedures to code and generate themes on a subset of free-text responses [11], (2) prompting ChatGPT-4o to use our codebook to generate its own themes and quotable quotes, and (3) debugging and validating the AI responses. We first familiarized ourselves with the data by reading and rereading the first 200 responses to the free-text question. Then, we hand-coded the first 200 responses, generating a preliminary code book and iteratively improving it through team discussion. Next, we created a list of preliminary themes and illustrative quotes following traditional thematic analysis procedures.

After providing the data and our codebook to ChatGPT-4o (GPT-4 variant, May 13, 2024; used via Google Chrome browser), we prompted it to generate relevant keywords, keyword counts, themes, and illustrative quotes. We performed several steps to validate the AI-generated results, including comparing AI-generated themes and quotes with themes and quotes generated by qualitatively trained human professionals and comparing keyword counts with counts generated by a commonly used qualitative analytic software, NVivo (Lumivero) [12]. Finally, we put the findings into the context of the current literature.

Phase 1: Generating a Valid Approach

Prior to initiating the analysis, we conducted a literature review to identify how AI had been used in previous qualitative research. In particular, our methods were shaped by studies by Nashwan and Abukhadijah [13] and Van Veen et al [14], who provided prompt examples. Only limited studies have used AI approaches for analyzing free-text responses, so we also compared ideas on how to validate our AI findings with other academic teams doing similar work. These discussions yielded several suggestions, including comparing themes side by side between AI and human-generated themes and doing a keyword count comparison with commonly used qualitative software.

Phase 2: Familiarization With the Data and Hand Coding

First, a physician scientist with experience caring for patients at the end of life and graduate-level qualitative training (EMG) and two other qualitatively trained research staff members (MB and JGP) read the first 200 participant responses. Then, 2 team members (EMG and MB) inductively coded the responses to the first 200 responses together. At this point, we reached thematic saturation, whereby no new significant themes continued to emerge, and concluded manual coding [15,16]. During this session, the team members created a preliminary coding schema in which we identified 6 major codes and several subcodes relating to end-of-life decision-making (Table S1 in Multimedia Appendix 1). The codebook was then iteratively refined through group discussion. Then, a third team member independently coded the same 200 responses (JGP). Finally, the principal investigator of the study (LHN), who designed the survey and had performed a preliminary human qualitative analysis after reading a sample of the responses, provided additional suggestions on the codes from her analysis, which we incorporated into the final codebook after team discussion.

Phase 3: Human-Identified Themes, Subthemes, and Illustrative Quotes

Once we felt confident that our codebook reflected the topics and scope of the data from the first 200 respondents, we generated themes and quotes that were illustrative of the major and minor themes. Investigator triangulation (multiple investigators with multiple areas of expertise) was used to establish the trustworthiness of our findings [17]. We compiled a table to display themes and quotes.

Phase 4: AI Analyses

Next, we used ChatGPT-4o to extend our human-generated analysis by providing it with the entire dataset of 3931 responders to the free-text question. We also piloted other generative AI tools (Microsoft Copilot and ChatGPT-3) but found that ChatGPT-4o provided the most usable responses and allowed us to upload larger amounts of data. Before entering the dataset in ChatGPT, we cleaned the data by removing all nonresponses. Free text containing responses of “none,” “n/a,” “na,” “no,” “nothing,” “no answer,” or “…,” were removed from the dataset submitted to ChatGPT to limit error. These responses were instead coded as “0” to indicate no free text was available. Data were uploaded via a Microsoft Word document containing a table with headings labeled “Free Text Responses,” “Age,” “Race/Ethnicity,” and “Gender.” ChatGPT processed the entire document at one time and required no chunking of information submission. No application programming interface or interface was used.

Our initial prompt to ChatGPT provided important context on the task, including that we planned to perform a qualitative analysis of free-text responses and that the AI should act as a qualitative analyst using our codebook and data to identify themes and quotable quotes. Although we considered having ChatGPT-4o inductively code the responses without human input, we anticipated that this would require much more investment by our team on the back end to iteratively prompt and refine themes. We also wanted to avoid the risk of irrelevant or erroneous themes or excessive AI “creativity.” We also prompted it to find the most common keywords in the text and generate keyword counts.

To illustrate how we completed this analysis, so it can be replicated, we share several sample prompts and how we validated responses:

1. The first prompt set up the research study context, its methods, and the AI’s role. An example prompt is: “This is a large survey study focused on end-of-life decisions, and you are a qualitative researcher analyzing the free-text responses to these questions.”
2. Next, we asked ChatGPT-4o to code the data using our codebook. A sample prompt includes: “This is a preliminary coding schema for our research. Using this codebook, code the entire sample. How did you do this?”
3. We asked ChatGPT-4o to create themes and subthemes and identify quotable quotes or key phrases that “make an impression on people from the text.” We then manually searched the dataset in NVivo to ensure the quote was verbatim and not fabricated.

Then, we asked the AI to create keywords identified in the entire dataset. We first quantified keywords by querying NVivo using the “word frequency/query” function. NVivo created a list of the most frequently used words in the dataset with the exception of proper names. NVivo provided the keyword, its frequency in the dataset, and the weighted percentage out of all words in the dataset. Then, we provided each keyword to ChatGPT-4o in addition to supplying it with a data file which included the full dataset of responses. We then queried ChatGPT-4o for exact keyword counts and weighted percentages, for example, “How often is the word ‘life’ found in this dataset, and what is the weighted percentage of ‘life’ among all words in the dataset?” by asking it to find how many times the specific keyword could be identified in the dataset.

Ethical Considerations

The study was found to be exempt by the Johns Hopkins Homewood institutional review board (HIRB00012400), and we followed SRQR guidelines in reporting this work. Our dataset did not include any personal health information as this was an anonymous survey of a nationally representative group. Thus, we did not provide any personal health information to ChatGPT. The survey participants received compensation for their participation; the amount is unknown and proprietary due to a lump sum getting paid to Ipsos, the survey company.

Results

Overview

Free-text responses were created by 3931 (65%) surveyed participants. Free-text responders were significantly older, had higher educational attainment, were less likely to be working full time, and were more likely to be a caregiver for another adult than nonresponders. They were also significantly more likely to be married, have higher income, and carry a diagnosis of a chronic illness. Over half of free-text responders were people with chronic illnesses. A total of 1438 (37%) of them identified as caregivers, of which 601 (42%) stated that they provided ≥20 hours of care each week (Table 1). No significant differences were noted for sex, race, or region of the country.

Human coding identified six major themes: (1) rationales provided for making end-of-life decisions, (2) the role of physical function in decision-making surrounding end-of-life care, (3) the role of dementia in decision-making surrounding end-of-life care, (4) impact on caregivers, (5) quality of life, and (6) basing decision on personal preferences or “I would” statements. Subthemes identified by human coding included: expected trajectory, drawing on individual experiences to decide, the caregiver feels a duty to choose to extend life, patient autonomy or preferences, ethics, emotional strain on family, limited quality of life with dependence on others, and living a life with value (Table 2).

ChatGPT-4o identified 7 major themes from the free-text responses and also identified subthemes and representative quotes (Table 3). We further asked ChatGPT-4o to indicate the demographics of the responder (Table S2 in Multimedia Appendix 2) and indicate if the proxy was choosing comfort care, life-extending care, or both, which it performed accurately (Table S3 in Multimedia Appendix 3). In Table S3 in Multimedia Appendix 3, we also categorized errors made by the AI, including fabrication of quotes and substantial modifications of quotes. To aid in the replication of this work, we summarized the additional steps taken to ensure rigor in our analytic approach in Multimedia Appendix 4.

Validation and Lessons Learned Using Generative AI

In Table 4, we depict a comparison of human- and ChatGPT-4o-generated major themes and subthemes. Several themes overlapped between human (black text) and AI (purple text) thematic analysis. While the AI did not always name the theme the same as our human analysts, the meaning was similar. Overall, there were several differences observed between human-generated and AI-generated tables, including that the AI closely relied on the codebook to generate themes, representative quotes were shorter, and more themes were generated by the AI. In addition, AI-generated themes were more descriptive—summarizing directly what participants said—rather than interpretative (eg, offering latent meaning or suggesting why participants may have said what was stated). ChatGPT created themes such as “quality of life” without extrapolating underlying motivations or cultural values, such as human-derived “drawing on individual experiences to decide.”

We note that the AI generated 3 themes not included in our human-generated analysis: decision-making challenges, ethical and religious considerations, and patient dignity. While codes relevant to these topics existed in the human codebook, they were not prioritized in the final theme set due to their relatively lower frequency in the subset of responses reviewed manually. It is possible that these themes appeared more frequently across the full dataset analyzed by ChatGPT, which highlights the potential for AI methods to surface lower prevalence but potentially meaningful perspectives that might be missed in human review.

During the validation of ChatGPT-4o-generated quotable quotes by human analysts, we identified 3 major errors. First, ChatGPT-4o would often combine statements from multiple participants into one quote. Second, it sometimes ignored the prompt to create verbatim quotes and instead modified the quote to be more concise or grammatically correct. Third, it sometimes fabricated quotes entirely (hallucination). For the first error, combining statements, we addressed the challenge by reformatting the data into a table rather than using Microsoft Word, so the AI could more easily identify when one quote ended and another started. This improved ChatGPT-4o’s quotable quote reliability substantially. Our previous data format separated participant responses with spaces only. For the second error, paraphrasing, we reprompted ChatGPT-4o that exact quotes were needed, and they could not be modified. With reprompting, ChatGPT-4o was able to list verbatim quotable quotes. Below we display an example of the second error. While the meaning is the same, the content is paraphrased. For the third error, complete fabrication of quotes, we removed these from the tables and reprompted the AI to choose new verbatim quotes followed by verifying their existence in the dataset. We show more examples of all 3 types of errors in Table S3 in Multimedia Appendix 3.

Original quote by responder:

Life is precious and we should not try to end it of our own accord. It is God’s role to extend or end a life.

ChatGPT-4o-generated quote:

Life is precious and should be preserved as long as possible, according to God’s will.

In Table 5, we listed the 11 most common keywords identified by NVivo within the free-text responses and provided the counts and weighted percentages for keywords in NVivo and ChatGPT-4o. We chose the 11 most common keywords because starting at the 12th keyword, the frequency of the count was much lower. We identified that NVivo and ChatGPT-4o were using different methodologies to provide keyword counts and weighted percentages. Initial prompts to ChatGPT-4o generated counts and weighted percentages that were lower than NVivo, suggesting undercounting. We show how we corrected the discrepancy with the following example. To correct the count for the keyword “life,” we told ChatGPT-4o that NVivo had come up with a count of 2051 for the keyword “life” and asked it what it had done incorrectly, as its initial count for life was 1984. ChatGPT-4o responded that the NVivo count was correct, and what it was doing wrong was “The initial discrepancy was due to not properly accounting for cases where ‘life’ was followed by punctuation or appeared at the end of a sentence or line.” We then asked ChatGPT-4o to repeat the experiment using the correct method. Counts were still discrepant after this correction, though the ChatGPT-4o counts were closer overall to the NVivo counts after this reprompting. We then discovered that NVivo excluded “stop words,” which were considered insignificant. When we changed the NVivo settings to include the stop words, we then produced considerably closer weighted percentages, which are shared in Table 5.

Time Considerations and Replication

Human coding was time-intensive; our team spent 4 hours coding the first 200 responses to the free-text question, recoding the first 200 responses, and creating and revising the initial codebook. ChatGPT-4o took <10 seconds to generate themes using our codebook. However, preparing the dataset for ChatGPT-4o, verifying the content, subsequent reprompting and verification, identifying methods for debugging, and validating took several weeks.

Due to ChatGPT-4o being a large language model (LLM), replication of similar outputs using the same prompt and dataset was challenging. We often received varying answers to the same command using ChatGPT-4o, although the overall major themes identified within the dataset remained relatively consistent. Once we reformatted responses into a table from the initial word file, ChatGPT-4o had more consistent response outputs to the same prompt.

Discussion

Principal Findings

In this ChatGPT-4o-assisted qualitative analysis, we identified several themes related to how surrogates make end-of-life care choices for patients. Proxies’ decisions are informed by prior caregivers’ prior lived experiences, expectations of quality of life, how advanced the stage of dementia is in the patient, and also personal values, including religious beliefs. In performing this analysis, we also learned several lessons about using AI tools to extend the capacity of our qualitative team. We learned that ChatGPT-4o is capable of using a human-created codebook to search free-text responses to create themes, representative quotes, and keywords. In addition, we learned that ChatGPT-4o-generated themes are primarily descriptive and closely match codes identified by humans. We also found that validation by humans is necessary, and ChatGPT-4o can sometimes provide insights on errors and correct them when prompted to do so.

This unique cross-randomized experiment allowed our team to provide new insights into what role dementia plays in surrogate decision-making. We found that many surrogates consider the stage of dementia to make end-of-life decisions. Caregivers may correctly identify that dementia is incurable and that extending life when dementia is advanced may not allow the individual to live the way they once did, free of the support of others. Others identified that quality of life in dementia may be tied to the ability to recognize and socialize with others, and when recognition diminishes, extending life may not be desired. Prior research also found that some surrogates equate dementia with loss of the “self” and view recognition as a key marker of dignity and quality of life [18]. Some felt extending life was warranted when the patient previously requested it or because of their religious beliefs.

Another common theme was drawing on individual experiences or preferences to make decisions. A key finding in the quantitative survey and the AI qualitative analysis was that when preferences were not clear, surrogates made choices that aligned with their own wishes. Personal beliefs about how one would feel if faced with these life choices were weighed in making the decision. However, ethically, proxies should make choices in line with patient preferences and not their own [19]. In some cases, proxies chose comfort care after projecting what their future self may feel like when aged or if they had the functional status of the patient described in the vignette. These survey and qualitative findings could inform clinical practice in several important ways: (1) clinicians should feel empowered to inform proxies that decisions should be made that are in line with patient preferences and (2) education should be provided to proxies about advanced dementia and its impact on quality of life, as well as the impact life-extending care may have on patient comfort and well-being.

Other literature on end-of-life choices and surrogate choices have found that while 90% of caregivers acknowledge that dementia is incurable, only 40% see it as life-limiting [20]. One study indicates that caregivers of people living with dementia who have chronic illnesses and who discuss goals of care with clinicians are more likely to have accurate prognostic understanding on the end-of-life care for persons living with dementia [21]. In addition, caregivers with better prognostic understanding were more likely to state a preference for comfort-focused care, and their care recipients were less likely to receive burdensome interventions and have greater comfort during the dying process. Studies have also recognized racial differences in advanced care planning, with African American persons reporting lower rates of advanced care planning completion (89% vs 73%) and lower preference for comfort care [22]. Several studies indicate that more education of proxies about the stages of dementia and end-of-life care options would be helpful to improve the quality of proxy decision-making [21,22].

The use of natural language processing AI-augmented qualitative analysis has been described as the “Way of the Future” [23]; however, few investigators have used it to perform qualitative analysis [24]. One team used structured topic modeling coupled with thematic analysis to analyze free-text data from nearly 38,000 English patrons with the aim to improve the user experience of public-facing services in the United Kingdom [25]. This team tested models with 5 to 40 topics and differing covariates and then had 2 human coders conduct thematic analysis to interpret the topics. In contrast with this approach, our approach did not require team members to have requisite skills in machine-learning methods and thus may be more accessible for qualitatively trained teams. Other AI approaches to analyzing large quantities of free text have included using natural language processing and machine learning models to develop robust classification models [26], but this approach also requires data science or related technical expertise.

Other teams have used ChatGPT-4o and more accessible LLMs to analyze qualitative data [23,27,28]. Experiences of these investigators can be summarized as overall positive in that these LLMs were found to be capable of producing satisfactory analyses that come close to human-generated analyses, but the performance of different LLMs varied, was not predictable, and serial prompting was necessary. Many have stated that themes are more descriptive than interpretive, yet it was easy to use and required less effort than manual coding. We also found that AI themes often did not go beyond what was explicitly stated by the participant and failed to explore latent meaning or social context. This could pose a barrier to using these tools in research where interpretative themes are essential. In addition, researchers found that even when using the same prompts and dataset, slightly different output was generated as AI generative chatbots “learn” over time. While this could improve performance, it also introduces problems with reproducibility. Thus, our team found it important to carefully log all prompts and responses in a log. In contrast to our approach, Carvalho et al [27] used a “series of chain-of-thought prompts to dig deeper into the provided response.” This approach had 2 advantages. It allowed the investigators to guide the chatbot to provide them with more positive themes when negative ones were initially suggested, and it surfaced less commonly encountered themes, allowing for more diversity of perspectives. Carvalho et al [27] also found that the paid version of ChatGPT-4o, which our team also used, generated the most accurate responses compared to the unpaid version and Google Gemini. This was particularly true when it was well-prompted and after multiple repetitions were performed in the same analysis.

Because there is currently no established gold standard for using advanced chatbots for qualitative analysis, our team made several deliberate methodological decisions. These included providing the AI a human-generated codebook rather than allowing it to inductively code responses and generate themes “independently.” Other teams may choose to allow the AI full discretion in its coding, and this approach could be more efficient and lead to more novel themes. We made this choice because our team had ample qualitative expertise, which allowed for us to code a subset of responses and ensure the codebook reflected key domains of interest. However, this approach may have reduced the AI’s ability to discover novel themes and constrained “creativity.”

Barriers to the use of AI in qualitative research include concerns over data privacy, a need for upskilling in AI among trained qualitative investigators, and a lack of established approaches to use the tool [24]. Information such as transcripts uploaded to AI platforms is used to develop the underlying algorithms that allow the computer to learn, which some academics feel is problematic from a data privacy standpoint. It is important for academics to follow institutional best practices such as obtaining approval from their ethical review boards prior to engaging in this work. Some institutions may advise using enterprise AI solutions or other tools that allow investigators to select that inputted data will not be externally shared or used to improve the AI’s algorithm. Another barrier has been that senior investigators trained in qualitative methods may not possess the skills to use AI for qualitative analysis or teach others how to use it. Similarly, a roadmap is not currently available that outlines how rigorous qualitative analysis can be done with AI assistance. We believe that AI will be increasingly used to analyze large qualitative datasets, and as its use and popularity grow, investigators must learn through trial and error how to best use it and debug it.

Limitations

Free-text responses are an important source of data in surveys that collect quantitative measures because they can provide context to answer choices; however, unlike semistructured interviews, there is no opportunity to prompt the interviewee to elaborate on points or explain their statement when it is not clear. We found sentence fragments or short replies that made it difficult to understand what the surrogate’s perspective was and could be interpreted differently depending on the person reading the text. Although our research team has a multidisciplinary background, we recognize that we bring our own bias to our work, and this may have affected our interpretation of our findings. We made efforts to increase the validity of ChatGPT-4o’s analysis by asking it to show its work, comparing its output with existing qualitative software, and completing our own a priori human analysis on a subsample of the qualitative data, but the use of this tool for qualitative analysis is still new, and thus, there may be errors that we did not recognize.

Conclusions

Participants in a vignette-style nationally representative survey asked to act as proxies provided several different rationales for why they chose to extend care or focus on comfort care in their hypothetical care recipients. Choices were often made based on whether the care recipient had dementia and how advanced it was, perceived quality of life, or the caregivers’ lived experiences or values, rather than making decisions aligned with the care recipients’ previously stated wishes. This suggests that it is in an individual’s best interest to share their care preferences with their intended proxy and engage in advance care planning prior to losing health care decision-making capacity.

Using an AI tool to perform qualitative analysis on free-text responses may help extend the work of qualitatively trained investigators, especially for large datasets such as free-text responses to large surveys. Our advice to investigators who aim to use ChatGPT-4o or other AI generative chatbots is to first complete a human-generated analysis using traditional qualitative coding, provide a codebook to the AI, anticipate the need for multiple prompts, try different LLMs, plan validation approaches and time for debugging, and keep a prompt and response log.