Voice-Enabled Intelligent Virtual Agents for People With Amnesia: Systematic Review

Introduction

Background

Older adults often complain about amnesia or increasing memory problems, although these cognitive changes affect some individuals more than others [1]. Although some degree of cognitive slowing is typical of normal aging, when the acquired cognitive impairment has become severe enough to compromise social or occupational functioning, the diagnosis of dementia is typically established [2]. Alzheimer disease (AD) is the most commonly diagnosed form of dementia. If the functional abilities of patients are still essentially preserved while their cognitive abilities are in between those associated with normal aging and dementia, people are typically diagnosed with mild cognitive impairment [2].

This gradation in cognitive abilities does not suggest a necessary sequence of normal cognitive slowing to mild cognitive impairment to dementia; most people will only experience normal cognitive slowing upon aging, some will develop mild cognitive impairment and some will develop dementia. For the latter category, the period between first serious cognitive complaints and the diagnosis of dementia can be 10 years and depends on many factors such as age, sex, and general physical premorbid condition [3]. The duration of survival after AD diagnosis is 10.2 years for men aged 65 years and 13.2- years for women of the same age [4]. In this period after diagnosis of AD, the period for need for home care typically lasts 3.7-4.7 years and for institutional care, 2.2-3.2 years [4].

During the period of normal cognitive decline without the need for additional care, people may benefit from personalized support. In follow-up phases, where people need extended home care or institutional care, the need for personalized support increases. In the early phases, such support is often provided by partners, children, friends, or other relatives. When these are not available, but the patient is still living independently at home, professional help by, for example, case managers, district nurses, and meal delivery services may be needed upon the indication of a patient’s general practitioner. If the need for institutional care arrives, people may move to a nursing home and be cared for by nurses and other health care professionals. However, in each of these phases, informal caregivers or health care professionals are often not sufficiently available for the needs of the patient. For example, Buchan et al [5] have identified a global shortage of 17 million health care workers in 2019. This shortage will only increase owing to the growing percentage of older adults in the total population (9% in 2019 and 12% in 2030) [6]. Hence, society is looking for alternative solutions such as technological systems that could support health care professionals in their work by taking over automatable tasks. One such potential solution can be offered by intelligent virtual agents (IVAs).

IVAs can be defined as interactive digital characters that exhibit human-like qualities and can communicate with humans using natural human modalities such as facial expressions, speech, and gestures [7]. This broad definition includes intelligent virtual characters that manifest themselves as text-based chatbots on smartphone or tablet and virtual characters in the form of a human head or a complete person on a tablet or computer screen. Several other terms are used for this type of agent, such as virtual assistant, (embodied) conversational agent, cognitive assistant, chatbot, intelligent assistive technology, and virtual human. IVAs have also been the subject for a series of conferences on this theme organized by the Association for Computing Machinery since 1998 [7]. As it has been a topic of research for more than 3 decades now for many applications, a large volume of research papers have been published on this subject.

For the purpose of this study, the main interest is in IVAs implemented as onscreen virtual characters that support older adults through autonomous verbal interaction. This specification is based on 3 reasons. First, from the perspective of the patient, verbal interaction is considered the easiest and most natural way to communicate and build rapport with an agent [8]. Second, a virtual character is more inviting to have a verbal communication with than a text-based chatbot. Third, once a person is used to this type of communication, a many-year support period during various forms of slowly progressing cognitive decline could be possible: talking is something people can do for a long period, whereas pushing buttons on a touchscreen still assumes some digital literacy, and this may disappear with cognitive decline [9].

IVAs have potential advantages for organizations that deliver home or institutional care. These organizations could decrease the need for a 24×7 human support team if a large part of the frequently asked support questions can be handled through an intelligent dialogue with an IVA. Furthermore, IVAs are immediately available and there is no need to wait in line for the availability of a health care organization employee. Interaction data can be stored and analyzed both on a personal and on an aggregated level, where the latter is of interest to the health care organization as well as to governmental health care control institutions. Obviously, ethical and privacy concerns need careful consideration and as a start can be addressed by a compliance check with the ethics guidelines for trustworthy artificial intelligence (AI), as published by the European Commission [10,11]. At the same time, IVAs have not yet been widely introduced in the consumer market and can thus far be found primarily in laboratory environments. Thus, the question arises as to whether IVAs have actually been developed and evaluated as verbal coaches or companions for cognitively impaired older adults. Therefore, this research first considered related systematic reviews and then identified the current state of the art through a systematic literature review.

Other Reviews

Other systematic reviews on virtual agents in health care revealed that the number of experiments with voice-enabled agents was rather limited. Xie et al [12] conducted a review on AI specifically for caregivers of persons with dementia, but did not report on any virtual agents for which an actual experiment had been conducted. Schachner et al [13] reviewed papers on AI-based conversational agents for chronic conditions. Although they found 2052 articles, only 10 met their inclusion criteria for research on chronic diseases involving an AI-based conversational agent. Of these 10, 2 papers dealt with dementia, but did not include a virtual agent.

Ienca et al [14] performed a systematic review on intelligent assistive technology for several forms of dementia including AD and included 571 studies. They examined the technological type of the interventions among others, but did not make a clear distinction on whether the intervention involved a virtual agent capable of 2-way verbal communication.

Bevilacqua et al [15] conducted a systematic review of the effectiveness of coaching through technology among older adults. From their original set of 2186 articles, 8 met their inclusion criteria, among which the criterion of the study was a randomized controlled trial. This criterion was more stringent than the ones used in this study, meaning that studies that could be included in this study might be excluded in their search strategy. In contrast, none of the studies they found aimed at people with cognitive problems.

Car et al [16] conducted a scoping review on conversational agents in health care. They did not report on any virtual agents such as those depicted in Figure 1, as their results were restricted to smartphone apps. Laranjo et al [17] also conducted a systematic review on conversational agents in health care. From the initial 1513 search results, they included 17 studies that evaluated 14 different conversational agents with unconstrained natural language input capabilities. From their results, 2 studies focused on an embodied conversational agent similar to that in Figure 1, but one of these studies was aimed at military personnel with posttraumatic stress syndrome (PTSS) and the other one, on training for people with autism spectrum disorders.

‎View this figure

King and Dwan [18] created an inventory of electronic memory aids for people with dementia who experience memory loss increasing with age. They found 16 studies that met their inclusion criteria, one of which also met the criteria drawn up for this study: a study by Tokunaga et al [19] on a memory-aid service agent.

Provoost et al [20] reviewed embodied conversation agents in clinical psychology. They included 54 publications after an initial search result of 1117 references, but the disorders studied were autism, depression, anxiety, PTSS, schizophrenia, and substance abuse. Depression and anxiety are often associated with dementia. However, the studies related to these subgroups did not meet the criterion of a virtual agent.

Loveys et al [21] specifically examined the design features of embodied conversational agents and the extent to which these have an effect on the relationship quality between human and agent. Their systematic review resulted in 43 studies that examined design features such as language use, behavior, emotional expression, embodiment, appearance, personality, environment, and a combination of these. However, none of these studies were aimed at people with dementia.

Milne-Ives et al [22] performed a recent systematic review on the effectiveness of AI conversational agents in health care. They found 31 studies including a variety of conversational agents, among which were 14 chatbots and 6 embodied conversational agents; however, none specifically targeted people with cognitive problems such as dementia.

It is possible that some of the papers that were excluded from the aforementioned reviews met the inclusion criteria for this study. In any case, there is a gap when it comes to IVAs for people with amnesia, dementia, or AD, and with whom 2-way verbal interaction experiments have been conducted. This gap calls for a specific systematic review on that topic.

Objectives

This study aimed to evaluate IVAs and their interaction functionality, which have demonstrated a verbal communication for people with amnesia. Our hypothesis is that IVAs designed to assist older adults with memory problems have a positive usability. This review reports on studies with various voice-based IVA applications that have been developed to support these patients and the studies, which included (pilot) evaluations with this target group.

Methods

Eligibility Criteria

For the purpose of this review, an IVA was defined as an agent with a virtual embodiment (full body or face only) that was capable of speaking to a human (ie, playing a generated audio file). The agent shall be capable of having a 2-way verbal dialogue, that is, the human could speak to the agent, the agent could process the human input, and as a result, the agent would speak to the human. Therefore, the agent should also be capable of speech recognition, that is, converting the human audio signal (human utterance) into a text string. This text string should then be made available for response determination or natural language understanding, that is, intent, belief, or desire determination and action selection. Verbal response actions should be input for a text-to-speech function. The agent should be displayed on a permanently available screen and should not require the mounting of a virtual reality headset that is difficult to wear in a 24×7 setting and may also cause motion sickness and disorientation [23].

Studies eligible for review were further required to (1) include an experiment, pilot study, or randomized controlled trial with experimental results, thus not describing only requirements or designs; (2) target participants from an older adult population with potentially memory-related problems; and (3) be published in a peer-reviewed journal or in peer-reviewed conference proceedings.

Studies were excluded if (1) the agent was a physical robot, a purely text-based chatbot, or a virtual reality or augmented reality character; (2) the study concerned a Wizard of Oz study in which the researcher mimicked the agent and circumvented automated speech recognition and natural language processing (NLP) challenges; (3) the paper language was other than English; (4) the search result concerned a thesis or dissertation (as far as these were not peer reviewed and therefore did not meet publishable standards); (5) the search result was an abstract only, a PowerPoint, or a website; and (6) the full text was not available for the authors.

Search Strategy

A systematic search was conducted for keywords on papers included in electronic databases from health and computer sciences including PubMed, SCOPUS, Microsoft Academic, Google Scholar, Web of Science, and CrossRef. The search term was (Virtual Agent OR Virtual Assistant OR Virtual Human OR Conversational Agent OR Virtual Coach OR Chatbot) AND (Amnesia OR Dementia OR Alzheimer OR Mild Cognitive Impairment) and the search period was 2010 to present. This period has been defined for several reasons. First, the statement by Wargnier et al [24] that they did not find a publication on a usability interaction of a talking virtual agent with older adults with cognitive impairment. Second, 2010 was named as the start of the era of deep learning in speech recognition, which caused an explosion in the success of speech recognition applications [25,26]. Third, the limited number of relevant papers originating before 2010 and found in other systematic reviews [12-16,18,20-22]. The Publish or Perish tool was used to conduct the search, collect the results, and export them via csv-files to an Excel spreadsheet for study selection [27].

Study Selection

Two reviewers (RB and YvdS) independently conducted the search and compared and agreed on the results. Titles were screened upon clearly including the words of the search or being strongly related, based on the assumption that the title of the research paper should be clear on its contents. In case of doubt, the abstract was consulted. From this selection, the abstracts were reviewed for a second selection. The abstracts should include a reference to a verbal interaction experiment of a virtual agent with older adults and should be published in a peer-reviewed journal paper or conference proceedings. Abstracts that met these criteria were subsequently discussed and selected for full-paper text analysis. Studies deemed eligible for review were included in data synthesis.

Data Collection

The data of the selected papers are provided in a table format. The top row includes the authors, title of the paper, journal or conference proceedings where it was published, and the year of publication. Row 2, column 1 describes the data on the study: country, study design type, target sample size, controls, and study aims. In row 2, column 2, the actual experimental population is reported, whereas row 2 and column 3 details the intervention. Row 3, column 1 specifies the typical interaction between the IVA and the human, with specific attention to the verbal interaction options for the human and the NLP techniques that were applied. Row 3, column 2 provides a summary of the reported results, and row 3, column 3 presents an image of the IVA.

Quality of Study Evaluation

The standard quality assessment criteria for evaluating primary research papers developed for qualitative research by Kmet et al [28] were used by reviewers TB and ST to evaluate the quality of the studies found and the risk for bias. This method uses the checklist presented in Textbox 1.

The eligible papers were scored for the aforementioned items as yes (2 points), partial (1 point), no (0 points), or not applicable. Agreement was reached on whether an item could be scored or defined as not applicable and ignorable. The score of the reviewers TB and ST per item was averaged. The overall score per paper was calculated by dividing the summed score by the total number of scored items, multiplied by 2. This resulted in a score between 0 and 1.

Results

Study Selection

The search was conducted in the first week of March, 2021. In total, 2599 papers were found, and the study flow in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology can be found in Figure 2 [29]. Most of the papers (2305/2599, 88.69%) found by the reviewers YvdS and RB were obtained from Google Scholar search. A total of 486 records were removed before screening, because they were published in a different language (n=54, 11.1%), they appeared more than once in the list (sometimes under a slightly different title; n=347, 71.4%), or for other reasons (n=85, 17.5%) such as not showing an author or having an illegible title. The resulting 2113 papers were screened on the text of the title and papers that did not clearly mention one of the search terms or a synonym in the title were excluded. In case of doubt, the abstract was consulted. This resulted in 204 papers that were subsequently screened by the reviewers for their abstracts. After screening abstracts, 26 papers were selected for full-text analysis. After full-text analysis, 8 papers that satisfied all the criteria were included in this review. Multimedia Appendix 1 [29] and Multimedia Appendix 2 [29] present the PRISMA checklists.

‎View this figure

Selected Studies

This review resulted in 8 selected studies. The study data are summarized in Table 1 and sorted by avatar. The average number of participants in the studies was 20 (SD 12).

‎View this figure

‎View this figure

‎View this figure

‎View this figure

‎View this figure

‎View this figure

‎View this figure

‎View this figure

A total of 3 studies might have been included but on detailed consideration have been excluded. Parsons et al [36] reported on a human avatar that is portrayed as a physician and conducts a neuropsychological assessment. However, this was not explicitly aimed at older adults with memory problems, and the verbal interaction description lacked details to judge the dialogue form. In all, 2 studies reported the recording of human utterances in response to an avatar interview question, which were subsequently analyzed offline on the prevalence of cognitive impairment [37,38]. Therefore, they were not considered real dialogue.

Study Quality Evaluation

The results of the quality assessment are presented in Table 2. The average quality score was 0.69 (SD 0.08).

Discussion

Principal Findings

Despite the fact that IVAs have been a topic of study since 1998, the number of studies that actually show a 2-way verbal interaction with older adults with amnesia is relatively low, and only 8 studies were found. One of the main bottlenecks is the quality of the speech-to-text function. For example, Sidner et al [39] described this function as a technical challenge, and therefore chose touch buttons on a screen as human input means to the virtual agent. Several other studies followed the same approach and were not included in this review.

In the studies found, it was particularly difficult to assess participants’ perception of verbal interaction. In 4 studies, the user could only give yes or no or a few other short commands [24,33,34,40]. The user verbal command options in the studies by Tokunaga et al [19,32] were unclear. Only the studies by Ali et al [30] and Razavi et al [31] report a short (3-5 turns) open dialogue on a range of topics such as weather, pets, retirement, life goals, growing older, and spirituality. A total of 9 studies also provided little information on their NLP pipeline. Ali et al [30] and Razavi et al [31] designed a pattern-matching solution based on gist-clauses, which are a combination of the IVA question and the answer received. Wargnier et al [24] and Tsiourti et al [33] used Microsoft speech recognition components for speech recognition, and the utterance text strings were matched with predefined commands. Oliveira et al [35] used an in-house developed speech recognition subsystem. In addition to this information, a few other details are provided that allow us to compare the advantages and disadvantages. The other 3 studies did not provide any information regarding the NLP pipeline. However, all 8 studies reported a generally positive attitude of the participants toward the agent. Video recordings would have been helpful in assessing the details of human-agent interaction.

The main qualitative outcome reported in these studies was the usability of the system. This usability was measured differently between the studies, and the methods and data are provided in Table 3. Ali et al [30] and Jegundo et al [34] used the well-known System Usability Scale [40]. Wargnier et al [24] and Tsiourti et al [33] did not use existing usability questionnaires from the literature, such as the System Usability Scale, but created new usability questionnaires. The studies by Tokunaga et al [19,32] were more oriented toward the functional performance of the agent, whereas Oliveira et al [35] did not report clear usability measurement methods and results. A comparison between the usabilities is difficult to make because of the difference in the methods and scales used. The difference in scales also does not allow us to calculate an aggregated mean value of the usability for the studies combined. Furthermore, the reason for the relatively low number of participants (µ=20) is probably the explorative or feasibility assessment character of the studies found. A system usability study should preferably include between 20 and 30 participants [40]. Our hypothesis that IVAs designed to assist older adults with memory problems have a positive usability, given the data from the table, indicates that there is reason to believe that it is true.

Other outcome data reported were efficacy of nonverbal improvement (1/8, 13%), efficacy of verbal improvement (1/8, 13%), conversation quality (1/8, 13%), feasibility of interaction (2/8, 25%), observer usability assessment (1/8, 13%), critical incident registration (1/8, 13%), and speech recognition quality (2/8, 25%). The reported values were difficult to combine at an aggregated level and to make them result in a general recommendation.

All agents found in this review were female characters, except for one. Studies from Western countries show white-skinned agents with varying hair colors and ages and a realistic look, whereas studies from Japan feature an anime-influenced character with a Japanese female look. These papers provide little information on facial muscle motions and lip synchronization when talking. Of the 8 studies, 5 (63%) showed mainly the head and shoulders of the agent, whereas the remaining 3 (37%) studies showed almost the complete agent body.

The duration of the interactions between the participant and the agent was relatively short, with a few turns per topic. These studies provide little information on the exact duration and how and when the interaction was stopped. These studies were concentrated in the United States (2/8, 25%), Europe (4/8, 50%), and Japan (2/8, 25%). No studies from China or Korea were found; however, this may be because of the selection of papers in the English language.

Reports on the cognitive status of the participants vary in the 8 studies. For 5 (63%) of the 8 studies, no related details were given, 1 (13%) study included participants with self-reported mild difficulties, 1 (13%) study included participants diagnosed with mild cognitive impairment, and 1 (13%) study conducted a Mini Mental State Examination among the participants. The latter study reported an average Mini Mental State Examination score of 22.9, consistent with mild dementia and a relevant target group for an intervention with a virtual agent.

According to our assessment, the quality of the studies varied between 0.58 and 0.83. Kmet et al [28] did not specify an absolute value for these outcomes, for example, in the sense that papers could be classified as having a good or bad methodological quality based on that value. Nevertheless, the overview in the study quality evaluation section of this paper enables comparison between the included studies. Although the study objectives and design were generally clear, only 1 (13%) of the 8 studies included a random allocation of participants to separate conditions [30]. The average number of participants in the studies (n=18) was rather low, and most articles seemed to focus more on the technical development of the system than on a thorough user evaluation. Moreover, verbal interactions were generally short. Most studies also lacked control of confounding parameters.

The results of this study, in terms of the number and scope of the studies found, were compared with the findings of relevant reviews mentioned in the Introduction section of this paper. The observation from Car et al [16] that there is a predominance of text-based conversational agents, with only a few apps using speech as the main mode of communication, remains valid. Although speech is considered a comfortable interaction modality for older adults, the difficulty of realizing free speech-based interaction with an agent is still present.

Xie et al [12] called for more systematic designs and evaluations of AI systems, and this is supported by the results herein showing a limited number of experimental studies targeting older adults with memory problems. Schachner et al [13] also found that the number of studies is scarce and mostly quasiexperimental, and Bevilacqua et al [15] concluded that the number of studies should increase.

For further research, it would be useful to evaluate IVAs, as shown in Figure 1, targeting people with other health conditions. For example, Laranjo et al [17] found 2 studies, one for military personnel with PTSS and one on training for people with autism spectrum disorders. Provoost et al [20] reviewed embodied conversation agents in clinical psychology, targeting individuals with autism, depression, anxiety, PTSS, schizophrenia, and substance abuse. Milne-Ives [22] discovered virtual agents for alcohol counseling, depression, and suicide prevention.

Regarding the design features of the virtual agent, Loveys et al [21] provided input for the design process by evaluating requirements for language use, behavior, emotional expression, embodiment, appearance, and personality. These findings may be considered when developing future virtual agents for people with amnesia.

Limitations

This study has some limitations. Although the authors made an effort to screen the titles, abstracts, and papers carefully and applied a snowball method to identify additional studies by checking the references in selected papers, we do not exclude the possibility that some studies were overlooked. Second, in many studies, the actual implementation of the dialogue and the information exchange was difficult to assess. Third, many studies describe only the requirements or designs of virtual agents but provide no or very little information on the experiments conducted. Such studies were excluded, but we cannot rule out that, by doing so, relevant studies were missed. Fourth, memory problems is used in this paper as an umbrella term for the more formal terms dementia, Alzheimer, amnesia, or mild cognitive impairment, but was not used as an explicit search term, and this may have caused that a study was overlooked.

Conclusions

Few studies have described actual experiments with IVAs in dialogue with older adults with memory problems. The dialogue contents are quite simple and superficial, especially on part of the participants, and often limited to only yes or no. More research is needed to develop real, useful, and prolonged dialogue between virtual agents and older adults. Another conclusion is that more research into the effectiveness of IVAs is needed, for example, through randomized controlled trials.

The reporting on the human-agent interaction characteristics often lacks many details, such as the exact contents of the dialogues, the starting and ending of the dialogue, and the graphical features of the avatar (static and dynamic). This makes it difficult to compare the experiments and to assess the status of the applied technology. A more standardized approach toward reporting human-agent interaction characteristics would be helpful for future research. Audio and video recordings of such interactions would provide even more information that will benefit the research community.