The study protocol was approved by the institutional ethics committee of the National Center for Geriatrics and Gerontology (1539) and prospectively registered in the UMIN Clinical Trial Registry (UMIN000046256). All participants with a Clinical Dementia Rating (CDR) scale [29] score of 0 or 1 provided informed consent themselves in accordance with the Declaration of Helsinki. For those who were considered to have an insufficient capacity to consent due to cognitive decline equivalent to CDR 2 and 3, informed consent was obtained from their family members, and the procedures were explained to the participants in plain language to obtain their approval. To comply with ethical principles, all data collected were anonymized and stored in a locked file or on a password-protected computer.

This pilot study was conducted as a single-arm, self-controlled, interventional study. Two local nursing care facilities that had no previous experience implementing socially assistive robots were selected as the experiment sites.

Participants were recruited among the residents of the 2 nursing homes. The inclusion criteria were as follows: a significant decline in cognitive function interfering with independence in the performance of everyday activities, the ability to maintain a sitting position for 15 minutes or more, the ability to communicate using simple words, and the ability to follow 2-step instructions. These criteria were first assessed by nursing home staff members through assessments performed as part of daily nursing care procedures. Regarding cognitive decline, candidates either had a previous formal diagnosis of dementia from their physician or received a diagnosis from one of the researchers (EO, a physician); they also had confirmed evidence that cognitive decline was present and that the decline was not due to delirium or other mental disorders. Ultimately, all the participants met the diagnostic criteria for dementia in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5). Those with unstable physical or mental conditions or evident higher cognitive dysfunction due to causes other than dementia were excluded.

The participants were taken to a private room or a place with minimum environmental noise, and their faces were recorded in a resting state for 30 seconds. This was referenced as the control image for calibrating the facial expression analysis (to be described below). Next, the socially assistive robots were presented to the participant by a familiar staff member in 1 session using a predetermined 3-step procedure. The staff explained in advance that they wanted the participants to share how they felt after experiencing the robots. In the first step, the robot was placed on a desk in front of the participant (visual stimulus). In the second step, the robot was manipulated to produce a gentle voice or meow (visual and auditory stimuli). In the third step, the participant was encouraged to touch the robot (visual, auditory, and tactile stimuli) and was able to handle it freely, including petting and holding. The 3 patterns of sensory stimulus produced by the robots were presented for approximately 30 seconds in the context of assessing the participant’s immediate responses unless the participant refused (Figure 1).

Each participant was presented with 3 types of commercially available robot with typical characteristics—a humanoid-type robot capable of voice communication (RoBoHoN; Sharp Corp), a doll-type robot with the appearance of a stuffed toy and a voice recognition and reproduction system (Chapit; RayTron Inc), and an animal-type (cat-shaped) robot that can meow, move its tail, and recognize sound (Amaenbou-Nekochan; Digirect Co, Ltd)—resulting in 3 sessions per participant. The order of presentation, which was determined in advance using a random number table, differed for each participant (Figure 2).

The averages of the emotions expressed in response to the robot (ie, happy, sad, angry, surprised, scared, and disgusted) were compared to each other using the nonparametric Wilcoxon signed-rank test, with the P values multiplied by the number of tests according to the Bonferroni method. The correlations between subjective and objective emotional assessments were also examined using Spearman correlation coefficients. The average expression intensity values during the 30 seconds for each sensory stimulus obtained by facial expression analysis were compared for all 3 patterns (visual only; visual and auditory; or visual, auditory, and tactile) using the Friedman test with the Wilcoxon signed-rank test as a post hoc test. The average of the 3 robot presentation sessions was used in this part of the analysis. Additionally, focusing on positive emotions, the expression intensity values for “happy” were compared between the different types of robots using the Friedman test with the Wilcoxon signed-rank test as a post hoc test. Statistical analyses were performed using STATA/SE (version 13.1; StataCorp). Any P value less than .05 was considered statistically significant.

This study aimed to quantitatively evaluate the psychological and emotional responses of people with dementia to complex sensory stimuli provided by socially assistive robots. To summarize, the values for “happy” were found to be the largest in the overall response to the robots among the emotional elements in both the self-recorded assessment and objective facial expression analysis. However, correlations between the subjective and objective emotional assessments were found to be relatively low or insignificant. When comparing the 3 different patterns of sensory stimulation, there was a significant increase in some of the expression intensity values when all stimuli (visual, auditory, and tactile) were presented. In the comparison between different types of robots, focusing on “happy,” there was a significant increase when all stimuli were presented by a doll-type robot and by an animal-type robot but not by a humanoid-type robot.

The emotional responses revealed by the objective facial expression analysis showed the same trend as the participants’ subjective assessment, with happy emotions having the largest values in both assessments. This result suggests that the facial expression analysis system can successfully detect the expressions that emerge in people with dementia, which is consistent with a previous study [40].

However, considering that the facial expression intensity was detected as relatively high in some cases where the subjective rating was moderate, the facial expression analysis may be able to capture emotions that are not sufficiently represented by a subjective 5-point Likert scale. In fact, a previous study supports the appropriateness of facial expression analysis as a nonverbal pain assessment for people with dementia when they lack the ability to self-report [41]. Therefore, facial expression analysis may also be useful as an accurate nonverbal assessment of emotions for people with dementia.

Although some studies have already demonstrated positive responses in people with dementia when presented with socially assistive robots [42,43], this study is the first to objectively and quantitatively describe emotional responses using facial expression analysis. Additionally, these results are inconsistent with those of Goda et al [19], who found that a 5-minute talk session with a socially assistive robot caused stress in people with dementia. In contrast to this study, the socially assistive robot’s interaction in the study by Goda et al [19] was mainly through verbal communication; thus, it was inferred that these stimuli were burdensome for people with dementia, who tend to have a decline in verbal communication skills. The positive expressions shown in this study are considered to have been observed as responses to sensory stimuli and not verbal stimuli.

Regarding the relationship between the type of sensory stimulus and emotional responses, the expression intensity values for “happy,” “sad,” and “disgusted” significantly increased with increasing sensory stimuli. Particularly, happy emotions were revealed to be elicited most strongly in persons with dementia when different varieties of sensory stimuli were presented. With respect to the values for “sad” and “disgusted,” we consider it natural that participants became somewhat careful or timid toward unknown experiences when interacting closely with robots. However, these emotional elements in automated analysis should be interpreted with caution, as they can occasionally include other negative emotions, such as fear [44], especially when accompanied by facial movements around the lips or jaws [31]. Nevertheless, the values for these negative emotions remain small compared to the value for “happy,” showing that positive emotions were dominant when visual, auditory, and tactile sensory stimuli were offered.

Notably, this increase in positive emotional responses may include changes over time, because the sensory stimuli were added over time in this study. Given the procedures in this study, the results could have been influenced by the novelty effect [20], which is derived from curiosity toward a new experience. However, the intensity of each emotional element did not change linearly when sensory stimuli were added over time. Moreover, the addition of an auditory stimulus did not significantly increase the expression intensity value, whereas the addition of a tactile stimulus did. These findings suggest the importance of adding tactile stimuli to visual and auditory stimuli. This is plausible considering that tactile information connects through several pathways to the insular cortex, which evokes emotional responses [45,46]. The finding that tactile stimuli evoked positive feelings is supported by previous studies that showed that haptic or tangible input was effective in helping people with dementia understand and adapt to their surroundings [47,48]. Regarding auditory stimuli, the results of the Questionnaire on Hearing did not indicate that the participants had very good hearing, suggesting that the effects of auditory stimuli on emotion elicitation may have been relatively small. Since hearing loss is associated with the risk of developing dementia [49], and a high percentage of people with dementia actually have hearing loss [50,51], this issue may also be of great clinical importance.

Furthermore, it is noteworthy that a significant increase in expression intensity values for “happy” with increased sensory stimuli was observed for doll-type robots and animal-type robots, but not for humanoid-type robots. One of the distinctive characteristics of the doll-type and animal robots used in this study was that they were covered by soft, fur-like materials. The importance of soft materials is commonly discussed in the field of soft robotics for medical use or human assistance [52,53]. Softness is considered effective not only in terms of safety for the human body but also in terms of the imitation of reality or the creation of familiarity [54] and emotional processing [55]. However, the emotional effects of various tactile sensations in people with dementia have not yet been studied; consequently, given the findings of this study, the effectiveness of soft tactile stimuli in dementia care may be worth exploring in future research.

This study has a few limitations. First, it included a small number of participants and a single experimental group. Changes in facial expressions were reliably detected by using the participants’ resting states as controls. However, another study design, such as a randomized controlled trial, is required to confirm these effects more clearly. Second, the generalizability of our findings may be limited, as most of the participants in this study were female. Previous studies report that there are gender differences in emotional responses to some types of sounds [56], emotion expression [57], and emotion regulation [58], although gender and facial expression have been reported to have no significant correlation [40]. Further research with male participants will be needed to reveal possible gender differences in responses toward social robots. Third, since there is no prior literature that has identified a minimum detectable change or a minimal clinically important difference for expression intensity, the clinical significance of the changes in expression intensity values demonstrated in this study needs to be explored further. Finally, this study only investigated the immediate responses to socially assistive robots, with patients allowed to interact with each robot for only 1.5 minutes in total. However, in real clinical settings, people with dementia might express more diverse patterns of responses, using these robots as they would like. Moreover, any enthusiasm resulting from the novelty effect may diminish over time. Thus, further investigation is required to reveal the long-term emotional effects of socially assistive robots on people with dementia, including variability in positive responses over several hours or days of use and the effects of these robots on their neuropsychiatric symptoms.

This study quantitatively examined the emotional reactions of people with dementia to socially assistive robots. The expression intensity values, especially the values for “happy,” significantly increased with multiple sensory stimuli, including visual, auditory, and tactile stimuli. Therefore, this study shows that socially assistive robots may be more effective in arousing positive emotions when multiple sensory stimuli are involved. Further studies, including randomized controlled trials, are required to further explore the effectiveness of and the optimal methods for using socially assistive robots in dementia care.