We implemented a multistep UCD process to design and evaluate SAR-VR activities that would encourage HHI mediated through human-computer interaction (HCI; nonimmersive VR) and human-robot interaction (HRI). Three overarching UCD principles guided the process throughout: (1) focus on users and tasks, (2) measure usability empirically, and (3) design and test usability iteratively [40,41]. Figure 1 displays the UCD timeline. There were distinct phases and activities to the UCD study process that are detailed below. These included problem analysis, prototype design development and evaluation, and field testing and validation.

Participants consisted of older adults residing in an assisted living facility and staff. Eligibility criteria for older adults were age 65 years and older, at least 3 months of residence at the facility, the ability to hear, speak, and understand English, and the ability to sit comfortably and move both arms. Staff criteria included involvement directly or indirectly with the activities offered to residents within the setting and willingness to work with the researchers on the SAR-VR system. Investigators contacted several assisted living facilities by phone and email. Facilities needed to be within an hour’s drive of the Vanderbilt University School of Engineering. Using institutional review board (IRB)–approved script, we explained the study purpose and our needs to conduct the study, that is, physical space adequate for the robot and nonimmersive VR setup, physical space conducive to the activity and evaluation (eg, adequate lighting and minimal noise or interruptions), and willingness to allow staff to interact with the research staff. Upon receiving permission from site administrators, Vanderbilt University engineers spoke to potentially interested residents at a resident meeting and to staff at a staff meeting, demonstrated the robot, and provided information regarding the study needs. Interested residents and staff contacted the research team directly, and a meeting was arranged to determine eligibility and conduct the informed consent process.

The first step was problem analysis, that is, to specify the context of use of SAR-VR activities with pairs of frail older adults. First, our interdisciplinary team brought expertise to the area of inquiry, including engineers from various disciplines (ie, mechanical engineering, electrical engineering, and computer science) as well as nurses and physicians who specialized in geriatrics and LTC. Based on our areas of expertise, we searched PubMed to (1) identify aspects of activities that effectively engage older LTC adults, especially those who have apathy, (2) SAR characteristics acceptable to older adults, and (3) nonimmersive VR factors suitable to older adults.

Results from our literature review were discussed with an interdisciplinary advisory panel (n=8), consisting of LTC activity directors, nurses, managers, and a doctoral-prepared occupational therapist who specialized in geriatrics. Based on the literature review and conversations with the advisory panel and older adults, inspiration for activities was based on real-life activities the older adults performed at their LTCs and hobbies they enjoyed. Criteria for activities included (1) flexibility to accommodate older adults with varying levels of physical and cognitive abilities, (2) varying levels of collaboration required between 2 older adults, and (3) sufficient variation of activities to minimize monotony. Lastly, the system must have a user-friendly graphical user interface so that LTC staff can easily operate it and older adults can manipulate it. We conducted multiple design iterations and internal testing of prototype activities in the Vanderbilt University Engineering Laboratory. We presented ongoing work over several meetings with the advisory panel. The 2 LTC residents reviewed and approved the initial prototype activities for field testing via teleconferencing (due to COVID-19 isolation protocols).

This phase focused on the choice of robots, software, and hardware development, and the choice of activities for each robot type.

Prior to the first session, baseline information was gathered from each older adult, consisting of demographic information (age, sex, education level), frequency of use of technology (smartphones, computers, pads), the Apathy Evaluation Scale-Self Rated (AES-S), and the Self-Administered Gerocognitive Exam (SAGE). The AES-S [57] consists of 18 items that measure behavioral, cognitive, and emotional domains of apathy. Each item is rated on a 4-point scale ranging from 1 (not at all) to 4 (a lot), with a total score ranging from 18 (low apathy) to 72 (extreme apathy). Although there are no standard cutoff scores to indicate clinical apathy, a score of 40-42 on the AES-S generally indicates mild apathy in older populations. The AES-S has good internal consistency (Cronbach α=.86-.94), retest reliability (r=0.89-0.94), and convergent and divergent validity.

The SAGE [58] assesses cognition in 5 domains—language, reasoning and computation, visuospatial abilities, executive memory, and orientation. The SAGE has good interrater reliability (intraclass correlation coefficient=0.96), specificity (88%-95%), and sensitivity (62%-95%). A score of 17-22 (maximum) suggests normal cognition, 15-16 suggests mild cognitive impairment, and <15 suggests dementia.

The same study procedure was followed at all sessions at both sites. Figure 4 shows a setup, materials, and a session in progress. Each session was attended by 3-4 researchers. Participants sat in front of and facing the system. E4 physiological sensors (Empatica Inc) were placed on both participants’ nondominant wrists. Calibration data and a 2-minute resting physiological signal were recorded for each participant. Each participant wore a clip-on microphone to record their speech. NAO was positioned in front and to the side of a large television screen. The Kinect was placed facing the 2 participants. An administrator operated the study workstation placed to the side of the participants. A total of 2 webcams were used to record the participants’ behaviors and interactions, as well as the SAR and VR environment (Multimedia Appendix 3).

Each participant participated in 5 separate sessions lasting 30-45 minutes. Each session encompassed 2 activities with a 5- to 10-minute break between activities. At the start of each activity, participants completed an orientation where they were informed of the activity and objective and how to use the wands to map their movements onto the VR environment, and provided the opportunity to practice the basic motions needed for the activity. Most sessions were completed with an older adult pair; in the event of scheduling conflicts with older adults, some sessions were completed with one older adult and a research partner. Between each session, engineers modified activities and the SAR-VR system.

Once the field testing was completed with the older adults, the final SAR-VR system was tested with LTC staff to determine whether they could administer the system. A total of 5 staff from Site 2 (mean age 34.2, SD 15.8 years) were recruited and consented to the study. Components of the system were set out; all components were labeled and color-coded. Staff were provided a written manual with a step-by-step guide in which components were illustrated (Multimedia Appendix 3). In addition, every section had a video guide. Staff participants were asked to set up and run the system based on the instructions. After each session, the system setup and interface were refined based on objective and subjective feedback.

Approval for this study was obtained from the IRBs of the research team at The Ohio State University (IRB #2020B0342) and Vanderbilt University (IRB #210386). The procedures used in this study adhere to the tenets of the Declaration of Helsinki [59] and regulations of the IRBs. The 2 types of participants were recruited for this study—older adults residing in LTC and employed staff. Safeguards were in place for both groups to ensure participation was voluntary. Older adults were informed that their decision whether to participate would have no bearing on the care received at the LTC facility, and staff were informed that their decision regarding participation had no bearing on their employment. Written informed consent was obtained from all individuals participating in the study prior to data collection and study procedures. To ensure older adults understood the study purpose, risks, and benefits, we used the University of San Diego Brief Protection of Human Subjects Capacity to Consent instrument [60]. If a participant could not demonstrate understanding, they were ineligible. Research assistants underwent training to recognize signs of frustration, anxiety, or stress displayed by participants; any sign of discomfort resulted in termination of the session. Research assistants were trained in procedures to protect privacy and confidentiality. Participants were assigned unique study identification numbers, and all deidentified data were stored in a secure database and password-protected. Participants received financial compensation for study activities, including the interactive sessions and data collection procedures, to acknowledge their time and reduce attrition. An external safety officer, designated by the National Institute on Aging (NIA), reviewed and approved the study protocol and informed consent documents prior to the start of the study. Biannual reports were provided to both the safety officer and the NIA program officer to review progress and any untoward events.

A total of 12 older adults participated (Table 2). In the first field study, 6 older adults initially consented to participate. A 3-week delay occurred between consent and implementation due to a COVID-19 outbreak in the facility. Subsequently, 2 participants chose not to continue with the study, having lost interest over time; both these participants had significant cognitive impairment. The 4 remaining Site 1 participants (mean age 85, SD 9.3 years; mean AES-S score 29.3, SD 5; mean SAGE score 11.5, SD 3.6) completed all sessions. In Field Testing 2, eight residents (mean age 80, SD 4.7 years; mean AES-S score 26.35, SD 2.7; mean SAGE score 14.3, SD 4) were eligible, consented, and participated. Of the total sample, 4 participants had no experience with technology. Self-reported apathy scores were below the 40-cutoff point. Cognitive screening revealed that 3 (75%) of the 4 older adults at Site 1 had scores indicating dementia, whereas only 3 (38%) of 8 older adults at Site 2 had scores indicating dementia.

Attention to the environment was necessary. At the first site, older adults had difficulty seeing the monitor because of glare; subsequently, blinds needed to be drawn at all sessions. The voice of NAO was difficult for older adults to hear and understand clearly. After several trials of altering its pitch, style, volume, and speed, we found a setting that was the most clearly understood (Multimedia Appendix 1). Early on, we observed that older adults became fatigued with holding and manipulating the wands. They suggested having chairs with armrests and sitting at a table would provide them with the opportunity to rest their arms. The VR environment underwent a number of changes to enhance the visualization, including adjustments to the size, color, and placement of components.

Technology issues occurred in both the hardware and software components. Originally, NAO was programmed to do a short dance to celebrate the successful completion of an activity. After one of the NAO fell off the table and broke its hip, we elected to forgo that component despite the older adults’ obvious enjoyment. NAO would give feedback based on the queuing of each observation of the older adults’ performance, either positive or corrective. This resulted in a long list of feedback statements, ultimately causing a delayed response that did not correspond to the older adult’s current performance. The mismatch of feedback with current behavior caused confusion and, at times, frustration. NAO was reprogrammed to provide feedback only after an older adult performed an errant behavior 3 times in a row. The customized wands were remodeled several times to enhance grip and controls; wands were also color-coded to match the person’s cursor color in the VR environment.

Throughout the study, older adults provided suggestions to enhance the activities to make them more enjoyable. Table 3 highlights major changes that occurred during Field Testing 1 and 2.

After each session, participants at both sites rated their degree of comfort and confidence in (1) using the wands, (2) interacting with the robot, and (3) interacting with the VR environment. Between the first and final sessions, participants at Site 1 increased their comfort and confidence in 5 of the 6 categories. The average increase across all 6 categories was 0.58 (95% CI 0.04-1.12; Cohen d=0.45; Wilcoxon signed rank test P=.03). The category-wise mean improvements and standard deviation for each item are displayed graphically in Figure 5 (see Multimedia Appendix 4 for numerical values).

Participants at Site 2 showed enhanced comfort and confidence with an average increase of 0.71 (95% CI 0.43-0.99; Cohen d=0.73; Wilcoxon signed rank test P<.001) between the first and final sessions. The category-wise mean improvements and standard deviations are shown in Figure 6 (see Multimedia Appendix 4 for numerical values).

Analysis of the recorded videos revealed that the average interaction of older adults with their partner increased steadily across sessions, with iterations from 0.16 interactions per minute during the first set of sessions to 1.14 interactions per minute in the final set of sessions. The average number of robot interventions decreased from 1.10 per minute to 0.73 per minute. The average number of researcher-assisted interventions decreased from 0.34 per minute to 0.10 per minute. These results suggest an increased engagement and interaction among older adults and an increasing effectiveness of the system with each iteration.

Participants were asked open-ended questions (Multimedia Appendix 4) about their experience with the activities. Participants reported they enjoyed the activities and expressed interest in playing them again. They described the games as fun and interesting, noting that once they became familiar with the mechanics (eg, casting the fishing rod), the experience was enjoyable and engaging. They also indicated that the robot’s instructions were clear and sufficient, requiring no additional guidance.

A total of 5 staff members evaluated the system from the perspective of ease of use and convenience. They set up and ran the system following written instructions prepared by the researchers. The first 2 participants who worked with the system had difficulty following the steps in the correct order and required prompting from the research staff. To remedy this, the instruction manual was divided into 3 parts, that is, presession, during-session, and postsession. The presession consisted of initialization and calibration of components; during-session procedures involved running the system, activity selection, and changing the activity difficulty levels; and postsession procedures consisted of securely closing all applications and sensors, saving the data, and turning off the system. This division helped avoid confusion and led to a smoother setup by other participants. All the staff participants commented that the first-time setup had a learning curve, but they would feel more confident if they were to set up the system again. Overall, they found the integration of the different components of the system to be satisfactory and user-friendly.

Multimodal physiological and behavioral data were collected during the sessions to inform future research and system development. Physiological signals, including blood volume pulse, electrodermal activity, skin temperature, and accelerometer data, were recorded using the E4 sensor. These data can be analyzed to infer participants’ affective states (eg, stressed vs relaxed, engaged vs disengaged) by examining the fluctuations in their autonomic responses during interaction with the system. Such insights can guide the design of adaptive features that adjust task difficulty, robot behavior, or feedback strategies in real time to better match the user’s emotional and cognitive state.

Joint position and head orientation data were captured using the Kinect sensor. These data enable the quantification of motor performance (eg, range of motion, smoothness of movement) and the assessment of social engagement (eg, frequency of head turns toward the partner or robot, use of gestures, changes in posture). This allows for objective evaluation of both physical rehabilitation potential and the quality of social interaction.

Finally, time-stamped performance logs provide a record of task success, errors, and completion times, which can be correlated with physiological and kinematic measures. Together, these multimodal datasets will allow for richer models of user engagement, support longitudinal monitoring of progress, and inform further refinement of SAR-VR interventions for older adults.

In this user-centered design study, we used the literature, our clinical expertise and prior research, and an advisory panel that included older adults to identify activities that could be created for older adults residing in LTC settings using a combined SAR and nonimmersive VR system (SAR-VR). The SAR-VR system was designed to engage pairs of older adults with the long-term goal of using this technology in LTC settings for those with apathy. The underlying premise is that those with apathy are best treated by activities that are performed with others and that combine cognitive and physical aspects [11,12]. We conducted numerous sessions with LTC older adults, who had varying levels of cognitive and physical impairments, to achieve the development and validation of 4 SAR-VR activities for this specific population. We are currently testing the effectiveness of the SAR-VR system to mitigate apathy in a multisite randomized controlled trial.

Although others [61,62] have highlighted the importance of UCD with older adults and adapting processes for this population, few studies have actually involved older adults with cognitive impairment in the design of technology [34,39]. Second, it is important to involve staff in UCD such that they feel comfortable operating new systems and thus ensure system scalability in LTC settings. Thus, we focused on the 2 key stakeholders, LTC older adults and LTC staff, in our UCD of the SAR-VR system and activities. Our findings demonstrated the effectiveness of an iterative UCD approach for designing SAR-VR activities and systems for older adults in LTC settings.

The study provided valuable lessons about conducting UCD of SAR-VR systems in LTC settings. First, the timeline for development and testing was prolonged due to multiple waves of COVID-19, virtually shutting down access to the older adults. Second, as is often the case in the design of technology, our initial design decisions were based on our expertise and experience, the literature, and experts in the LTC field. Some of these decisions were discovered to be incorrect during field testing. For example, we designed the graphics to be seen easily by the older adults (contrasting colors, font size) and lowered the pitch of NAO’s voice to accommodate age-related vision and hearing changes [63]. However, we found our graphics required further modifications, and a series of voices had to be tested before older adults agreed to the pitch, tone, and speed. The initial creation of the handheld wand controller also required several changes to accommodate older adults’ grips, maneuverability, and vision. Third, accessing older adults in the LTC sites proved more challenging than anticipated. It was not uncommon for staff to cancel and reschedule our sessions based on their availability, competing demands, or illness among the older adults—all of which further prolonged the process.

Despite these challenges, we found the UCD process successful in developing the SAR-VR system to deliver multimodal activities for older adults with cognitive and physical impairments. With the increasing population of older adults with dementia and insufficient skilled care staff, technological interventions are a promising strategy in the delivery of care processes. We succeeded in developing an innovative SAR-VR system that could deliver a variety of activities that combined physical, cognitive, and social domains for pairs of older adults residing in LTCs. Such multidomain stimuli, along with HHI, have been shown to be effective in reducing apathy [9,11]. Each activity required the participants to cooperate and make joint decisions, thus encouraging interaction and conversations. While other work has attempted to reduce apathy using immersive VR technology [64,65], research on developing technology to foster and mediate natural HHI among older adults with minimal caregiver intervention is scarce. Out of the total 14 participants recruited for the study, 12 stayed for the full duration of the study. Participants’ confidence and comfort with the SAR-VR improved over time for both those who conducted the early development work (Field Testing 1) and the validation work (Field Testing 2); overall, the participants enjoyed doing the activities.

Since the system is designed to be deployed at LTCs, we tested the setup procedure of the system with LTC staff to ensure ease of use. All enrolled staff could set up and run the system with the instructions provided without any special training, thus proving the practicality and deployable nature of the system in an LTC. When asked if they would be more comfortable if they were to set up the system again, most replied in the affirmative and mentioned that the most challenging part was identifying components that they were otherwise unfamiliar with. The iterative design process allowed us to make modifications to the various components of the system, including the setup guide, until the system was suitable and enjoyable to both target populations, the older adults and the LTC staff.

While UCD has been used to develop immersive VR [39], eHealth applications [66,67], exercise games [68-70], and websites [71,72] for older adults, to the best of our knowledge, this is the first study to take an iterative UCD approach to developing a SAR-VR system and activities for pairs of older adults. The resultant modular architecture and activities set the foundation for investigating their effectiveness in mitigating apathy among older adults with dementia. We designed the system to easily be adapted to other robots, activities, and interaction devices. The intensity of the physical, cognitive, and social components of the activities can be individually tuned, as desired, to study the effect of each component.

Physiological, motion, and performance data collected from the study will provide insights for further analysis and refinement. Physiological data can provide insights into the level of stress and cognitive load of the participants during performance of the activities, which can later be used to design an affective state–based closed-loop system to automatically regulate the difficulty of the activities. The joint position data from the Kinect sensor can later help automate the detection of nonverbal communication by tracking head pose and gestures.

This work, although promising, is not without limitations. First, it is limited by its small sample size and purposive sampling, as well as the short duration of the study procedures, that is, 5 sessions. Small sample sizes are typical in initial UCD studies given the need to facilitate rapid iterative testing, as well as keep personnel and financial resources within reasonable limits [38]. A larger study of longer duration is required to gauge user perception accurately and mitigate the novelty factor. Our target population is older adults who reside in LTC, with apathy, and with cognitive or physical impairments; our sample did not have evidence of clinically significant apathy, and several were screened with normal cognition. The user perception of the system depends on several factors, such as cognitive abilities, previous exposure, and familiarity with similar technology, as well as personal preferences toward particular activities. A larger sample size is required to generalize the preferences, as data from smaller samples are easily skewed by outlier individuals. A larger sample size would also allow for the identification of personal characteristics associated with successful and ongoing involvement with the SAR-VR system. Second, we used commercially available robots, thus limited by the physical attributes of the robots. For example, we found that we needed multiple adjustments to the humanoid robot’s voice that would facilitate older adults’ ability to hear and understand its language. We had to position the humanoid robot onto a table to facilitate the visual field of the older adult who was focusing on the virtual environment displayed on a television or monitor. Nevertheless, we chose robots that have been available for many years to researchers because of their ease and versatility for customized programming. Lastly, we created only 4 activities during this study. We have since designed several more activities based on our experiences with this study that will allow for more variety in future studies.

Building upon our previous work on SARs and VR, this work details the use of user-centered iterative design techniques and processes to develop an innovative SAR-VR system that can provide activities that consist of physical, cognitive, and social aspects and are specifically designed to promote HHI through activities that require cooperation between participants. We created 4 activities—3 humanoid robot–based activities, where the humanoid robot NAO acted as the coach to provide hints and corrective feedback and mediate the interaction between older adults, and 1 animal robot–based activity, where an avatar in the VR environment provided the feedback and the puppy robot Aibo performed tricks as rewards to incentivize the older adults to complete the activity. A finite state machine was developed to provide appropriate real-time feedback in response to participants’ performance in the activities. The system also collected video, physiology, joint position, and performance data that will be used for future investigations into how stress and engagement patterns in older adults varied with activity difficulty level. We incorporated various tunable difficulty levels in the activities to accommodate different physical and cognitive abilities. A variety of activities using both the humanoid and animal robots were developed to enhance engagement and reduce monotony. The system has a simple user interface and an easy-to-follow setup guide for the LTC staff. Currently, we are conducting a multisite randomized controlled trial [73] to compare the effect of usual care to the SAR-VR plus usual care on reducing apathy (primary outcome) among 188 older adults with cognitive impairment and enhancing executive cognitive function (secondary outcome). Eligibility of state-licensed LTCs includes an hour’s drive from the principal investigators’ offices, certification by the US Centers for Medicare and Medicaid, and physical space to store the research equipment. Eligible older adults are those aged 65 years or older, with 3 months or more of residence at the LTC, evidence of mild to moderate cognitive impairment, and symptoms of apathy. The intervention takes place twice weekly for 8 weeks; 4 weeks participating in NAO activities and 4 weeks participating in Aibo activities. Data are collected at baseline, each session, and post experiment. Findings will aid in determining the effectiveness of the SAR-VR system and identifying characteristics of the system, activities, and older adults that are associated with effectiveness. We are also gathering information to identify barriers and facilitators to implementation; this knowledge is needed to address the scalability of the intervention, including the training needs of staff.