Participants were recruited from November 2020 to August 2021 in the United Kingdom (London, Guildford, Plymouth, and Birmingham) and the United States (Santa Ana, California). Research participants were enrolled if they were CU or diagnosed with MCI in the previous 5 years. In the UK study, participants diagnosed with mild AD in the last 5 years were also included. MCI due to AD and mild AD diagnoses were made according to the National Institute on Aging–Alzheimer’s Association core clinical criteria [24]. Participants were approached if they had undergone a prior amyloid beta positron emission tomography scan or a cerebrospinal fluid test (confirmed amyloid beta negative within 30 months or amyloid beta positive within 60 months). Eligibility was established by screening via video calls using a secure Zoom (Zoom Video Inc) link (UK study) or in-clinic assessment (US study), during which the Mini-Mental State Examination (MMSE) [25] was administered. For remote administration, no controls for potential environmental prompts to orientation questions (calendars, clocks, watches, etc) were implemented.

Inclusion criteria were as follows: age 50 to 85 years; MMSE raw score of 23 to 30 for participants with MCI or mild AD and 26 to 30 for CU; CU or clinical diagnosis of MCI or mild AD made in the previous 5 years; English as a first language; availability of a study partner for Clinical Dementia Rating scale (CDR) [26] semistructured interview; and access and ability to use a smartphone running an operating system of Android 7 or above, or iOS 11 or above.

Exclusion criteria were as follows: current diagnosis of general anxiety disorder, recent (6 month) history of unstable psychiatric illness, history of stroke within the past 2 years, or a documented history of transient ischemic attack or unexplained loss of consciousness in the last 12 months. Participants treated with medications for symptoms related to AD were required to be stabilized on these medications for at least 8 weeks before study entry and throughout the study. Participants with a current diagnosis of major depressive disorder (United Kingdom) or those with a current or 2-year history of major depressive disorder (United States) were excluded.

This research was approved by the institutional review boards of the relevant research authorities (UK research ethics committee reference: 20/WM/0116; US Institutional Review Board reference: 8460-JGDuffy). Informed consent was obtained at the study site (United States) or electronically in accordance with the health research authority guidelines (United Kingdom). The studies are registered at ClinicalTrials.gov (NCT04828122 and NCT04928976).

ASRT task responses were transcribed using Google’s speech-to-text [32] automatic speech recognition system, using an enhanced speech recognition model (the “video” model, suitable for recordings that may contain background noise). All task responses were also transcribed manually by following a standardized procedure, which included transcription of commentary, filled pauses, and partial words. The word error rate (WER) of the automatic transcript was calculated using the HuggingFace package [33], as the average number of errors per manual transcript word. This was calculated after removing punctuation, setting all text characters to lower case, and removing filled pauses and partial words from the transcripts before comparison.

Transcription was followed by automated textual analysis completed using a generalized match (G-match) score. G-match was computed in Python as the weighted sum of the cosine similarity between the embeddings of original ASRT text and the transcribed retellings, providing an automatic quantitative evaluation of similarity across the 2 texts. G-match provides an index of the proportional recall for each story, with potential scores ranging from 0 to 1 (hypothetically perfect performance). Mean G-match per triplet was also computed. The underlying representations of the model are based on a pretrained BERT model [34], which is a large language model pretrained on a corpus of more than 3000 million words, to produce generalized representations of language and how it is used.

Further analysis was performed using the statistical software package R v.4.0. Data were assessed for normality, followed by parametric and nonparametric analyses as appropriate. Adherence was defined as engaging with at least one ASRT story per day. Adherence patterns were examined with logistic regression models, predicting adherence at immediate and delayed recall, in relation to participant group, demographics, assessment day, and schedule. A large proportion of participants completed only 7 days of remote assessments, and longitudinal analysis of adherence was therefore limited to assessments on days 1 to 7. Participants were included as random effects. Demographics (sex, age, and years of education), assessment days (1-7), research schedule (schedule 1 or schedule 2), and participant group (CU and MCI or mild AD) were included as fixed factors.

The parallel-forms reliability of ASRTs was examined using pairwise correlational analysis. Only ASRT stories administered across both test schedules were analyzed, maintaining comparable sample sizes across comparisons and allowing for testing within the MCI or mild AD and CU subgroups. The convergent validity of these same ASRT stories was examined in relation to LM, PACC5, and CDR-SB. Analyses were repeated using the mean G-match score per triplet. Spearman rank correlation coefficients are reported throughout to maintain the consistency and comparability of reporting.

Task performance differences between groups, task administration variations, and change over time were modeled using longitudinal linear mixed-effects models. Data analyzed were restricted to remote assessment days 2 to 7, when assessments were novel and administered to all participants. The mixed model analysis included G-match as the response variable, and fixed effects of participant group, remote assessment days (2-7), order (1st, 2nd, or 3rd ASRT presented), long or short stories, and immediate or delayed recall. Demographics (age, sex, and education) were included as additional fixed effects. A random effect of participant with random slope and intercept was specified. Cohen d effect sizes for multilevel model objects were calculated using the lme.dscore command in the package EMAtools.

Analyses were repeated with the mean G-match per triplet, with equivalent random and fixed effects specifications, except for the story order, which was not included. The covariation of mean ASRT task performance across triplets with self-reported daily state was then examined by additionally incorporating fixed effects of self-reported mood, sleep, effort, and mind-wandering into the above model. The assumptions of all regression models were investigated by examining the distribution and patterns of residuals versus fitted values.

Group differences and effect sizes were also evaluated for traditional cognitive tests completed with a trained psychometrician via Zoom or in person during clinical assessments. Comparisons were only carried out for tests that were not directly or indirectly part of the study selection criteria (Digit-Symbol Coding, the Free and Cued Selective Reminding Test, and category fluency), thereby excluding MMSE (direct selection criterion), the LM Delayed Recall, and PACC5 (indirect). As participants were recruited from prior completed trials, in some of which performance thresholds on the LM delayed recall contributed to the MCI and mild AD group inclusion criteria, LM and PACC5 (of which the LM is a component) were not evaluated. The test distributions of traditional cognitive assessments were evaluated for normality, followed by parametric or nonparametric tests, as appropriate.

A total of 200 participants, 67 from the US study and 133 from the UK study, were recruited and completed the clinical assessment protocol. In total, 75.5% (151/200) of the participants completed at least one remote ASRT. Older participants (r=−0.15; P=.03), those with lower MMSE scores (r=−0.26; P<.001), and those with MCI or mild AD (33/106, 31.1% MCI or mild AD, compared with 16/94, 17% CU; χ²₁=5.4; P=.02) more often did not complete any remote assessments. There were no differences in sex ratio (χ²₁=0.4; P=.50) or years of education (r=−0.01; P=.87) between participants who contributed at least one remote assessment and those who did not.

Demographic information of the participants providing remote data are presented in Table 1. In this sample, the MCI or mild AD and CU groups did not differ with respect to age, years of education, sex, or amyloid status. The US study included proportionally more participants with cognitive impairment (22/34, 65% with MCI) than the UK sample (51/117, 43.6% with a diagnosis of MCI or mild AD). The MCI or mild AD group included a minority of participants with a diagnosis of mild AD (10/73, 14%), all recruited into the UK sample as per the inclusion criteria. A detailed breakdown of the sample characteristics by US and UK studies is provided in Table S5 in Multimedia Appendix 1.

Usability questionnaires were completed by 63.6% (96/151; CU: n=52 and MCI or mild AD: n=44) of the participants who completed remote assessments (Figure 1). Those completing usability questionnaires did not differ with respect to education level (r=−0.02; P=.78), age (r=−0.12; P=.14), or MMSE scores (r=−0.08; P=.32) compared with those who engaged in remote assessments but did not complete usability questionnaires. There was also no difference in the male to female ratio (χ²₁=0.1; P=.75) or the ratio of CU participants to participants with MCI or mild AD (χ²₁=0.7; P=.41) who did and did not complete usability questionnaires.

In total, 8% (4/52) of CU participants and 20% (9/44) of participants with MCI or mild AD reported technical difficulties. Where technical difficulties were encountered, most participants reported that these did not prevent them from completing the assessments, with no group differences (χ²₁=3.3; P=.07 and χ²₁=1.0; P=.32, respectively for technical difficulties reported, and inability to complete assessments). Most participants responded that the app was easy to use and that the task was reasonably interesting, with no group differences (r=−0.08; P=.47 and r=−0.04; P=.70, respectively for ease of use and interest in tasks).

Participants with MCI or mild AD completed fewer remote assessments than CU participants (adherence for immediate recall: 64.5% vs 77.5%; delayed recall: 61.5% vs 77.3%; Figure 2). Group differences were confirmed by mixed logistic regression analyses (immediate recall estimate=−0.97; P=.01 and delayed recall estimate=−0.84; P=.02). Adherence did not change over the assessment days (immediate recall estimate=−0.04; P=.34 and delayed recall estimate=−0.07; P=.11), but lower adherence to delayed recall was observed for the revised test schedule (estimate=−0.86; P=.03). Adherence was not associated with sex and education (all P>.20), but younger participants completed more immediate recall assessments (immediate recall estimate=−0.07; P=.02 and delayed recall estimate=−0.06; P=.06).

Figure 2 shows a heat map of the adherence patterns and task performance. In this figure, each participant is represented by a row, and task response and performance over the days of assessment are shown in colored blocks along the x-axis. Task performance is shown in color, with red to yellow grading representing low to high G-match scores. Missing data are shown in gray. This figure reflects the results reported above, with higher adherence in the CU group and no clear decline in adherence over the assessment period.

The average WER for participant recordings of automatic transcripts compared with manual transcripts was 0.11. The average WER differed across participant groups, with WER=0.09 in CU participants and WER=0.13 in participants with MCI or mild AD (t_108.1=−3.81; P<.001; Cohen d=0.63).

G-match for ASRTs and triplets showed good psychometric properties. Data generated showed no ceiling or floor effects (Figure 3A; Figures S1-S4 in Multimedia Appendix 1). Task performance characteristics are provided in Tables S6-S8 in Multimedia Appendix 1.

Parallel-forms reliability for individual ASRT stories at immediate recall are presented in Figure 4. Equivalent figures for delayed recall, separated by clinical group, are presented in Figures S5-S9 in Multimedia Appendix 1. Correlation matrices for triplets separated by immediate and delayed recall, and clinical groups, are shown in Figures S10-S12 in Multimedia Appendix 1.

Correlation coefficients in the full sample were moderate to strong for immediate recall (rho range=0.56-0.88; mean 0.73) and remained so after restricting analyses to participants with MCI or mild AD (rho range=0.31-0.87; mean 0.65) and CU participants (rho range 0.39-0.85; mean 0.65). Similarly, correlations between parallel ASRT stories were moderate to high for delayed recall (full sample: rho range=0.54-0.86; mean 0.73) and remained so when restricting analyses to participants with MCI or mild AD (rho range=0.37-0.88; mean 0.65) and CU participants (rho range=0.32-0.83; mean 0.64).

Parallel-forms reliability was higher when examined for mean scores obtained across triplets (immediate: rho range=0.77-0.88, mean 0.83; and delayed: rho range=0.76-0.89, mean 0.85), remaining consistently high in MCI or mild AD (immediate: rho range=0.57-0.88, mean 0.73; and delayed: rho range=0.60-0.89, mean 0.75) and CU subgroups (immediate: rho range=0.67-0.83, mean 0.76; and delayed: rho range=0.68-0.85, mean 0.77).

ASRT task performance correlated moderately with other cognitive and clinical measures (LM, CDR-SB, and PACC5) in the full sample across both immediate and delayed recalls (Figure 4). The mean correlation coefficients between immediate recall ASRTs with LM-immediate recall, PACC5, and CDR-SB were rho=0.56, 0.65, and 0.51, respectively. The mean correlation coefficients between ASRTs with LM-delayed recall, PACC5, and CDR-SB were rho=0.54, 0.66, and 0.50, respectively. Analysis results and figures for delayed recall and results separated by participant group are provided in Figures S5-S9 in Multimedia Appendix 1. Correlation coefficients remained in the moderate range after restricting analyses to participants with MCI or mild AD but were typically lower in CU participants. Correlations between ASRT triplets and other cognitive tests are provided in Figures S10-S12 in Multimedia Appendix 1.

The longitudinal mixed models are presented in Table 2, with similar results for individual ASRTs and triplets. Task performance improved across the week, with a modest linear daily improvement in G-match by assessment day. There was an effect of group with lower scores in the MCI or mild AD group for both individual stories and triplets, with an effect size of Cohen d=1.54. G-match was higher for immediate recall and shorter stories and higher for the latter ASRTs administered within each triplet. Demographics were not associated with task performance. Longitudinal data are displayed in Figure 3B, showing within- and between-subject variability.

After incorporating self-report assessments into the mixed model predicting G-match for triplets, the models revealed a significant effect of mood (estimate=0.007; SE 0.002; P<.001) and mind-wandering (estimate=−0.007; SE 0.002; P<.001), with better daily mood and lower mind-wandering associated with better daily task performance.

Traditional neuropsychological tests administered in person during in-clinic assessments were also predictive of MCI or mild AD diagnostic status, with large effect sizes identified: Digit-Symbol Coding: t₈₂=5.40, P<.001, Cohen d=1.07; category fluency total score: t t₁₄₈=7.16, P<.001, Cohen d=1.17; and the sum of free and total recall from the Free and Cued Selective Reminding Test: t₁₀₈=5.56, P<.001, Cohen d=1.01.

This study indicates that daily unsupervised and self-administered speech-based testing is acceptable and feasible for older participants with and without cognitive impairment. Participants engaged in daily optional assessments with moderate levels of adherence. There was no observable reduction in adherence levels over a weeklong period of assessment. The participants experienced infrequent technical problems and reported that the tests were easy to use and reasonably interesting.

Results indicate that remote automatic test administration and autoscoring of story recall can provide sensitive cognitive measurement in at-risk populations. The ASRT G-match, an automatically scored measure of proportional recall, showed consistent differences in task performance between cognitively healthy participants and those with MCI or mild AD. Separation in task performance between diagnostic groups was consistent across the assessment period and across individual ASRT stories (Figure 3), showing a strong effect size for differentiating CU participants from those with MCI or mild AD (Cohen d=1.54), while controlling for age, education, and sex. The equivalent area under the receiver operating characteristic curve was 0.86, based on previously published equivalence tables [35].

The effect size for ASRTs is larger than that seen for a range of traditional cognitive tests typically administered in person and under supervision. Comparisons with the LM delayed recall and PACC5 were not made, as participants in this study were recruited from prior trials in which test performance on LM delayed recall constituted part of the trial inclusion criteria for patients with MCI or mild AD, which would likely inflate effect sizes for these tests.

The ASRTs discrimination between clinical groups reported here outperforms those previously reported for differentiating CU individuals from those with MCI using other traditional cognitive tests administered in person and in the clinic, such as the MMSE (Cohen d=0.69), the 6-Item cognitive impairment test (Cohen d=0.65), and Addenbrooke’s Cognitive Examination-Revised (Cohen d=0.73), albeit with similar results reported previously for the Montreal Cognitive Assessment battery (Cohen d=1.45) [36]. The test also performs well in comparison with the Cogstate brief battery, when administered remotely and unsupervised, where effect sizes for differences between MCI and CU groups in subtests range from Cohen d=0.22 to Cohen d=0.62 [37].

Although the mixed clinical group examined in this analyses limits direct comparison with previously published metrics in subjects with MCI only, the mild AD group in this study comprised only a small proportion (10/73, 14%) of those contributing to the MCI or mild AD group. After excluding participants with mild AD from the linear mixed model analysis, this yielded an effect size of Cohen d=1.45 (equivalent area under the curve=0.85 [35]) for the difference between the CU participants and participants with MCI.

ASRT stimuli are carefully designed and balanced for key linguistic and discourse metrics, including the number of words, number of sentences, number of dependent clauses, mean sentence length, and ratio of dependent clauses to t-units (the number of shortest grammatically complete units into which a string of written or spoken language can be partitioned). This balancing of the stimuli is also reflected in good parallel-forms reliability between ASRT parallel stimuli, which is consistently high across immediate and delayed recall and with analyses constrained to clinical subgroups (MCI or mild AD and CU). The ASRT analysis pipeline also has significant advantages for test-retest reliability and parallel-forms reliability, as text similarity is evaluated in the same way every time, producing a standardized scoring system across the parallel test forms. A more objective quantification of text similarity, using a large language corpus for training, removes some of the more arbitrary features common to story recall task scoring, in which specific paraphrases are accepted, and the size of information units shows some variability [29].

Repeated exposure to the test stimuli may lead to unwanted practice effects that reduce the validity of the test as a measure of new learning, with research also showing differential practice effects across clinical diagnostic groups for tests such as list learning tests and LM [38]. Practice effects may be particularly important when considering where the same story recall stimuli are used repeatedly in longitudinal research or clinical monitoring or for diagnostic thresholding as cut-offs for research studies or clinical trials [16,39]. Other available story recall tests typically have a limited range of parallel forms.

The number of available parallel forms of the ASRT test allows for a higher-frequency (daily) assessment over a shorter period without test repetition, such as that carried out in this study. Alternatively, tests could be administered at larger intervals (weekly, monthly, or longer) to evaluate longer-term changes with little or no repetition of stimuli, thereby likely reducing practice effects.

Although alternate test variants can help reduce practice effects, they do not completely correct for retesting, which can be modified by repeated exposure to the task and greater familiarity with the test structure or method [40]. In this study, despite novel stimuli being presented during each assessment, test scores improved modestly during the week, indicating that increased familiarity with the app, testing procedure, and test structure resulted in a subtle improvement over time. Task performance improvements over the weeklong assessment period were modest (with an estimated daily change in G-match of only 0.4% of the initial intercept estimate value). The improvement in test performance, in combination with the absence of adherence changes over the course of the study, did not indicate any strong fatigue effects.

ASRT tests correlated moderately with a well-established test of verbal episodic memory, cognitive composites, and clinician-reported outcomes, indicating acceptable convergent validity, and with results comparable with, or better than, other studies of computerized or unsupervised remote assessments [41-43]. Correlations with LM and clinician-reported outcomes were in the moderate range, with lower correlation coefficients linked to test invariance owing to ceiling- or floor-level performance on these traditional clinical and neuropsychological assessments in CU individuals.

Task performance also varied with aspects of study design, with stories administered later in triplets delivering a more comprehensive recall. These effects appear to lead to greater variation between individual ASRT stories but are averaged out when the G-match is examined across story triplets. Analysis of story triplets showed higher parallel-forms reliability between the ASRTs administered and analyzed in threes, albeit with broadly unchanged differences in group performance. Task performance, as measured with G-match, was typically higher for shorter stories, indicating that responses more comprehensively covered the story source text where participants were asked to recall less material.

This study also showed within-subject variation in task performance, in part reflecting the measured effects of state factors on cognitive performance, in particular daily mood and effort. Variation from within-subject differences can make it challenging to differentiate clinical change from measurement error [8], and higher-frequency assessments carried out longitudinally can help generate more reliable estimates of cognitive function and change. Repeated measurements allow these state effects to be concurrently measured and included or controlled for in the longitudinal analyses.

To meet the eligibility criteria, participants were required to be able to use and access a smartphone. This may have biased the sample by overselecting those with higher familiarity with technology. Older and more cognitively impaired participants were less likely to contribute to the remote study component, and when they engaged in remote assessments, they contributed less frequently. However, the adherence statistics presented here reflect participants’ engagement in optional assessments, which may have differed had these been compulsory. Many home testing options require at least a modest level of technological fluency that some older adults may find challenging, challenges that may be compounded by cognitive impairment or comorbidities [44].

Therefore, the data presented may not reflect task performance in more impaired individuals or those with lower levels of technological familiarity. Assessments under supervision, either in the clinic or during a telemedicine visit, allowing for provision of additional support where required, could be better suited to more impaired individuals.

By collecting usability data during remote assessments, we were able to establish that most participants did not experience any technical problems and that the app was generally easy to use. However, more detailed qualitative feedback on the type of usability and any technical issues was not collected. Further evaluation of the nature of these difficulties is required. This information can be used to improve the user interface and participant engagement with remote assessments.

In response to participants’ and study centers’ feedback on the high participant burden of the initial test schedule, the testing schedule was altered in the middle of the study to reduce burden, thereby limiting the data available for certain ASRT test variants.

The design of the study makes it difficult to differentiate between the effects of individual stories themselves (ie, which ASRT story was used) and the effects of study design, such as the test order or day of assessment. Future studies may benefit from adopting a randomized design, with ASRTs randomly selected and allocated to different testing instances to derive test performance metrics independent of these additional confounders. For longitudinal studies, either short or long stories should be adopted to improve the consistency of test scores over time and help better characterize cognitive change.

We found differences in WER when comparing automated and manual transcripts of CU participants and those with MCI or mild AD, indicative of the differential intelligibility of speech or recording quality in these 2 groups. Differences in the performance of automatic transcription will impact the analysis further along in the analysis pipeline, indicating that group differences in scores likely reflect not only group differences in proportional recall but may also incorporate speech intelligibility and participant’s device use characteristics. However, these effects warrant further investigation.

The participants included in this study constituted a select sample. The sample was selected to exclude patients with concurrent neurological and mental health conditions. They were recruited from prior clinical trials completed in the United States and the United Kingdom and reflect a group of individuals who are actively engaged in clinical research. The participants lacked racial diversity (with most of the sample identifying as White and with only 2.6% [N=4] with Asian, Black, African, or African American background). Replication is now needed in more clinically and demographically heterogeneous samples.

The recent Food and Drug Administration approval for the first disease-modifying treatment for people at risk of developing AD highlights the importance of adequate screening and early detection as well as the importance of monitoring treatment response. Briefer, convenient, and lower-burden daily assessments may provide more reliable data to evaluate disease progression or treatment response than lengthy one-off assessments [9]. Brief digital assessments completed at home and repeatable over time could improve access to AD screening compared with current clinical standards, which typically require clinical visits and extensive neuropsychological assessment.

This study showed that brief, remotely administered, and automatically scored ASRTs are sensitive to early cognitive impairments commonly identified through more extensive clinical assessments. The tests showed good properties for repeated administration and convergent validity with established tests of episodic memory, cognitive composites, and clinician-reported outcomes (CDR-SB). The test showed good acceptability and usability for older adults with and without cognitive impairment. Furthermore, owing to the automatic administration and scoring of ASRTs, this test presents a minimal administrative burden, requiring no trained personnel or specialist equipment.

Speech is instrumental in daily functioning and a natural response modality for participants to use in response to current smart devices, such as smartphones. Speech responses are also a common component of cognitive tests; however, data generated in these tests, including those reported in this study, often relate simple pass or fail characteristics of response accuracy. New metrics using audio- and text-based artificial intelligence models to target other changes measurable in speech data (acoustic [45,46], semantic [47-49], and linguistic [46]) in early-stage AD could further leverage the information content of ASRTs, developing a new class of powerful, fully automated speech biomarkers.