This study was approved by the University of Washington (UW) Institutional Review Board (IRB) for human subject participation (review number STUDY00000790).

Participants were recruited from a research registry maintained by the Alzheimer’s Disease Research Center associated with the UW Medicine Memory and Brain Wellness Center clinic [40]. This registry is a continually updated database of individuals who have expressed interest and signed an IRB-approved consent form to be contacted about participation in Alzheimer disease (AD) and related dementias research studies, many of whom have been recently evaluated at the clinic and, hence, have a clinical diagnosis or evaluation. Those with listed addresses within a 70-mile radius of Seattle, Washington, were contacted by phone or email, based on information provided within the registry. If the person was unable to physically use an iPad, if the person was too cognitively impaired to understand or follow instructions, or if the primary contact (eg, spouse) indicated that the person was unable to participate, they were not recruited for the study. When study procedures were modified from in-person to remote administration due to the COVID-19 pandemic (approximately March 2020), participants outside the initial geographical range were contacted to explore remote testing capabilities. We required that these participants have access to either an iPad with iOS 10 or later or a touchscreen computer and Wi-Fi connectivity to participate in the study.

Using the provided primary cognitive diagnosis within the registry, participants were divided into one of three groups: (1) NC, indicated by subjective cognitive complaint or no diagnosis of cognitive impairment, some of which were self-reported; (2) MCI, representing both amnestic and nonamnestic subtypes; or (3) dementia, which included dementia due to AD, frontotemporal dementia, vascular dementia, Lewy body dementia, mixed dementia, or atypical AD.

A total of 5 participants were not recruited from the registry but via snowball sampling from other participants. The recruitment of these participants was simply due to convenience, typically a family member or friend that was also available at the time of testing. Out of these 5 participants, 4 of them were placed into the NC group after their self-reports denied symptoms or a history of cognitive impairment; these 4 were not patients of the memory clinic. The remaining participant was a patient from the memory clinic, just not a part of the registry, and was placed in the AD group based on their most recent diagnosis retrieved from their medical records. Testing for these 5 participants was administered on-site.

A total of 241 individuals were contacted to participate, and 99 participants completed the study. Demographic details of the participants are provided in Table 1. The three groups did not differ to a significant degree in terms of education, administration type, or recruitment type, but there were differences in age and sex.

We found that most participants in the NC group were able to complete the assessments, whereas the dementia group had a higher time-out rate, with the MCI group falling in between the two (Figure 1). The time-out function occurs when a participant cannot complete a trial of the assessment in 30 seconds; it is embedded in the assessments of the Stroop test and the Trail Making Test, Parts A and B (Trails A/B). Time-outs were mainly due to response delays, where participants were attempting the test but could not answer quickly enough. Overall, the dementia group took significantly more time to complete the BrainCheck battery (median 30.5, IQR 23.4-37.1 min) compared to the MCI group (median 21.5, IQR 19.3-24.2 min) and the NC group (median 17.8, IQR 15.4-19.6 min).

BrainCheck assessments were compared across the three groups using a linear regression model with age, sex, and administration type as regressors (Figure 2 and Table 2). Individual scores, such as the BrainCheck Overall Score, were normalized for age and device. Overall, participants with greater cognitive impairment showed lower BrainCheck assessment scores. All individual assessments except Trails B showed significant differences in performance between the NC and dementia groups, whereas two of the seven assessments (ie, Immediate Recognition and Digit Symbol Substitution) showed significant differences in performance between all three groups (Figure 2 and Table 2). Digit Symbol Substitution, Flanker, and Trails A/B assessments showed long tails in the scores of the dementia group because some participants in the dementia group only completed parts of the assessments or exhibited low accuracy (Figure S1 in Multimedia Appendix 1).

The BrainCheck Overall Score is a composite of all individual assessments within the BrainCheck battery, representing overall performance (see details in the Measurements section). Using an existing normative population database, partly compiled from controls in previous studies [38,39], the BrainCheck Overall Score was adjusted for age and the device used to generate the normalized BrainCheck Overall Scores. The normalized BrainCheck Overall Scores differed significantly among these three groups (P<.001). Pairwise comparisons with Tukey adjustments for multiple comparisons show that the NC group scored significantly higher than the MCI group (P=.002) and the dementia group (P<.001), and the MCI group scored significantly higher than the dementia group (P<.001; Figure 3).

Using ROC analysis, BrainCheck Overall Scores achieved a sensitivity of 88% and a specificity of 94% for classifying between dementia and NC participants (AUC=0.95), a sensitivity of 86% and a specificity of 83% for classifying between MCI and NC participants (AUC=0.84), and a sensitivity of 83% and a specificity of 77% for classifying between dementia and MCI participants (AUC=0.79; Figure 4).

Using methods described by Luo and Xiong for three-group classification [41], the optimal lower and upper cutoffs of the normalized BrainCheck Overall Score in maximizing diagnostic accuracy were –3.64 and –0.06, respectively. This achieved true positive rates of 80% for the NC group, 64% for the MCI group, and 81% for the dementia group (Figure 5).

Consistent with prior findings in concussion [38] and in dementia and cognitive decline [39] samples, this study demonstrated that BrainCheck is consistent in its capability to detect cognitive impairment and can reliably detect severity and differentiate between cognitive impairment groups (ie, NC, MCI, and dementia). As expected, participants with more severe cognitive impairment performed worse across the individual assessments and on BrainCheck Overall Scores. The BrainCheck Overall Scores separated participants of different diagnostic groups successfully with high sensitivity and specificity.

BrainCheck Overall Scores were more robust in distinguishing between these groups where participants in the dementia group had significantly lower scores than those in the NC group. The BrainCheck battery was able to distinguish between NC and dementia participants, with 94% sensitivity and 88% specificity. These findings show that the BrainCheck Overall Score demonstrates better accuracy for differentiating NC from dementia, compared to the MMSE, SLUMS, and MoCA screening measures [22,41,42]. People with MCI usually experience fewer cognitive deficits and preserved functioning in activities of daily living compared to those with dementia [43], and our findings of sensitivity and specificity with separating MCI from other groups were slightly lower than the NC versus the dementia differentiations (Figure 5). Nonetheless, the BrainCheck Overall Score showed sensitivities and specificities greater than 80% in distinguishing MCI from NC and dementia groups, which is comparable to the MoCA, SLUMS, and MMSE [18-22,42]. Furthermore, a review of validated computerized cognitive tests indicated AUCs ranging from 0.803 to 0.970 for detecting MCI, and AUCs of 0.98 and 0.99 in detecting dementia due to AD [44], which were comparable with the results found in this study.

Although not all individual assessments in the BrainCheck battery differentiated between NC, MCI, and dementia, we observed a general trend for each assessment showing that dementia participants had the lowest scores, whereas the NC participants had the highest scores. Individual assessments that did show significant differences in the scores between NC and MCI groups and between dementia and MCI groups included Immediate Recognition and Digit Symbol Substitution. Notably, Digit Symbol Substitution showed significant differences in performance between all three diagnostic groups, whereas a previous study found that Digit Symbol Substitution did not show significant differences between cognitively healthy and cognitively impaired groups (n=18, P=.29) [39], likely due to this study having a larger sample size. Individual assessments with no significant differences between the MCI group and the NC and dementia groups were the Stroop and Trails A/B tests (Figure 2 and Table 2). All of these tests include time-out mechanisms if participants are unable to complete the test, and time-out rates were higher in the more cognitively impaired groups (Figure 2). Therefore, when calculating the BrainCheck Overall Score, we have introduced a penalty mechanism for timed-out assessments.

In comparison to comprehensive NPTs, which can typically last a few hours and sometimes require multiple visits [43], BrainCheck demonstrated shorter test duration, with median completion times of 17.8 (IQR 15.4-19.6) minutes for NC participants and 30.5 (IQR 23.4-37.1) minutes for dementia participants (Figure 1). Shorter test durations observed in individuals with no or less cognitive impairment suggest that computerized cognitive tests could be useful for rapid early detection in this population, prompting further evaluation, whereas those with dementia have likely already undergone a comprehensive evaluation. The wide variance in completion time for the dementia group may have uncovered the difficulty that participants with more severe cognitive impairment may have faced in completing the BrainCheck battery, compared to the lower variance observed in the NC group.

A limitation of this study was that participants were not diagnosed by a physician at the time point of BrainCheck testing. Thus, participants were placed into the diagnostic groups based on their most recent clinical diagnosis available in their electronic health record, or for the few NC participants without medical evaluations, based on their report of no cognitive symptoms or diagnosis of cognitive impairment. The period from the most recent clinical diagnosis to the date of BrainCheck testing varied among the diagnostic groups; the dementia group had the fewest days from their latest clinical evaluation (median 82.5, IQR 44.5-141.25 days), followed by the MCI group (median 244, IQR 105-346.5 days) and the NC group (median 645, IQR 225.5-1112.5 days). These large time intervals in a degenerative population leave room for cognition to worsen over time, potentially blurring the lines in the severity of cognitive impairment, where participants may have progressed to MCI from NC and to dementia from MCI during that period. This would make distinguishing NC from cognitive impairment more difficult, yet diagnostic accuracy among the groups remains high. Furthermore, the median number of days since the last clinical evaluation for NC participants was as high as 645 days. This could suggest that the NC participants did not feel an inclination to seek out further cognitive evaluation during the extended time period, and may not have experienced noticeable cognitive decline. Future validity studies should ensure that a physician evaluation and diagnosis occur closer to the time of BrainCheck testing to address these limitations.

Another limitation was that although not all individual assessment scores could differentiate the three groups, the pattern of differences across these scores may contain useful diagnostic information. The use of the BrainCheck Overall Score as an average of all individual assessment scores appears to work effectively, but does not take into account the other relationships seen across individual scores. Furthermore, some individual scores may be more informative for detecting cases, whereas others may be informative for gauging severeness. Future studies recruiting a larger sample size in each group will allow for an investigation into whether machine learning methods can extrapolate these relationships and improve the diagnostic accuracy of BrainCheck.

When administration type was considered in linear regression model analyses, scores only showed significant differences among the three diagnostic groups instead of administration types. While remote administration was not designed into the original study, stay-at-home orders due to COVID-19 required modifications, and efforts were made to provide preliminary data for remote use. With preliminary outcomes indicating feasibility for remote administration, a more robust study and increased sample size will be needed to fully validate BrainCheck’s cognitive assessment via its remote feature.

The use of computerized cognitive tests provides the opportunity to increase test accessibility for an aging population with an increased risk of cognitive impairment. The findings in this study demonstrate that BrainCheck could distinguish between three levels of cognitive impairment: NC, MCI, and dementia. BrainCheck is automated and quick to administer, both in person and remotely, which could help increase accessibility to testing and early detection of cognitive decline in an ever-aging population. This study paves the way for a comprehensive longitudinal study, exploring BrainCheck in early detection of dementia and monitoring of cognitive symptoms over time, including further comparison to gold-standard neuropsychological assessments.