A Novel Digital Digit-Symbol Substitution Test Measuring Processing Speed in Adults At Risk for Alzheimer Disease: Validation Study

Introduction

Currently, Alzheimer disease (AD) affects more than 6 million Americans, and by the year 2050, this number is expected to rise to more than 13 million [1]. One in 3 seniors will die while experiencing AD or a related dementia, and the yearly deaths directly attributable to AD account for more than breast and prostate cancer combined [1]. AD and associated dementias also incur a substantial economic and social cost. It is estimated that in 2021, these cognitive diseases cost Americans US $355 billion; this figure is projected to rise to US $1.1 trillion by 2050. Socially, in 2020, more than 11 million unpaid caregivers worked 15.3 billion hours providing support for individuals with AD and related dementias—their time is valued at US $257 billion in lost wages [1]. Recognizing this dismal outlook, the research community has investigated many of the cognitive constructs associated with the disease in an effort to better understand the progression of the illness, detect signs earlier, and potentially develop methods to mitigate it [2].

One cognitive construct researchers have investigated in the context of AD is processing speed (PS). PS is defined as the rate at which an individual can analyze cognitive stimuli and complete cognitive tasks [3,4], and it has been shown to decrease significantly in individuals with mild cognitive impairment and further still in individuals with AD [3]. As such, PS is an important component in cognitive assessment protocols and AD-monitoring programs [5]. Several laboratory- and clinic-based assessments of PS have been developed [5], but the limited availability of these tests concerns researchers as neurobiological decline can occur 15 years before any cognitive deficits are measurable, indicating that testing should take place frequently and ubiquitously to ensure that cognitive decline is detected as early as possible [2,6,7]. More recently, to address the need for widespread, rapid testing, researchers have focused on the development of digital cognitive tests that can be taken quickly on a mobile device and demonstrate high scalability, efficiency, and convenience [6,8]. Although several digital PS tests have been developed and presented as valid assessments [9-11], the validation procedures followed in these studies only examined the linear relationship between novel test scores and a gold-standard test, not the agreement between individual scores as is required for proper validation [12]. The purpose of this study was to validate a novel PS task, a digital digit-symbol substitution test (DDSST), by examining its concurrent validity through comparison to a gold-standard test of PS. As tests of digit-symbol coding have been identified as gold-standard measures of PS [13], the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) coding test (RBANS-C) [14,15] was chosen as a comparison for its validity in the population analyzed in this study [15].

Methods

Study Design

This study implemented a cross-sectional design to evaluate concurrent validity between instruments. As a validation study, no random assignment was possible or required. Additionally, no form of blinding was used in this study design.

Participants

The goal sample size for this study was set at 200 participants to ensure adequate statistical power while allowing for potential attrition and incomplete data. Criteria for inclusion were adults aged between 45 and 75 years, BMI between 18.5 and 39.9 kg/m², and at least two of the following Alzheimer risk factors: high school education or less; BMI >25; and history of diabetes, hypertension, high cholesterol, or smoking. Participants were excluded if they had a diagnosed mental health condition, dementia, probable dementia, mild cognitive impairment, or other major health condition; a recent cardiovascular event; vision problems that would prevent viewing of a screen; learning disability; or more than one of the following Alzheimer protective factors: a high level of physical activity, a high level of fish consumption, or a high level of cognitive engagement. Participants were recruited in northwest Arkansas using local radio, emails, social media advertisements, news releases, and word of mouth.

Procedure

Participants reported to the laboratory and completed the RBANS assessment with an experienced test administrator. RBANS assessment procedures are described in detail elsewhere [14], but briefly, the RBANS-C assessment asks participants to match sequential symbols with corresponding numbers from a key on the same page, writing the correct number below each symbol. The RBANS-C raw score is calculated as the number of correct numbers filled in within 90 seconds. Assessments were scored by an experienced test administrator in accordance with the RBANS Testing Manual [14].

After a minimum of 30 minutes of unrelated physical testing, participants were instructed to begin using the self-guided cognitive testing platform (Neurotrack Digital Testing Platform; Neurotrack Technologies Inc) containing the DDSST test. The DDSST is based on the Digit Symbol Substitution Test [16]. The assessment provided both written and visual step-by-step, on-screen instructions; asked participants to determine if 2 symbols were equal or unequal based on a legend with 9 number-symbol pairs (Figure 1); and was objectively scored based on accuracy and speed using Neurotrack’s automated scoring algorithm. Specifically, raw scores were calculated as the number of correct responses given in 2 minutes. Data for this study were collected as part of an ongoing parent study (Digital Cognitive Multi-domain Alzheimer’s Risk Velocity [DC MARVel] Study) that is longitudinally examining changes in cognition and AD risk measures in at-risk adults randomized into either a health education or health-coaching intervention [17]. Further information regarding other subtests found in the Neurotrack Digital Testing Platform can be found in the published protocol for the DC MARVel Study [17].

‎

Data Analysis

All statistical analyses were performed utilizing the Analyse-It extension for Microsoft Excel. The linear relationship between the RBANS-C and DDSST raw scores was calculated using a Pearson product-moment correlation. For tests of mean differences, agreement, and equivalency, the raw scores for both tests were scaled to a distribution with a mean of 100 and an SD of 15 to allow for the direct comparison of scores without impacting the distribution of scores or the appropriateness of the tests being used. In the case of scaling the RBANS-C score, the obtained score was converted to a z score based on the accepted population mean and SD for RBANS-C raw scores and then multiplied by 15, but as no accepted DDSST population mean and SD exists, those obtained from this study were used.

Differences between group means were assessed using a 2-tailed dependent samples t test. Agreement was assessed with a Bland-Altman limits of agreement (LoA) plot with a 90% LoA. The LoA cutoff was set at the mean bias plus or minus the minimum reliable change (RC) score for the RBANS-C test obtained from previous literature (z=1.53) [12]. Researchers have stressed that RC is a more appropriate metric for assessing minimum change in the RBANS test than the minimum clinically important difference or standard error of measure [18]. A mountain plot was also generated to assess the y-axis distribution of the mean-difference plot.

Equivalency was tested using a two one-sided t tests (TOST) analysis, and the equivalency upper and lower bounds were determined [19] using the RBANS RC score obtained from previous literature described above. An a priori α level of .05 was used for all appropriate analyses.

Ethics Approval

This study was approved by the Institutional Review Board at the University of Arkansas (protocol #2009280813).

Informed Consent

Participants were informed of their rights as research participants and clearly notified that their participation was voluntary and they could withdraw at any time. All participants signed an approved consent form in accordance with the ethical standards of Helsinki.

Results

Participants

In total, 210 participants were initially enrolled in this study. After prestudy attrition and adjusting the data set for incomplete testing data, a final sample of 191 adults (female: n=138; male: n=53) was analyzed. The average age of the sample was 62.2 (SD 8.27) years. Further descriptive statistics are presented in Table 1.

Data Analysis

The moderately positive correlation obtained for the RBANS-C raw score and DDSST raw score was statistically significant (r=.577; P<.001). A 2-tailed dependent-samples t test showed no significant differences between the RBANS-C and DDSST scores within participants (t₁₉₀=–0.09; P=.93). Bland-Altman plots revealed the 90% LoA (–22.66 to 22.75) was within the a priori cutoff (–22.91 to 22.99), indicating that the scores for RBANS-C and DDSST were in acceptable agreement. Additionally, the mean bias score was near zero (0.04), indicating low systemic bias in scores, and there was no obvious linear pattern in the scatter plot distribution (Figure 2). The mountain plot distribution was roughly symmetrical, had a peak close to zero bias, and had no obvious tail skew (Figure 2). TOST analysis indicated that the scores for RBANS-C and DDSST were equivalent.

‎

Discussion

Principal Findings

The purpose of this study was to assess the concurrent validity of a novel DDSST test as compared to the RBANS-C, a gold-standard measure of PS. Results showed no statistically significant differences in mean test scores within participants; a significant, moderately positive correlation in individual test scores; and acceptable agreement and equivalency between the 2 assessments. These results indicate that the DDSST is a valid assessment tool for evaluating PS in the sample of middle-age and older adults.

Impact

The potential impact of an assessment such as the DDSST is greater when the efficiency, scalability, and convenience of a digitally enabled test is considered. Digital tests allow the instant reporting of results to the test taker and potentially clinicians; the immediate and ad hoc distribution of tests to any number of test takers; and the convenience of having the ability to take tests anywhere, at any time.

Compared to previous studies validating digital assessments of PS, the relationship between the novel test and gold-standard test was lower here (r=.577 vs r=.75-.8) [10,11]. Validation, however, should not be based upon correlation between scores as this does not accurately demonstrate agreement between individual scoring pairs but rather an overall linear relationship that minimizes residuals [12]. Of the previous studies of digital instruments evaluating PS reviewed by the authors, none analyzed or reported the agreement between paired measures from novel and gold-standard tests in their assessment of concurrent validity [9-11]. As such, this study conducted a more comprehensive analysis of concurrent validity through the additional examination of agreement and appears to be the first study of a digital PS test to base its validation on those criteria.

Limitations

First, this study may have been limited by its protocol. It was carried out as part of a larger trial, and the order of testing was not randomized in the protocol to minimize learning effects or cognitive fatigue effects. Second, the sampling procedure specifically excluded individuals with cognitive impairments. By excluding segments that exist in the general population, generalizability is reduced. Third, intrarater and test-retest reliability could not be assessed for the novel DDSST instrument as repeat trials were not conducted as a part of this data set.

Future Directions

Future studies should include individuals with cognitive impairments to assess validity in a broader population segment, as well as assess the ability of the DDSST to discriminate between levels of cognition. Although the test-retest reliability of the DDSST has been verified in a different sample [20], additional psychometric testing with a representative population is warranted to ensure the full utility of the measure.

Conclusion

Despite the study’s limitations, the DDSST shows promising clinical utility. DDSST scores were in agreement with, and equivalent to, scores obtained from the RBANS-C, a gold-standard test of PS. Given its demonstrated concurrent validity, the DDSST appears to be a suitable option for widespread and rapid cognition testing. Further investigation, however, is required to assess the reliability of the instrument.