This study implemented a cross-sectional design to evaluate concurrent validity between instruments. As a validation study, no random assignment was possible or required. Additionally, no form of blinding was used in this study design.

The goal sample size for this study was set at 200 participants to ensure adequate statistical power while allowing for potential attrition and incomplete data. Criteria for inclusion were adults aged between 45 and 75 years, BMI between 18.5 and 39.9 kg/m², and at least two of the following Alzheimer risk factors: high school education or less; BMI >25; and history of diabetes, hypertension, high cholesterol, or smoking. Participants were excluded if they had a diagnosed mental health condition, dementia, probable dementia, mild cognitive impairment, or other major health condition; a recent cardiovascular event; vision problems that would prevent viewing of a screen; learning disability; or more than one of the following Alzheimer protective factors: a high level of physical activity, a high level of fish consumption, or a high level of cognitive engagement. Participants were recruited in northwest Arkansas using local radio, emails, social media advertisements, news releases, and word of mouth.

Participants reported to the laboratory and completed the RBANS assessment with an experienced test administrator. RBANS assessment procedures are described in detail elsewhere [14], but briefly, the RBANS-C assessment asks participants to match sequential symbols with corresponding numbers from a key on the same page, writing the correct number below each symbol. The RBANS-C raw score is calculated as the number of correct numbers filled in within 90 seconds. Assessments were scored by an experienced test administrator in accordance with the RBANS Testing Manual [14].

After a minimum of 30 minutes of unrelated physical testing, participants were instructed to begin using the self-guided cognitive testing platform (Neurotrack Digital Testing Platform; Neurotrack Technologies Inc) containing the DDSST test. The DDSST is based on the Digit Symbol Substitution Test [16]. The assessment provided both written and visual step-by-step, on-screen instructions; asked participants to determine if 2 symbols were equal or unequal based on a legend with 9 number-symbol pairs (Figure 1); and was objectively scored based on accuracy and speed using Neurotrack’s automated scoring algorithm. Specifically, raw scores were calculated as the number of correct responses given in 2 minutes. Data for this study were collected as part of an ongoing parent study (Digital Cognitive Multi-domain Alzheimer’s Risk Velocity [DC MARVel] Study) that is longitudinally examining changes in cognition and AD risk measures in at-risk adults randomized into either a health education or health-coaching intervention [17]. Further information regarding other subtests found in the Neurotrack Digital Testing Platform can be found in the published protocol for the DC MARVel Study [17].

All statistical analyses were performed utilizing the Analyse-It extension for Microsoft Excel. The linear relationship between the RBANS-C and DDSST raw scores was calculated using a Pearson product-moment correlation. For tests of mean differences, agreement, and equivalency, the raw scores for both tests were scaled to a distribution with a mean of 100 and an SD of 15 to allow for the direct comparison of scores without impacting the distribution of scores or the appropriateness of the tests being used. In the case of scaling the RBANS-C score, the obtained score was converted to a z score based on the accepted population mean and SD for RBANS-C raw scores and then multiplied by 15, but as no accepted DDSST population mean and SD exists, those obtained from this study were used.

Differences between group means were assessed using a 2-tailed dependent samples t test. Agreement was assessed with a Bland-Altman limits of agreement (LoA) plot with a 90% LoA. The LoA cutoff was set at the mean bias plus or minus the minimum reliable change (RC) score for the RBANS-C test obtained from previous literature (z=1.53) [12]. Researchers have stressed that RC is a more appropriate metric for assessing minimum change in the RBANS test than the minimum clinically important difference or standard error of measure [18]. A mountain plot was also generated to assess the y-axis distribution of the mean-difference plot.

Equivalency was tested using a two one-sided t tests (TOST) analysis, and the equivalency upper and lower bounds were determined [19] using the RBANS RC score obtained from previous literature described above. An a priori α level of .05 was used for all appropriate analyses.

This study was approved by the Institutional Review Board at the University of Arkansas (protocol #2009280813).

Participants were informed of their rights as research participants and clearly notified that their participation was voluntary and they could withdraw at any time. All participants signed an approved consent form in accordance with the ethical standards of Helsinki.

In total, 210 participants were initially enrolled in this study. After prestudy attrition and adjusting the data set for incomplete testing data, a final sample of 191 adults (female: n=138; male: n=53) was analyzed. The average age of the sample was 62.2 (SD 8.27) years. Further descriptive statistics are presented in Table 1.

The moderately positive correlation obtained for the RBANS-C raw score and DDSST raw score was statistically significant (r=.577; P<.001). A 2-tailed dependent-samples t test showed no significant differences between the RBANS-C and DDSST scores within participants (t₁₉₀=–0.09; P=.93). Bland-Altman plots revealed the 90% LoA (–22.66 to 22.75) was within the a priori cutoff (–22.91 to 22.99), indicating that the scores for RBANS-C and DDSST were in acceptable agreement. Additionally, the mean bias score was near zero (0.04), indicating low systemic bias in scores, and there was no obvious linear pattern in the scatter plot distribution (Figure 2). The mountain plot distribution was roughly symmetrical, had a peak close to zero bias, and had no obvious tail skew (Figure 2). TOST analysis indicated that the scores for RBANS-C and DDSST were equivalent.

The purpose of this study was to assess the concurrent validity of a novel DDSST test as compared to the RBANS-C, a gold-standard measure of PS. Results showed no statistically significant differences in mean test scores within participants; a significant, moderately positive correlation in individual test scores; and acceptable agreement and equivalency between the 2 assessments. These results indicate that the DDSST is a valid assessment tool for evaluating PS in the sample of middle-age and older adults.

The potential impact of an assessment such as the DDSST is greater when the efficiency, scalability, and convenience of a digitally enabled test is considered. Digital tests allow the instant reporting of results to the test taker and potentially clinicians; the immediate and ad hoc distribution of tests to any number of test takers; and the convenience of having the ability to take tests anywhere, at any time.

Compared to previous studies validating digital assessments of PS, the relationship between the novel test and gold-standard test was lower here (r=.577 vs r=.75-.8) [10,11]. Validation, however, should not be based upon correlation between scores as this does not accurately demonstrate agreement between individual scoring pairs but rather an overall linear relationship that minimizes residuals [12]. Of the previous studies of digital instruments evaluating PS reviewed by the authors, none analyzed or reported the agreement between paired measures from novel and gold-standard tests in their assessment of concurrent validity [9-11]. As such, this study conducted a more comprehensive analysis of concurrent validity through the additional examination of agreement and appears to be the first study of a digital PS test to base its validation on those criteria.

First, this study may have been limited by its protocol. It was carried out as part of a larger trial, and the order of testing was not randomized in the protocol to minimize learning effects or cognitive fatigue effects. Second, the sampling procedure specifically excluded individuals with cognitive impairments. By excluding segments that exist in the general population, generalizability is reduced. Third, intrarater and test-retest reliability could not be assessed for the novel DDSST instrument as repeat trials were not conducted as a part of this data set.

Future studies should include individuals with cognitive impairments to assess validity in a broader population segment, as well as assess the ability of the DDSST to discriminate between levels of cognition. Although the test-retest reliability of the DDSST has been verified in a different sample [20], additional psychometric testing with a representative population is warranted to ensure the full utility of the measure.

Despite the study’s limitations, the DDSST shows promising clinical utility. DDSST scores were in agreement with, and equivalent to, scores obtained from the RBANS-C, a gold-standard test of PS. Given its demonstrated concurrent validity, the DDSST appears to be a suitable option for widespread and rapid cognition testing. Further investigation, however, is required to assess the reliability of the instrument.