A Novel Combined Audiovisual-Semantic Digital Tool for Early-Stage Cognitive Decline Detection: Development and Validation Study

Introduction

Dementia is one of the major causes of disability and dependency among older people‚ posing a significant public health challenge worldwide [1]. While dementia currently lacks effective treatment, it is understood to be a progressive continuum. There is cumulative evidence suggesting that early identification of and intervention for cognitive decline confer both clinical and health-economic benefits [2]. Notably, approximately 14.4% to 38% of individuals with mild cognitive impairment (MCI) who are identified through screening and receive interventions revert to normal cognitive functioning [3,4]. Consequently, screening tools with high specificity and sensitivity are critical. Current assessment methods, which primarily include physiological indicators, neuroimaging techniques, and questionnaires, are often invasive, costly, and time-consuming [5]. In contrast, some digital tools (eg, voice based or virtual reality [VR]) show substantial clinical promise by addressing these shortcomings [6,7]. However, these new tools still have limitations (eg, cybersickness in VR and accent-related transcription errors) and require further exploration [5,6,8].

Impairment of working memory is a hallmark early symptom of dementia [9]. Working memory is a limited-capacity cognitive system that provides temporary storage and manipulation of information to support a wide range of complex activities such as reasoning, comprehension, and learning [10]. Neuroimaging studies have demonstrated altered activation in working memory–related brain regions during cognitive tasks in the early disease continuum [11,12]. However, translating these findings into early diagnostic tools remains a challenge as it hinges on deconstructing working memory into its measurable component processes. The multicomponent model of working memory by Baddeley [10] offers a simple yet robust framework that makes such deconstruction possible. This proposed framework comprises 4 core components: the central executive system, phonological loop, visuospatial sketchpad, and episodic buffer [13]. External sensory input is initially handled by the phonological loop and visuospatial sketchpad. The episodic buffer then binds this information with existing knowledge from long-term memory. Finally, the central executive system acts as the control system, achieving cognitive processes by regulating attention, updating information, and inhibiting irrelevant information [13]. Previous research has reported that inhibitory control represents the most frequently impaired executive domain in MCI [14]. Indeed, the central executive system has been confirmed as an interconnected network where inhibitory control relies on attention, inhibition, and other functions that are integrated across neural networks in an interactive manner [14]. Furthermore, the association between working memory capacity and higher-order cognitive abilities is more pronounced, with these differences largely attributed to attentional capacity [15]. Therefore, researchers believe that access to working memory is governed by inhibitory attention processes [16].

The go/no-go task is widely used as a measure of inhibition, attention, and impulsivity [17]. By designing tasks that require participants to respond to specific stimuli (“go”) and withhold responses to “no-go” stimuli, which assesses the ability of executive control, extensive behavioral and neuroimaging studies have demonstrated the neurobiological foundation of the go/no-go task [17-19]. Cognitive control–related tasks consistently elicit increased frontal midline θ power, which is a key neural marker for cognitive control, response inhibition, conflict monitoring, and attention allocation [18]. The increase in frontal midline θ power during go/no-go tasks is driven by both inhibitory and noninhibitory (attention-related) processing [19]. Furthermore, rapid response inhibition activates the prefrontal cortex (PFC), particularly the middle and inferior frontal gyri, as well as the parietal and temporal lobes and their functional networks [20]. Abnormalities in these regions are considered predictive of cognitive status [20]. Significantly, the PFC is particularly vulnerable to aging [21]. Some researchers have even proposed the frontal or executive decline hypothesis, suggesting that age-related cognitive decline may stem from damage to the prefrontal lobe and executive functions [22]. Therefore, the go/no-go task is an experimental paradigm that effectively reflects executive control and has been reported to be a potential practical early screening tool for cognitive decline [23,24].

According to the model by Baddeley [10], processing stimuli involves a multisensory operation that necessitates cross-modal integration within the episodic buffer. However, go/no-go tasks predominantly use visual stimuli [25]. Given that sensory function decline is a natural consequence of aging, relying on a single sensory modality may introduce measurement bias. Previous research has indicated that multisensory integration accelerates reaction time (RT) in older adults, potentially serving as a compensatory mechanism to enhance brain information processing efficiency [26]. Beyond facilitation effects, multisensory processing exhibits illusory effects such as the McGurk effect and the double-flash illusion [27]. These illusory effects are mediated by anatomical and functional connections between different brain regions, particularly the cross-modal associations involving neural resources in the PFC [28]. Cross-modal information is represented within the capacity-constrained episodic buffer, where more complex inputs entail the binding of a greater number of information sources, thereby elevating processing load [29]. Moreover, multisensory integration can be affected by top-down factors, also termed endogenous attention [30]. Endogenous attention requires purposeful and effort-intensive cognitive processing, whereas multisensory integration performance may exhibit instability or illusions under high–cognitive load conditions.

Semantic memory is a key component of long-term memory. Its integration with working memory occurs in the episodic buffer, followed by complex processing in the central executive system to support motor behavior. Prior research suggests that working memory is not related to basic sensory processing (eg, color or shape discrimination) but shows a reliable association with semantic tasks [31]. A meta-analysis revealed that semantic deficits constitute a crucial and early feature of MCI and should be incorporated into routine clinical assessments [32]. Furthermore, the complexity of conceptual semantic representations systematically influences inhibitory processing [33]. However, most published go/no-go studies measure perceptual discrimination of stimuli such as shapes or numbers, with little attention paid to changes in abstract concepts [34]. This neglect may result in insufficient sensitivity of go/no-go tasks in capturing early and subtle cognitive decline.

To sum up, based on the theoretical framework by Baddeley [10] and existing empirical evidence, we hypothesize that combining visual-auditory fusion and semantic categorization with the go/no-go task can heighten demand on central executive resources, which might lead to a better identification of cognitive decline. The aim of this study was to develop an integrated audiovisual-semantic go/no-go task and evaluate its performance in discriminating cognitive decline, as well as the task’s acceptability among older adults.

Methods

Study Design and Participants

This cross-sectional pilot study was conducted to preliminarily investigate and develop a digital tool integrating audiovisual-semantic stimuli for the detection of cognitive decline. From April 2025 to August 2025, a total of 156 participants aged 60 years and above were recruited from 7 communities in Hangzhou, China. Exclusion criteria were (1) individuals who had significant verbal and hearing impairments and were unable to see or hear the stimuli despite adjusting the volume or image size (by controlling the resolution in the E-Prime software [Psychology Software Tools]) and (2) those with major depression and other psychiatric disorders.

Data Collection

Overview

Data collection was conducted during a single session. All participants or their legally acceptable representatives signed a written consent form agreeing to take part in this study. After that, all participants underwent face-to-face interviews conducted by a trained staff member to collect the demographic information (eg, age and gender). We administered the Montreal Cognitive Assessment (MoCA) and the go/no-go tasks in an interleaved and alternating manner using a number-calling strategy, with at least a 30-minute interval between the 2 assessments. The MoCA assessment was conducted by trained research personnel.

Cognitive Assessment

The MoCA is a well-validated, multidomain, paper-and-pencil test that assesses a range of cognitive functions, including visuospatial and executive functions, naming, memory, attention, abstraction, and orientation [35]. We defined participants’ cognitive status using the following MoCA cutoff scores (adjusted education; the MoCA should be scored based on educational level to reduce bias. According to the standard MoCA scoring rules, one point is added to the total score for participants with less than 6 years of education): a score of 23 or higher was categorized as no cognitive impairment (NCI), a score ranging from 14 to 22 was categorized as MCI, and a score of less than 14 was categorized as dementia [5].

The Modified Go/No-Go Task

The task design was built on previous research and finalized through consensus among all authors [24]. This study comprised 6 go/no-go tasks (Figure 1), including visual, auditory, and audiovisual integration tasks. Every task consisted of 5 runs of 16 trials each (80 total trials). The total completion time for the 6 tasks was approximately 18 minutes (170 seconds per task). Participants were allowed a 1-minute break after every task. The tasks were conducted in a quiet environment (≤45 dB), verified using a decibel meter (model DLX-PS02303; Delixi Electric). Participants sat at approximately 58 cm from a 14-inch laptop computer (ThinkPad T480s-i7; Lenovo). Auditory stimuli were presented through wired headphones (model HD350BT; Sennheiser Electronic SE & Co). Volume was adjusted to a level clearly audible to participants before testing commenced. Visual materials were sourced from a canonical set of line drawings [36], and auditory materials were obtained from the Aigei website. As prior research has suggested that humans are more sensitive to biological sounds compared to nonbiological sounds, the auditory stimuli were designed as animal sound recognition tasks [37]. Visual and auditory stimulus materials underwent processing to standardize the image size and sound pressure level before being inserted into E-Prime (version 2.0). The sequence of stimuli in each task was fixed. A single-key keyboard (model K1-LY-WX; Yingyu IoT) was connected to the computer and positioned in front of the participant’s dominant hand. Each stimulus was presented for 500 ms, with a 1500-ms intertrial interval during which a blank screen with a fixation cross was displayed. Each task was preceded by 12 practice trials at the same pace as the formal study trials (go/no-go ratio of 3:1) to ensure participant comprehension of task requirements. The go/no-go metrics evaluated were as follows:

• RT: mean response time for correct “go” trials
• “Go” trial accuracy: percentage of correct responses on “go” trials
• False alarm rate: percentage of incorrect responses on “no-go” trials
• Inverse efficiency score: RT divided by the overall proportion of correct responses

‎

A lower score reflects better inhibitory control by integrating speed and accuracy. All go/no-go metrics were standardized into Z-scores before performing regression analysis.

Informal Interview

Following completion of the MoCA and modified go/no-go tasks, we conducted informal interviews lasting approximately 3 to 5 minutes with participants. Semistructured questions included the following: “How did you feel about the overall testing process of Go/No-go?” “How was your experience with the Go/No-go task compared with the MoCA questionnaire?” “How do you feel about implementing this Go/No-go task for cognitive decline screening in a community setting?” Interviewers recorded key content words related to feelings, with each unique word recorded only once per participant.

Data Analysis

Statistical analyses were performed using SPSS (version 31; IBM Corp). As the data in this study did not pass normality tests, medians and IQRs were used for description. Kruskal-Wallis tests were used for demographic characteristics and go/no-go task–related metric analyses. Logistic regression analyses were conducted while controlling for age and educational level (P<.05 in both cases). We used backward likelihood ratio logistic regression analysis, with a P value of .05 for entry and .10 for removal. Backward likelihood ratio is considered more robust based on overall model fit changes [38]. The features extracted from the task were entered into a logistic regression model, where the diagnostic group (NCI vs cognitive impairment, MCI, or dementia and MCI vs dementia) was considered as the dependent variable and the task-extracted features were used as the independent variables.

To examine go/no-go diagnostic validity, receiver operating characteristic (ROC) curves were used to establish area under the curve (AUC) values for the following groups: NCI vs cognitive impairment (cognitive impairment including MCI and dementia), NCI vs MCI, NCI vs dementia, and MCI vs dementia. The optimal Youden index was used to calculate discrimination metrics, including sensitivity and specificity. AUC 95% CIs were calculated using bootstrap methods. To compare the discriminative validity of different tasks, additional ROC analyses were performed using the DeLong method. To analyze the qualitative data, the R package wordcloud2 (R Foundation for Statistical Computing) was used to generate word clouds.

Ethical Considerations

All procedures were approved by the Ethics Committee for Human Studies of the Fourth Affiliated Hospital of Zhejiang University School of Medicine (2025-0326-1-F1). Written informed consent was obtained from all participants prior to measurements, in accordance with the Declaration of Helsinki. No identifying details are included in this paper or the supplementary materials. Participants did not receive compensation. Access to raw data is restricted to research team members.

Results

Characteristics of the Participants

A total of 156 participants took part in this study, including those with NCI (n=75, 48.1%), MCI (n=66, 42.3%), and dementia (n=15, 9.6%). Baseline characteristics (Table 1) showed significant differences among the 3 groups in age, educational level, and MoCA scores (P<.001). No statistically significant differences were observed in gender or smoking or alcohol consumption habits across the 3 groups.

Task Performance and Logistic Regression

The results for task performance and logistic regression are shown in Table 2, Multimedia Appendix 1, and Figure 2.

‎

Across all 6 go/no-go tasks, performance on the RT, “go” trial accuracy, inverse efficiency score, and false alarm rate metrics demonstrated a significant declining trend from NCI to dementia (Table 3), with no collinearity issues among metrics (variance inflation factor<10). Compared with participants with NCI, participants with both MCI and dementia exhibited slower RTs, lower accuracy, higher omission rates, and poorer inhibitory control (Table 3).

ROC Results

As shown in Table 2 and Figure 2, we examined the discriminatory ability of different tasks for varying degrees of cognitive decline. The AUC ranges for the diagnostic comparisons were 0.877 to 0.926 for NCI vs cognitive impairment, 0.861 to 0.916 for NCI vs MCI, 0.909 to 0.968 for NCI vs dementia, and 0.724 to 0.855 for MCI vs dementia. Considering accuracy, sensitivity, specificity, and AUC simultaneously, tasks 5 and 6 demonstrated optimal performance in discriminating NCI from cognitive impairment compared to other discrimination models. Task 5 exhibited the most stable performance in discriminating NCI from cognitive impairment, MCI, and dementia (accuracy: 0.837-0.933; sensitivity: 0.742-0.933; specificity: 0.606-0.933; AUC: 0.744-0.977).

In the DeLong test for paired ROC curves, the AUC of task 5 was significantly higher than that of task 2 in discriminating NCI from cognitive impairment (Z=–2.129; ΔAUC=–0.048, 95% CI –0.093 to −0.004; P=.03) and from MCI (Z=–2.182; ΔAUC=–0.055, 95% CI –0.104 to –0.006; P=.03). The AUC of task 2 significantly outperformed task 1 when discriminating NCI from dementia (Z=–2.05; ΔAUC=–0.049, 95% CI –0.095 to –0.002; P=.04).

Acceptability

In informal interviews, “fun/interesting” emerged as the most frequently mentioned term (76/156, 48.7%), followed by “thinking” (65/156, 41.7%) and “high-tech vibe” (61/156, 39.1%). Participants also expressed perceptions of task difficulty, such as “easy” (57/156, 36.5%), “lagging” (43/156, 27.6%), and “difficult” (13/156, 8.3%). When comparing with standardized scales, participants mentioned the terms “engaging” (43/156, 27.6%) and “novelty” (30/156, 19.2%), with practical application perceptions including “convenient” and “roll out” (Multimedia Appendix 2).

Discussion

Principal Findings

Overview

This study provides empirical evidence that the increased cognitive load imposed by integrating visual-auditory stimuli and semantic categorization into a go/no-go task effectively amplifies performance differences across cognitive states, thereby supporting its utility as a discriminative tool. This novel approach indirectly assessed the episodic buffer’s ability to form coherent episodic representations, enabling early detection of cognitive decline. Our findings showed that the task complexity incrementally increased from unimodal stimuli (visual or auditory) to multimodal stimuli (audiovisual integration) and from a basic level (concrete objects such as “dog”) to a superordinate level (abstract categories such as “animal”), consistent with our hypothesis. Older adults with cognitive decline exhibited progressively worse performance in these incrementally difficult tasks.

Measuring Using Multimodal Stimuli

Unimodal tasks (task 1 and 3), which relied on basic sensory discrimination, did not demonstrate superior overall performance in discriminating between cognitive statuses. Such an observation is likely due to dependence on automatic processing and low-level perception, yielding similar performance across groups. In contrast, both cross-modal matching tasks (tasks 5 and 6) performed well; in particular, they outperformed the auditory-only MoCA-attention cognitive domain (AUC=88.7%, 95% CI 88.0%‐89.4%), suggesting that multisensory integration is a reliable biomarker for cognitive impairment [39]. This is supported by findings from The Irish Longitudinal Study on Ageing, which reported broad associations between multisensory integration and multiple cognitive domains rather than links to any specific subdomain [40]. This implies that measuring multisensory integration could serve as a window into global cognition. Furthermore, as noted previously, multisensory stimulation may serve as a compensatory strategy for older adults, enhancing behavioral performance in those with NCI [26]. However, this compensatory mechanism is limited as behavioral responses are governed by the central executive system, which may account for the more pronounced deficits observed in individuals with progressive cognitive decline. Bayesian integration theory further supports this premise, proposing that perceptual reliability weighting affects audiovisual integration. In other words, the brain optimally combines sensory inputs based on their relative reliability, but cognitive decline may change this weighting process [41]. Apart from that, some scholars regard temporal synchrony and functional connectivity as 2 primary mechanisms that facilitate audiovisual integration [42]. The integration time window posits 2 sequential, interdependent stages: initial parallel peripheral processing of stimuli followed by central processes involving audiovisual integration and motor response initiation [43]. This process may help explain impairments observed in cognitive decline. Previous research indicates that patients with MCI and Alzheimer disease exhibit delayed audiovisual integration compared to older adults with normal cognitive function, with MCI populations perceiving more sound-induced flash illusions [27]. However, patients with MCI and Alzheimer disease experience attentional deficits, and these patients become distracted by irrelevant information, demonstrating impaired attention-switching speed [44]. This aligns with the theory by Baddeley [10]. Impaired performance of the episodic buffer may contribute to working memory deficit, particularly in cognitively declining populations [10]. Therefore, we propose that assessment methods involving multisensory processing can evaluate central executive performance and better identify cognitive decline.

Task Complexity

Task 5, rather than the more cognitively demanding task 6, exhibited the most stable discriminative performance across cognitive states. This suggests that excessive complexity may overwhelm the limitations of the episodic buffer and central executive system. A study indicated that older adults tend to adopt conservative action strategies when performing complex tasks, reflecting speed-accuracy trade-offs. Specifically, they adjust their decision boundaries to increase response caution, which results in prolonged RT as a means to enhance or maintain accuracy [45]. However, a study by Chen et al [46] showed different results, with MCI and dementia populations potentially exhibiting random performance without deliberation, resulting in shorter RTs when compared to normal groups. Nonetheless, the adaptive training trials at the beginning of each task in this study could, to some extent, strengthen participants’ understanding of the tasks and avoid performance variability. However, such training for task 6, which placed the highest demands on central executive resources, may still have been insufficient, leading participants to adopt response strategies that influenced the outcomes. According to the Yerkes-Dodson law (an arousal-modulated mechanism) [47], task 5 likely reaches the peak of the inverted U-shaped curve, enhancing arousal and performance discrimination, whereas the increased cognitive load imposed by task 6 exceeds this optimum, resulting in overload and diminished utility. This may also explain why task 5 demonstrated greater stability than task 6. The findings further suggest that task difficulty must be carefully calibrated to avoid overloading the episodic buffer, which could undermine measurement consistency. Additionally, the fixed task order (task 1 to task 6) used in this study means that the decline in performance on task 6 may be partially attributable to time-on-task effects, manifested as decreased attention and less effective decision-making [48]. Previous research has shown that tasks shorter than 15 minutes are unlikely to induce significant fatigue, whereas subjective fatigue increases after 20 minutes of task engagement [49]. Although rest breaks were provided between tasks, this potential confounder should be considered when interpreting the results. Future studies should use task randomization or repeated measurements to rule out this alternative explanation.

Prospective Clinical Application

At present, the go/no-go task is typically used as an experimental paradigm in conjunction with other objective indicators and has not been widely adopted for screening purposes. In contrast, traditional instruments such as the Mini-Mental State Examination and MoCA require 5 to 15 minutes for administration [24,35]. These traditional tools require trained assessors and additional time and workload and could introduce potential bias from personnel differences [5]. Regarding diagnostic accuracy, our tool demonstrated a performance comparable to that of emerging digital screening instruments, including VR and voice-based tools (AUC=0.83-0.89; sensitivity=0.79-0.83; specificity=0.67-0.91) [7,8,50]. Nevertheless, in comparison with these tools, our novel tool effectively retains the diagnostic benefits of multisensory stimulation while eliminating adverse physical effects and high scene production costs [5-7].

Notably, after accounting for the influence of educational level, the tasks retained their independent predictive value. Within the tasks with independent predictive value, task 5, as the best-performing task, can be completed within 3 minutes. Thus, the use of task 5 as a stand-alone tool for efficient and rapid screening in community settings is highly promising. Task 5 can be integrated into various digital platforms (eg, smartphones or tablets) to quickly flag at-risk individuals for follow-up assessments and can also be deployed as a self-administered tool, thereby enabling timely interventions in resource-limited environments.

Results from the informal interview with community-dwelling older adults showed a positive perception of the modified go/no-go task, with nearly half of the participants describing it as interesting. This favorable experience is important for promoting community-based screening and aligns with previous reports of older adults’ increasing willingness to use digital tools [51]. Additionally, participants’ subjective evaluations of task difficulty were consistent with the study sample proportions of participants with NCI and dementia, indicating that the task’s difficulty gradient was perceptible.

Limitations

The uneven distribution of participants with dementia, which accounted for 9.6% (15/156) of the total sample, might lead to wide CIs and present a risk of overfitting. This may be related to our community-based data collection. This potential bias calls for future validation studies expanding the sample size of older adults with dementia. Additionally, this study did not repeat measures for test-retest reliability. Therefore, we are currently conducting a follow-up study to fully assess the reliability of the modified go/no-go task. Furthermore, as a cross-sectional study, our findings demonstrate associations but cannot determine the tool’s predictive validity for future dementia onset, which requires longitudinal investigation.

Conclusions

This study modified and validated a novel go/no-go task incorporating audiovisual stimuli and semantic categorization as an effective tool for early detection of cognitive decline. Positive participant feedback on engagement and novelty supports its feasibility. The applicability and reliability of the modified go/no-go task warrant further investigation in a larger-scale community-based setting.