The dataset used in this study is derived from the ADReSSo 2021 Challenge and consists of speech recordings from a picture-description task [8], where cognitively normal participants and individuals diagnosed with AD were asked to describe the Cookie Theft image from the Boston Diagnostic Aphasia Examination [17,18]. In total, the dataset contains 237 recordings, split into a 70/30 train–test partition with demographic balancing, resulting in 166 samples for training and 71 for testing. Within the training set, there are 87 AD and 79 non-AD (healthy control) recordings.

A key strength of ADReSSo is its focus on reducing common sources of bias in dementia screening benchmarks. The dataset was carefully curated to mitigate confounds such as repeated speech samples from the same individual, variation in recording quality, and imbalances in age and gender. Demographic matching was performed using a propensity-score procedure as described by Luz et al [8], yielding standardized mean differences below 0.001 for both age and gender, which helps ensure that downstream model performance reflects disease-related signal rather than demographic artifacts.

We preprocess the ADReSSo picture-description recordings to obtain aligned audio, text, and image representations, which serve as inputs to our multimodal fusion architecture. For the audio modality, we extract embeddings using Wav2Vec 2.0 [19], a self-supervised speech representation model trained to learn rich acoustic features directly from raw waveforms. Given an input recording, we obtain a sequence of frame-level hidden states and apply mean pooling over time to produce a fixed-dimensional 768-D audio embedding per sample.

For the text modality, we first transcribe each recording using OpenAI Whisper [20], which provides robust automatic speech recognition for spontaneous speech. To better preserve clinically relevant phenomena commonly observed in AD speech, we supply an initial decoding prompt (“Umm, Uhh, let me think like, hmm... Okay, here’s what I’m, like, thinking.") that encourages retention of fillers, disfluencies, repetitions, and incomplete phrases, rather than normalizing them away. We then encode the resulting transcripts using ModernBERT [21], a modern bidirectional Transformer encoder optimized for producing strong sentence-level representations. Similar to the audio pipeline, we derive a fixed-length 768-D text embedding for each transcript, providing a compact semantic representation that captures content, coherence, and lexical organization.

For the image modality, we represent the Cookie Theft stimulus using contrastive language–image pretraining (CLIP; ViT-L/14), a vision-language model trained via contrastive learning to align visual and textual representations in a shared embedding space. We extract the image encoder output and use a 768-D image embedding, which provides a high-level description of the visual scene structure and salient objects.

A central component of our fusion approach is bidirectional cross-attention, where each modality is updated by attending to the others, allowing text, audio, and image embeddings to exchange information in both directions rather than being combined through simple concatenation. Unlike simple fusion methods (eg, concatenation), cross-attention allows the model to learn which parts of one representation are most relevant given the other representation, which is especially important in dementia screening where different modalities may carry complementary cues. Formally, we use the standard attention operation represented by Attention(Q,K,V)=softmax(QKTdk)V, where Q (queries) represents what we want to “look for,” while K (keys) and V (values) represent the information we want to retrieve from.

Intuitively, bidirectional cross-attention can be understood as a 2-way exchange between modalities. In one direction, the text representation asks which parts of the audio or image are most relevant for interpreting what was said; in the other direction, the audio or image representation is also updated by attending back to the text. This allows the fused representation to reflect not only the content of the spoken description, but also how it was delivered and how well it aligns with the shared Cookie Theft picture context. In practice, the bidirectional design helps the model avoid treating one modality as fixed context for another and instead encourages reciprocal refinement of the modality embeddings across layers.

To combine information from text, image, and audio in a way that goes beyond simple concatenation, we propose an attention-based fusion module (Figure 1) that learns how modalities complement each other when predicting AD status and cognitive outcomes. The key idea is that each modality carries different but related information: text reflects linguistic content and coherence, audio captures acoustic patterns such as hesitation and fluency, and the image provides the visual context that grounds the picture-description narrative. Rather than assuming each modality contributes equally, our fusion approach learns to emphasize the most informative signals dynamically.

Our architecture performs multimodal fusion using bidirectional cross-attention, enabling each modality to iteratively incorporate information from the others. Rather than simply concatenating embeddings, the model allows the text representation to attend to complementary cues from the image and audio, while the non-text modalities also attend back to the evolving text representation. This bidirectional exchange supports richer alignment between what is said, how it is spoken, and the visual context of the picture-description stimulus. As a result, the fusion module produces a single multimodal embedding that reflects both modality-specific evidence and cross-modal dependencies, which is then used for downstream AD classification and its severity assessment.

For the implementation, we use a multilayer attention block that alternates between the 2 inputs, allowing one modality to “look at” and incorporate information from the other. Intuitively, this helps the model learn cross-modal interactions. For example, how acoustic hesitation patterns relate to lexical choice or how a visually grounded narrative differs between cognitively normal and impaired speakers. The final output is a single fused embedding per sample, which is passed to downstream classifiers or regressors for AD detection and its severity assessment.

We train the proposed trimodal fusion model using the ADReSSo training split and further reserve 20% of the training data as a validation set for model selection and early stopping. This validation split is created randomly but reproducibly using a fixed seed of 42, ensuring consistent comparisons across experiments. To enforce reproducibility, the same seed is applied across Python, NumPy, and PyTorch (central processing unit and Compute Unified Device Architecture), with deterministic CUDA deep neural network behavior enabled. During training, samples are loaded in batches of 32. The fusion backbone uses a shared projection dimension of 768, with 4 attention heads and 3 cross-attention layer pairs per fusion stage, and models are trained for up to 50 epochs with a dropout rate of 0.1. To reduce scale mismatch across modalities, we apply L2 normalization to the input embeddings prior to fusion.

The model parameters are optimized using AdamW with a learning rate of 0.0001 and weight decay of 1 × 10⁻⁴. For AD detection, we minimize cross-entropy loss, and for AD severity assessment, we use mean squared error loss. To stabilize learning, we use a ReduceLROnPlateau scheduler that monitors validation loss and reduces the learning rate by a factor of 0.5 after 2 consecutive epochs without improvement. We additionally apply gradient clipping with a maximum norm of 1.0. Early stopping is based on validation loss with a patience of 5 epochs, and the model weights from the epoch with the lowest validation loss are restored for final evaluation and embedding extraction.

The AD detection task is formulated as a binary classification problem, where the goal is to distinguish between participants with AD and cognitively normal (non-AD) participants using picture-description responses from the ADReSSo dataset. Because dementia-related signals can manifest across multiple modalities, we evaluate models using (1) audio-only representations, (2) text-only representations, and (3) multimodal fusion representations derived from combinations of text, audio, and image. Audio representations are extracted from Wav2Vec 2.0 embeddings, text representations are extracted from ModernBERT embeddings computed over Whisper transcripts, and image representations are extracted using CLIP ViT-L/14. We primarily use 3 standard machine-learning models for the AD classification task. These include support vector classifier (SVC), random forest (RF), and logistic regression (LR), all of which are implemented from the scikit-learn package [22-25].

In addition to AD classification, we evaluate our approach for AD severity assessment by predicting MMSE cognitive score, formulated as a regression problem. The objective is to predict a participant’s cognitive score directly from their picture-description response, enabling a finer-grained estimate of cognitive status beyond binary diagnosis. Similar to the classification setting, we investigate the predictive use of multiple modalities by training models with audio-only, text-only, and image-only representations, as well as multimodal fusion representations that combine complementary cues across modalities. Specifically, we use support vector regressor (SVR), random forest regressor (RFR), and ridge regression (Ridge) as our machine learning models [23-26]. For this task, we use the same feature representations described in the preprocessing pipeline: Wav2Vec 2.0 embeddings for audio, ModernBERT embeddings derived from Whisper transcripts for text, and CLIP ViT-L/14 embeddings for the image stimulus. We compare our proposed attention-based trimodal fusion model against unimodal baselines and bimodal fusion models.

We evaluate performance separately for the AD classification and AD severity assessment tasks. For the binary classification setting, we report accuracy, precision, recall, and F₁-score, providing a balanced view of overall correctness as well as performance on the positive AD class. Additionally, we also show the receiver operating characteristic (ROC) curve for modality comparisons and report the area under the ROC curve. For the regression task, we quantify prediction error using root mean squared error (RMSE), which measures the average magnitude of score prediction deviations. For all classical machine-learning baselines, optimal hyperparameters are selected using grid search with 5-fold cross-validation performed on the training split only. This ensures that model configuration is tuned in a statistically robust manner while keeping the held-out test set strictly reserved for final evaluation.

To better characterize uncertainty in model performance, we additionally report 95% CIs for the major classification and regression results using nonparametric bootstrap resampling of the held-out test set. For each metric, CIs were estimated from the empirical bootstrap distribution across 1000 repeated resamples. Note that the observed performance differences were not tested for statistical significance to avoid overreliance on arbitrary significance thresholds (eg, P<.05) and potential misinterpretations of P values.

To assess the impact of reciprocal information exchange during multimodal fusion, we compare 2 attention configurations under an otherwise identical fusion framework. The cross-attention-only, namely standard cross-attention, performs fusion using a single cross-attention direction, where 1 modality attends to another to incorporate complementary information. In contrast, the bidirectional cross-attention applies cross-attention in both directions, allowing each modality to be updated using information from the other in a sequential manner. Both variants use the same pretrained embeddings, projection dimensionality, optimization settings, and train or validation split, enabling a controlled comparison of the attention mechanisms.

To assess whether the proposed attention-based multimodal fusion provides benefits beyond standard approaches, we compare it against 2 widely used conventional fusion strategies for the AD classification task: early fusion and late fusion. These baselines are commonly adopted in multimodal machine learning due to their simplicity, but they often struggle to capture complex interactions between modalities. For early fusion, we construct a single feature vector by concatenating the modality-specific embeddings (text, audio, and image) and train a classifier directly on the combined representation. While straightforward, early fusion does not explicitly model cross-modal relationships and instead relies on the downstream classifier to learn interactions implicitly from the concatenated embedding.

For late fusion, we train separate unimodal classifiers and combine their predictions at the decision level using majority voting. Because the visual stimulus is identical across all samples in the ADReSSo picture-description setting, an image-only classifier would not be meaningful for late fusion. Therefore, we report late-fusion performance using only the text and audio unimodal models. This comparison helps isolate the benefit of our attention-based fusion, which can jointly reason over modalities and learn cross-modal dependencies rather than combining independent unimodal decisions.

In addition to these conventional fusion strategies, we include a neural network baseline trained on the concatenated multimodal embeddings. The neural network architecture consisted of 2 hidden layers with 512 and 128 nodes, respectively, using the rectified linear unit activation function. This funnel-like bottleneck design by progressively stepping down from the high-dimensional input space to 512 and then 128 nodes was chosen to systematically compress the multimodal features. This encourages the network to learn robust, high-level representations while mitigating the risk of overfitting on the relatively small dataset [27]. The network was optimized using the Adam solver with an initial learning rate of 0.001. To further mitigate overfitting and ensure optimal generalization, we implemented early stopping. Specifically, 10% of the training data were set aside as a validation set, and training was terminated if the validation score did not improve by at least 1 × 10⁻⁴ for 10 consecutive epochs. This baseline provides a stronger learned comparison than linear or tree-based classifiers on early-fusion features, while still operating on the same underlying embedding inputs. In this way, we can assess whether any observed gains arise simply from using a more flexible classifier on concatenated features or from the proposed bidirectional fusion mechanism itself.

The ADReSSo Challenge data used in this study are available through DementiaBank with approved credentials. The studies involving human participants were reviewed and approved by the DementiaBank consortium. All enrolled participants provided informed written consent to participate in this study. All data analyses in this work were conducted using deidentified data.

Figure 2 compares ROC curves across bimodal fusion settings and the proposed trimodal multimodal fusion model for AD classification. Among the bimodal approaches, text + audio achieves the highest discrimination performance (area under the receiver operating characteristic curve [AUC]=0.8714), followed closely by text + image (AUC=0.8595), while audio + image performs notably lower (AUC=0.7937). The proposed multimodal fusion model yields the strong overall ROC performance with an AUC of 0.9032, indicating improved separability between participants with AD and without AD relative to all bimodal alternatives.

Table 6 compares the proposed multimodal fusion with conventional early fusion, late fusion, and a neural network baseline trained on concatenated embeddings. For SVC, both early fusion and late fusion achieve an accuracy of 0.8169, with F₁-scores of 0.7937 and 0.8000, respectively. The neural network baseline improves this comparison to an accuracy of 0.8311 and an F₁-score of 0.8099. In contrast, the proposed multimodal fusion achieves a higher accuracy of 0.8722 and a higher F₁-score of 0.8667. A similar pattern is observed across the other classifier families, indicating that the advantage of the proposed fusion is not limited to comparison against only simple conventional baselines.

We evaluate 2 fusion variants to quantify the impact of the bidirectional cross attention. Across all 3 classifiers (Table 7), the bidirectional cross-attention configuration achieves stronger overall performance than cross-attention only. For SVC, cross-attention alone attains an F₁-score of 0.8571 with accuracy of 0.8732, while our bidirectional cross attention improves results to an F₁-score of 0.8667 and an accuracy of 0.8722. A similar trend is observed for LR. For RF, the cross-attention only baseline achieves an F₁-score of 0.8125 (accuracy=0.8309), and the bidirectional cross-attention variant improves to an F₁-score of 0.8371 (accuracy=0.8437). Overall, bidirectional cross-attention produces balanced AD classification performance metrics.