Handwriting in Mild Cognitive Impairment: Reliability Assessment and Machine Learning–Based Screening

Introduction

In the last decades, the world has witnessed progressive aging. This trend brings about various challenges at the social, economic, and health care levels, as it directly affects citizens’ health, compromising several aspects of their daily lives [1]. First, older individuals experience greater difficulty in responding to external stimuli, a condition that leads to isolation and inertia. Second, a gradual, albeit sometimes slow, decline is observed, causing a state referred to as “frailty” [2]. In this state, the individual becomes more susceptible to motor-related accidents, resulting in disability and an increase in hospitalization. Furthermore, aging is linked to a higher prevalence of neurodegenerative diseases, for which treatments that slow down the progression of the disease are the only available interventions.

Among these disorders, dementia is one of the most prevalent, being characterized by a decline in cognitive function, memory loss, and changes in behavior [3]. Dementia leads to a gradual loss in the ability to perform daily tasks independently, causing patients to often require round-the-clock care and supervision. This can translate into emotional and physical toll on family members and caregivers, who may experience stress, burnout, and financial strain. The health care systems are not immune to the negative consequences of the disorder, since it is associated with extensive long-term care, medications, and support services [4].

Clinically, a transitional phase between the healthy state and dementia has been identified and denoted as mild cognitive impairment (MCI) [1]. MCI can manifest heterogeneously. While patients with amnestic MCI experience memory-related cognitive loss, participants diagnosed with nonamnestic MCI face impairments in the domains of attention, language, and visuospatial ability [5]. From an epidemiological standpoint, the aggregate prevalence of MCI among older people ranges from 15% to 20% [6-9]. The problem lies in the relevant yearly percentage (up to 15%) of cases that are diagnosed with dementia after having previously been in the MCI state. This underscores the need for MCI early screening, aimed at preventing the progression to dementia [10]. Unfortunately, the broad spectrum of symptoms and the typically nonsignificant impact on the participants’ independent life make the detection of MCI quite challenging.

Cognitive tests are typically used in clinical practice for this purpose since they are quicker and cheaper than imaging techniques [11,12]. The existing tests, all conducted under the clinician’s supervision, vary in the assessed cognitive domains (eg, memory, language, attention) and in the tasks proposed to the individual, but are in general capable of providing a snapshot of one’s cognitive status. The most common are the Mini-Mental State Examination (MMSE) [12], the Montreal Cognitive Assessment (MoCA) [13], and the clock drawing test (CDT) [14]. Despite being largely used, such clinical tests experience a few drawbacks. As they depend on the clinician administering the test, interrater reliability and subjectivity in the assessment are a concern. Moreover, to assess all the relevant cognitive domains, a combination of different tests should be considered, thus leading to increased administration time. Finally, the scales are usually characterized by low granularity, thus making it difficult to establish universally accepted cutoff scores. Thus, there is a need for a cheap, noninvasive, objective solution to complement the standard clinical procedure for MCI screening [6].

In this sense, the recent literature claimed the importance of handwriting: given its complexity, it can serve as a biomarker for cognitive decline [15]. This is because handwriting is the result of the interaction between various cognitive processes: attention, language, short and long-term memory, motor planning, visual-spatial skills, in-hand manipulation, fine motor control, and sensory awareness of the fingers [16]. With the goal of characterizing the handwriting gesture of patients with MCI or with dementia with respect to healthy participants, several studies have been conducted. In Werner et al [17], a group of 41 controls, 31 patients with MCI, and 22 patients with mild dementia were recruited and performed copying tasks of increasing complexity on a digitizer. Significant differences emerged, with both groups of patients showing increased execution times and reduced pressure during the motor task. Conversely, the Livescribe Echo Pen, used on the Livescribe paper sheet, was used in Kawa et al [16] to compare the handwriting of 37 controls and 37 patients diagnosed with MCI. Again, writing slowness emerged as a peculiar characteristic of the latter group. This was coupled with significantly higher and wider written traces. The extracted features were then used to build a linear discriminant classifier, achieving an accuracy of 70%. The combination of a specific digital pen and paper was also adopted in [18], investigating Chinese handwriting in patients affected by Alzheimer disease, for which MCI is a common precursor. The results confirmed the existing literature, as worse temporal measures, both on paper and in the air, were associated with the disease. A higher degree of pressure applied to the writing surface and a poorer control of the same were revealed as well. A classification study, aimed at distinguishing patients in the early stage of dementia from healthy participants, was conducted in [19]. Cursive loops were acquired with a WACOM digitizer, to then apply unsupervised clustering followed by a Bayesian-based classifier. The approach reached an accuracy of 74%. Execution speed, kinematics complexity, acceleration, and pressure were computed from the data recorded by a WACOM digital graphic tablet in [20]. Handwriting tasks, ranging from drawings to spontaneous sentence production, were performed on a paper sheet fixed onto the tablet surface by a group of 17 healthy participants (healthy control [HC]), 12 patients with MCI, and 23 patients with dementia (AD). Performance results ranged from 69.2% (HC vs MCI vs AD) to 96.6% (HC vs MCI). Pressure features were the most influential in the model output. In Chai et al [21], simple graphic tasks, going from repeatedly writing the letter “T” to drawing a pentagram, were performed by 39 HCs and 40 patients with MCI on a commercial tablet using a digital pen. After extracting 8 features, a binary support vector classifier achieved an F₁-score of 83.1%.

Given the evidence that emerged in the literature, this work aims to evaluate handwriting in patients with MCI by changing the paradigm in terms of the acquisition tool and the proposed handwriting tasks, with an eye on the feasibility of remote, ecological monitoring. Indeed, while able to provide relevant, objective data on the handwriting performance, limitations can be pointed out for the approaches found in the literature. Digitizers can still represent a barrier for older adults [22], who are typically the ones affected by MCI. Moreover, the writing execution on the digitizer surface, characterized by a reduced friction with respect to paper, is inevitably altered [23]. On the other hand, the methods proposed in studies by Kawa et al [16] and Qi et al [18] rely on the combination of a digital device and writing surface, potentially hampering their everyday employment. For these reasons, in this study, handwriting data were collected with a sensorized ink pen [24] that writes on normal paper. As for the tasks, the “Parole-non-Parole” test (PnP), aimed at assessing the origin of dysgraphic manifestations associated with MCI, was administered together with 2 unconstrained exercises, aimed at mimicking the natural handwriting one could perform in the domestic scenario. From the collected data, a series of indicators was extracted, and 3 specific objectives were investigated. First, the test-retest reliability of the indicators was assessed, both for the clinical PnP test and the 2 ecological tasks, to evaluate whether the proposed approach could be suitable for the longitudinal monitoring of handwriting in patients with MCI. Second, the indicators were examined in terms of their quantitative support for the clinical evaluation. This was done by assessing the indicators’ correlation with the scores of clinical tests. Moreover, the PnP was specifically analyzed to understand whether the indicators can be used to support the identification of lexical and phonological dysgraphia. Last, the unconstrained handwriting tasks were considered to build classification models to distinguish between patients with MCI and healthy participants, with the aim of enabling the ecological, domestic monitoring of handwriting as a screening tool for the first signs of cognitive decline.

Methods

Ethical Considerations

The protocol received the approval (ID 07_20/05/2021) from the Ethical Committee of the section IRCCS Fondazione Don Carlo Gnocchi of the Ethical Committee IRCCS Regione Lombardia. An information sheet for study participation was provided to all eligible participants, and informed consent was obtained from each of them.

Participants and Protocol

Participants were recruited by IRCCS Fondazione Don Carlo Gnocchi Milano. The inclusion criteria were (1) confirmed diagnosis of MCI according to the recommendations by Albert et al [25]; (2) being aged older than 65 years; (3) having completed elementary school; (4) having a Clinical Dementia Rating (CDR) scale [26] score less than or equal to 1; (5) obtaining negative neurological examination results for both lateral and extrapyramidal signs; (6) having trail making test A (TMT-A) results within the normal range for age and education according to the Italian calibration by Giovagnoli et al [27]; and (7) not having motor limitations that hinder task performance. Candidates were excluded in case of the presence of cognitive impairment of other origin (cerebral stroke, multiple sclerosis, and Parkinson disease), psychiatric disorders in either recent or remote anamnesis, severe sensory or intellectual deficits, and history of substance abuse or alcohol misuse. The recruited patients underwent clinical assessment, including MMSE, CDR, CDT, TMT-A, basic activity of daily living (B-ADL) [28], and instrumental activity of daily living (I-ADL) [29].

The sample size was determined by taking the assessment of the extracted indicators’ test-retest reliability as the main aim, since the scientific literature lacks similar works on patients with MCI. The only comparable study pertains to a systematic review that reports the psychometric properties of motor assessments (mobility, walking, balance, and functional performance) for patients with dementia, yielding intraclass correlation coefficients (ICC) between 0.42 and 0.99 [30]. On the other hand, the reliability of the proposed sensorized pen was demonstrated, the results showing 0.68<ICC<0.99, only in older adults without any diagnoses [24]. Given the scarcity of specific evidence in the population with MCI, a conservative approach was selected. Considering the minimum ICC value of 0.42 as a target, a statistical power of 80% and a dropout rate of 10%, the sample size was set to 40.

The protocol was administered under the clinician’s supervision in a test-retest fashion on the same day, with at least a 30-minute interval between the 2 repetitions (test and retest) of the protocol. The time interval was chosen to ensure that the participants were under the same conditions between test and retest. In this sense, potential learning effects were not a concern given the nature of the protocol (no memory-related tasks were administered). Each patient was provided with blank sheets of paper to perform three writing tasks in random order, 1 minute apart: (1) writing a grocery list of at least 6 items (List); (2) composing a content-free text of at most 7 lines (Text); and (3) executing the PnP test [31], where the participant wrote a series of 15 words under dictation. These included 7 regular words (simple and easily understandable Italian words: “tavolo,” “cartone,” “attore,” “sorella,” “postini,” “cuoceva,” and “democratiche”), 3 irregular words (requiring proper semantic and lexical comprehension of the Italian language to be written correctly: “cieco,” “nacquero,” and “conoscenza”), and 5 made-up (MU) words (words that do not exist in the Italian dictionary: “sabomi,” “fule,” “descia,” “sterpanzi,” and “getrunna”). The order in which words were presented was fixed across participants and protocol repetitions. Orthographic errors were then evaluated to assess the eventual presence of lexical dysgraphia, which is associated with errors in irregular words, or phonological dysgraphia, linked with errors in MU words.

Sensorized Ink Pen

The handwriting exercises were carried out using a sensorized ink pen [24], able to record linear acceleration and angular velocity on 3 orthogonal axes, together with the force applied on the pen tip with a sampling frequency of 50 Hz. The device was successfully used for the quantitative analysis of drawing tasks in patients with Parkinson disease [32] and demonstrated to be capable of recognizing healthy, older adults of different ages from unconstrained handwriting samples [33]. Data acquisition was controlled by the operator administering the protocol through a custom iOS app. On top of managing the start and stop of the recordings, the app allows users to store 4 different types of labels upon pressing buttons displayed on the user interface. Labels were used to tag the 3 types of words of the PnP test, so that they could be successively analyzed.

Data Analysis

The data recorded by the sensorized ink pen were elaborated in MATLAB R2022b (MathWorks, Inc) to extract 106 indicators measuring heterogeneous characteristics of the handwriting gesture, namely the temporal performance, the force patterns applied on the writing surface, the movement fluency, the pen inclination with respect to the vertical axis (tilt), the high frequency oscillations both in the domains of time and frequency. The complete description of the indicators can be found in [32-35]. The indicators presented in the Results section are described in Textbox 1 for each domain. The details on their computation can be found in the Multimedia Appendix 1. The ones with (S) after the name were selected a priori for the analysis of reliability, relationships with clinical scores, and MCI dysgraphia characterization. If all the indicators within a domain were selected a priori, (S) is found at the end of the name of the domain.

Statistical Analysis

Overview

The same software was used to run the statistical analyses, with the significance level set at 5%. Given the differences in the indicators in cursive and block letters (BL) handwriting—the choice of the allograph was left free to the participant—the analyses were conducted separately for the 2 allographs. The pipeline was common for each goal: after testing for the indicator normality with the Lilliefors test, the proper statistical test was applied.

Reliability

The indicators test-retest reliability in the MCI group for the List, Text, and PnP tasks, as well as for PnP regular, irregular, and MU words separately, was the first aspect to be evaluated. Before computing the reliability, a paired sample test was conducted, namely the t test for normally distributed indicators, the Wilcoxon test otherwise. If the paired test rejected the null hypothesis (ie, the mean/median of the difference between test and retest value is equal to 0), the indicator was deemed not reliable. Otherwise, the ICC (absolute agreement) or the Kendall W was computed if the indicator was normal or nonnormal, respectively. Both parameters range between 0 and 1. The indicator reliability was interpreted as follows: nonreliable if ICC<0.5 or W<0.2; fairly reliable if 0.2≤W<0.4; moderately reliable if 0.5≤ICC< 0.75 or 0.4≤W<0.6; good reliability if 0.75≤ICC<0.9 or 0.6≤W<0.8; optimal reliability if ICC≥0.9 or W≥0.8. Notably, the test-retest reliability analysis did not involve the indicators measuring variability (namely, all the indicators measuring variance or coefficient of variation [CV]).

The successive statistical analyses were solely based on the indicators extracted from the first repetition of the task, to avoid including eventual learning phenomenon in the data.

Relationship With Clinical Scores

For the second aim (ie, support to the clinical evaluation), the correlation between the indicators and the clinical parameters (MMSE, CDT, B-ADL, I-ADL, TMT-A, and number of errors in PnP test, either globally and divided by word type) was computed to measure to which extent the former can be used to approximate the results of cognitive/functional tests. Either the Pearson (r, normal) or Spearman (ρ, nonnormal) correlation coefficients were considered with the following interpretation: weak correlation, 0.2≤∣r, ρ∣<0.4; moderate correlation, 0.4≤∣r, ρ∣<0.6; strong correlation, 0.6≤∣r, ρ∣<0.8; very strong correlation, ∣r, ρ∣≥0.8. In this case, the PnP task was considered as a whole (ie, no differentiation among word types).

MCI Dysgraphia Characterization

To provide quantitative information for the detection of lexical and phonetical dysgraphia, handwriting indicators were statistically compared among the different types of words in the PnP task. After averaging the indicators across words of the same type, the differences among the 3 types were studied using statistical methods for paired samples: repeated measures analysis of variance for normal data and Friedman test for the nonnormal case. Following such tests, in case the null hypothesis was rejected, a post hoc comparison was conducted to determine which word groups exhibited significant differences using the Bonferroni method.

Classification of Unconstrained Handwriting Tasks

Last, binary classification models for the distinction between patients diagnosed with MCI and HC participants were developed in Python (version 3.10; Python Software Foundation). The models were based only on the unconstrained writing tasks presented in this study (List and Text) to collect evidence on the suitability of ecological handwriting for MCI detection. The data for HC participants were collected during other acquisition campaigns, using the same device. These participants signed an informed consent for their voluntary participation. It is worth noting that only participants with age ≥65 years and MMSE ≥27 were included in the HC group, to avoid considering participants with possibly relevant cognitive deterioration [36]. The age threshold of 65 years was based on the age of the youngest participant in the MCI group. Importantly, HCs were excluded if diagnosed with any neurological, musculoskeletal, or cardiovascular conditions that could impair handwriting.

The starting datasets were as follows: (1) List in cursive allograph, L (C); (2) Text in cursive allograph, T (C); (3) List and Text in cursive allograph, L+T (C); (4) List in BL allograph, L (BL); (5) Text in BL allograph, T (BL); (6) List in all allographs (L); and (7) Text in all allographs (T). Independent of the considered dataset, the model’s input consisted of handwriting indicators only, excluding the ones showing a correlation coefficient greater than 0.9 in absolute value with at least another indicator [37]. A 10-fold stratified repeated cross-validation approach was adopted in the training phase, setting the F₁-score as the metric to be optimized, given that, in general, the datasets were not balanced. At each iteration, the seed was changed and the whole dataset was divided into train, validation, and test sets, adopting 0.70, 0.15, and 0.15 ratios, respectively. The validation set was exploited to optimize the algorithm hyperparameters through a randomized search approach running 30 iterations. The test set was used to evaluate the model’s performance on data not seen by the algorithm in the previous phases. The global performances were then assessed by averaging the classification metrics (accuracy, recall, precision, F₁-score, and area under the precision recall curve) over the 10 folds. Within this framework, several strategies were adopted. Given the general imbalance of the target classes, both undersampling and oversampling techniques were implemented in the training and validation sets through the imbalanced-learn library [38]. The former included random undersampling, the NearMiss method, and the Estimated Nearest Neighbors method. As for the latter, Synthetic Minority Over-sampling Technique (SMOTE), SMOTE-support vector machine (SVMSMOTE), Borderline SMOTE, and ADASYN were considered. Last, the selectKBest feature selection method [39] was used to reduce the dataset dimensionality, with the number of features K to be tried ranging from 10 to 50 with discrete steps of 10. Support vector classifier, random forest, and a pool of boosting algorithms—namely AdaBoost, gradient boosting classifier (GBC), XGBoost (XGB), LightGBM (LGBM), and CatBoost—were tried. The details on the space of explored hyperparameters for each model are provided in the Multimedia Appendix 1. After the identification of the best performing models on the test set, the importance of indicators in their predictions was evaluated through Shapley Additive Explanations (SHAP) analysis [40]. The results of the SHAP analysis were used to qualitatively explore the patterns of misclassified samples: for each model, the trend of the 5 most relevant indicators for the predictions in misclassified samples was compared with the median trend of the same indicators in the MCI and HC groups through polar plots. To do so, the indicators were normalized between 0 and 1 for visualization purposes.

Results

Participants and Dataset

The characteristics of the recruited participants are reported in Table 1, separately for the whole sample of patients with MCI, which was exploited for the classification analysis, and for the sample that performed test and retest acquisitions, considered for the statistical analyses. Among the 57 patients with MCI enrolled, 40 were amnestic and 17 were nonamnestic. In more detail, the sample included: 3 pure amnestic MCIs (with isolated memory disturbance); 37 multiple-domain amnestic MCIs (with memory disturbance associated with disturbance of other cognitive faculties); 8 multiple-domain nonamnestic MCIs (with disturbance of other cognitive faculties, eg, attention, executive, spatial, and language, without memory disturbance); 9 single-domain nonamnestic MCIs (with disturbance of a single cognitive faculty, mostly dysexecutive).

As for the HC sample, information is displayed separately for the List and Text tasks. For this group, only the MMSE score was available. When comparing the MMSE score of participants with MCI and HC, the score of the MCI whole group was significantly lower than that of the HC sample who performed the List (P=5.4×10^–05) and the HC sample who performed the Text (P=6.4×10^–07).

The time required for completing the PnP test, given in seconds as median (IQR), was 126.42 (102.56-155.27) seconds at test and 122.56 (95.16-141.47) seconds at retest.

The allographs adopted by patients who underwent the test-retest protocol are reported hereafter:

• List, 29 cursive, 12 BL, 4 mixed (ie, BL in one repetition and cursive in the other).
• Text, 33 cursive, 11 BL, 1 mixed.
• PnP, 31 cursive, 11 BL, 3 mixed.

The allograph distribution of the data used in the classification problem is shown in Table 2. Here, the information refers to the available handwriting data, since both the test and retest acquisitions of the MCI group were considered, while HC participants performed the tasks multiple times.

Reliability

The analysis regarded the data collected from participants with MCI, denoted as “MCI Test-Retest” in Table 1. Participants who adopted different allographs across the test-retest protocol were excluded. For the sake of brevity, the results are reported for a subset of temporal, force, fluency, and tilt indicators, since these domains are the most commonly investigated in the literature.

Table 3 displays the results obtained on the 2 unconstrained handwriting tasks, separately for cursive (“C” columns) and block letters (“BL” columns) allograph.

In the List executed in cursive, optimal, good, and moderate reliability was revealed for 6 of 15, 8 of 15 (mean pause not reaching significance), and 1 of 15 indicators, respectively. The trend was confirmed in the Text task, with 7, 5, and 2 indicators showing optimal, good, and moderate reliability. Only a single unreliable indicator emerged, namely “Mean Force.” When dealing with BL, the results were comparable for the Text, with only 2 indicators being not reliable. Among the reliable ones, statistical significance was not reached in 4 of 13 cases (mean in air, mean pause, LDLJ A, mean tilt). The situation was instead slightly worse for the List task: reliability was mostly of moderate intensity (10/15 indicators), coupled with 8 cases with a P value greater than the critical threshold of .05.

The indicators’ reliability related to the PnP task performed in cursive (Table 4) was optimal, as the analysis yielded 14 out of 15 significant results of at least good reliability. Significance was not revealed only for the mean in air. The breakdown of the 3 types of words (Table 4) revealed that the positive results obtained on the whole task were due to the contribution of regular (Force NC being the only nonreliable indicator) and irregular words (besides cons peak diff G, all the indicators were significantly reliable). MU words were indeed characterized by 5 out of 15 unreliable indicators and 2 nonsignificant results among the reliable ones. Furthermore, in this case, the BL allograph was associated with worse outcomes in the whole PnP (Table 5: 5 unreliable indicators and 6 at least moderately reliable but not significant ones). Here, the poorest results happened with irregular words (6 reliable indicators reaching significance, 6 not reliable), while MU words revealed better reliability values than the corresponding in cursive, although only 4 showed a P value lower than .05.

Figure 1 displays a heatmap of the overall reliability intensity across indicators—the same reported in Tables 3-5—and tasks.

‎

Relationship With Clinical Scores

The statistically significant correlations of handwriting indicators with clinical parameters are reported in Table 6. Overall, significant correlations were identified mainly for the unconstrained tasks in BL allograph. The I-ADL score was also significantly correlated with the indicators extracted from the Text in cursive allograph, while the cursive List exhibited no significant results at all. The PnP test, on the other hand, revealed relationships with the computed indicators mainly when considering the number of errors in the irregular words. The correlation absolute values ranged from moderate to strong, and their direction followed the expected trends.

MCI Dysgraphia Characterization

The statistically significant results of the post hoc comparison of indicators (ie, the indicators for which the repeated measures ANOVA or Friedman test rejected the null hypothesis) among regular, irregular, and MU words are reported in Table 7 for both cursive (“C” columns) and block letters (“BL” columns). Most significances regarded irregular words.

Classification of Unconstrained Handwriting Tasks

The best classification performances for each dataset are displayed in Table 8, together with the adopted training strategy in terms of under- or oversampling and feature selection. Classification metrics on the test set are given as average (SD) over the 10 folds of stratified repeated cross-validation.

The top 5 most important indicators for each model, according to the SHAP analysis, are presented in Table 9.

A total of 21 indicators resulted among the possible 35 (5 top relevant indicators for 7 datasets), the most relevant ones. The most prevalent was Tilt CV, being present in 5 out of 7 datasets (one occurrence of Tilt CV_T in the L+T [C] dataset), followed by G peak power (4/7). The indicators A peak frequency and force NC were found in 3/7 cases, while cons peak diff G, A ApEn, and execution time resulted in 2/7 datasets. The remaining indicators occurred only once. Importantly, the indicators occurring multiple times were always characterized by the same direction in the MCI group. A high degree of consistency was revealed for the 3 models trained on cursive samples. Indeed, Tilt CV was the most important indicator in all 3 cases, while cons peak diff G, A ApEn, and G peak power were found in the 2 models trained with List cursive samples (L [C]; L+T [C]). For the 2 models trained on BL samples, the opposite trend occurred, with no superpositions among their 5 most relevant indicators.

Given these outcomes, the analysis of misclassified samples was carried out for cursive-samples models only.

In the L (C) model, G peak power showed trends compatible with those of patients with MCI in 50% (5/10) of false positive (FP) samples. The other recurrent altered indicators in FPs were cons peak diff G and air sheet ratio (4/10, 40%). Tilt CV and G peak power were found to be in line with that of HC participants in 83% (5/6) and 67% (4/6) of the false negatives (FNs), respectively.

In T (C), FPs were mainly associated with problems in temporal indicators (execution time: 11/16, 69%; mean pause: 10/16, 63%) and in a peak frequency (11/16, 69%). This last indicator was the main contributor to the presence of FNs (12/16, 75%), together with Tilt CV and mean pause (10/16, 63%).

When combining the information of the 2 tasks (L+T [C]), Tilt CV_T was found to deviate from the expected values in most misclassified samples: 67% in both FPs (4/6) and FNs (2/3). The other main contributor to FPs was A NC (3/6, 50%).

Interestingly, 3 HCs and 2 MCIs were misclassified across all 3 cursive datasets. As for the HC group:

• PEN024 was consistently misclassified because of trends in A ApEn and temporal indicators (execution time, mean pause, and air sheet ratio) compatible with those of patients with MCI.
• PEN025 instead showed alterations in the kinematics of the handwriting movement, with high values of Tilt CV in the T (C) task and great angular velocity oscillations power in both tasks (G peak power).
• PENT002 exhibited a heterogeneous pattern, with difficulties in the force and pen inclination domains in the T (C) task. Altered values of cons peak diff G were among the reasons for the wrong prediction both in L (C) and L+T (C).

Moving to the patients with MCI:

• DGSMN07 revealed a high coherence across tasks. The modulation of rotational kinematics (cons peak diff G and G peak power) was similar to that of HCs, in turn causing the same behavior in Tilt CV.
• DGMSN17 was misclassified in all datasets due to the trend of Tilt CV. The effect of temporal indicators was identified in the T (C) task.

The indicators’ trend for these samples is presented in the following radar plots for each classification model (in order, L [C], T [C], and L+T [C]). For each model, the most relevant indicator is displayed at the top of the radar. Then, indicators are displayed counterclockwise in decreasing order of relevance. All the values are normalized between 0 and 1 for visualization purposes. The dashed green line represents the indicators’ median value, while the green area represents the interval between the 25th and the 75th percentile in the HC group. The red elements represent the same for the group of patients with MCI. The solid lines represent the indicator’s value for the sample under examination. Blue hues are used for HC samples in Figure 2, while orange hues are used for MCI samples in Figure 3.

‎

‎

Discussion

Principal Findings

This work presented the quantitative analysis of handwriting in a population of patients diagnosed with MCI. With respect to the existing body of literature on the topic, an ecological approach for data acquisition was proposed. Handwriting tasks were executed on traditional paper using a sensorized ink pen able to record kinematic and dynamic signals. Moreover, content-free handwriting composition tasks were proposed to the participants to mirror the writing that can be carried out in everyday life. These aspects were deemed necessary for the development of a solution that potentially unlocks the monitoring of handwriting in the home environment. However, such features alone are not sufficient for the purpose.

That is why the first goal of the work was the assessment of the test-retest reliability of the information extracted from the acquisition protocol. High reliability is an essential requirement for any measurement instrument, particularly for solutions meant to detect anomalies in unsupervised longitudinal frameworks, in order to avoid high FP rates and alert fatigue. Importantly, the reliability analysis was conducted by separating different writing styles, namely cursive and BL, given that they differ in both temporal and kinematic parameters. The results were excellent for the cursive allograph, considering the 3 tasks, 42 out of the 45 presented indicators resulted significantly reliable, with reliability values mostly rated as good or optimal. This percentage dropped for BL: this was caused mainly by P values not reaching statistical significance, rather than by low reliability values. The outcome could be explained by the reduced number of participants who adopted the BL allograph in the recruited sample, which could also be the reason behind the overall better results found for cursive. Interestingly, the best results were achieved with the unconstrained tasks (List and Text), regardless of the adopted allograph. This could imply that one’s peculiar handwriting process characteristics are preserved independently of the content itself, thus making the analysis of unsupervised handwriting activity a promising approach for the remote tracking of MCI progression. On the other hand, the reliability of the indicators from the standardized PnP test in BL was not always verified. The unreliability of the mean pause indicator, measuring the average duration of moments longer than 2 seconds spent with the pen in the air, could, in theory, be due to differences in the dictation timing of the operator between test and retest acquisitions.

Regarding the support for the clinical evaluation, the correlation analysis revealed a scarce presence of significant results (around 10% of the total correlation values). Such an outcome should not be interpreted negatively for 2 reasons. First, the administered clinical tests are not meant to assess the handwriting performance specifically. Even the CDT and the TMT-A, which require the patients to draw and trace lines for their completion, are not evaluated according to the graphomotor performance itself, but rather on their accuracy. The same holds for the PnP test: the scoring is based on the orthographic correctness of the written words, while the proposed indicators are aimed at characterizing the handwriting process. Thus, a direct correspondence between the clinical scores and the handwriting indicators does not exist. Moreover, the CDR and B-ADL scores were characterized by low granularity (the former presented with only 2 distinct values, namely 0.5 and 1), resulting in poor significance. Second, and most importantly, the obtained significant correlations were of at least moderate intensity and in line with the existing literature [17]. Diving into detail, the highest number of correlations was found between temporal handwriting indicators and the MMSE and the I-ADL scores. As the MMSE and I-ADL increased, rel stroke num and on on-sheet ratio increased as well, while the mean in air, the mean pause, pause num, and air sheet ratio decreased. Therefore, in the recruited sample, participants who are more cognitively intact and functional in daily life tend to write faster thanks to a reduced degree of hesitation. Another relevant finding concerns the 2 tests involving the use of a pen or pencil. Those who performed well in the CDT and TMT-A applied a greater force on the writing surface, coupled with a higher number of changes in the force signal direction (the opposite signs of the correlation values are due to the opposite interpretation of the tests score). Such a pattern is associated in the literature with the healthy status [19]. Last, significant correlations between indicators and the number of errors in the PnP test emerged only for the errors in Irregular words. Here, when the BL allograph was used, a higher number of errors was associated with a faster performance on the sheet: the mean time spent with the pen on paper (mean on sheet) and the percentage of time spent on paper when writing a word (on sheet ratio) both showed a negative correlation. This could suggest that errors in irregular words occurred because the participants paid reduced attention (hence the increased speed on the sheet) to the words to be written, despite requiring orthographic knowledge to be written correctly. Linking the observed correlation patterns with specific domains of cognitive function is not trivial. The clinical tests used in the correlational study are useful in providing general measures of one’s cognitive functioning or some of its high-level variables, such as processing speed, or measures of the patient’s level of autonomy. In contrast, they are not thought of, nor have they shown particular accuracy and validity in detecting dysfunctions of individual cognitive functions such as memory, attention, executive functions, critical thinking, language, spatial abilities, visual perception, and so on. Any attempt to explain the obtained correlations in terms of individual cognitive domains would therefore be highly speculative and unsupportable with the available data. However, to a first approximation, the correlations between temporal indicators and MMSE and I-ADL scores seem to closely resemble what clinicians call cognitive hesitancy, an aspect of general insecurity in test responses that can be framed in the context of executive function dysfunctionality.

Dealing with the quantitative information derived from the PnP test, another neat trend emerged for irregular words. Independent of the handwriting style and orthographic errors, irregular words posed the greatest challenge to the recruited sample. With respect to the other types of words, irregular ones were consistently and significantly associated with worse in-air temporal performances (ie, longer times spent with the pen in the air), as revealed by mean in air and mean pause. This translated into a significantly slower rate of grapheme production (Rel stroke num) and, consequently, into the longest times required to write the words (execution time). In the BL allograph, alterations were also found in the on-sheet execution, as irregular words exhibited around 10% less time spent with the pen on the writing surface (on-sheet ratio). Last, irregular words were linked to a reduced variability in the pen inclination (Tilt CV), possibly suggesting an impaired manipulation of the writing instrument when dealing with this type of word. According to the classical cognitive models of handwriting, the marked alteration of the handwriting process found in the current work for irregular words, which require semantic and lexical knowledge to be written correctly compared with regular and MU words, could underline alterations in the lexical pathways as a result of dysfunctions in the orthographic retrieval process.

Overall, the implemented quantitative indicators seem able to capture relevant aspects of the handwriting gesture that are linked to the participant’s general conditions. On top of that, the indicators can also identify altered patterns that can provide a deeper understanding of the nature of the patient’s handwriting impairments. This stresses once again the suitability of handwriting monitoring in the population with MCI, as impairments in the gesture could be associated with a cognitive or functional decline.

The last evaluated aspect was the ability of the indicators to distinguish unconstrained handwriting samples generated by patients with MCI from those written by older adults without any diagnoses and MMSE ≥ 27. The choice of focusing on the List and Text tasks was motivated by their potential application in the domestic scenario. In such a condition, the information could be gathered in ecological modality without the need for the operator’s supervision. The classification performances were good, with F₁-scores selected as the evaluation criterion given the datasets’ imbalance, ranging from 0.81 to 0.92. Importantly, the results do not appear to be task or allograph-dependent. These results highlight the power of the proposed analysis: the extracted handwriting process indicators do provide relevant information on the participants’ cognitive status, regardless of the writing content and the allograph adopted to realize it. When comparing with the existing literature, the current results are placed between the accuracies of 74% in [19] and 96.6% in [20]. It is, however, worth pointing out that the latter classification performance, related to the execution of a spontaneous sentence, was obtained in different conditions. A digitizer was used, thus providing information also on the written trace, not available in this study, on a narrow sample size (12 patients with MCI and 17 controls). Moreover, the authors reported the participants’ age, which was not balanced between groups, as one of the most discriminative features for the model. Such a variable was not included in the models presented here.

The exploration of the relevance of indicators in the predictions of models provided great insight into the aspects that mostly differentiate the execution of patients with MCI and HC participants. This was true particularly for the cursive allograph, since heterogeneous results were found in the models based on BL samples, likely because of the reduced sample size. In general, the outcome of the SHAP analysis was not the expected one. Despite anticipating temporal indicators to carry the highest discriminative power, reflecting hesitations and greater processing times for patients with MCI, their occurrence among the most relevant was somewhat limited: only 5 out of 35 temporal indicators emerged, never placed in first or second position. The biggest determinant in the classification models was the variability of the pen inclination with respect to the vertical axis (Tilt CV), as it was found among the 5 most relevant indicators in 6 out of 7 models, being in the first place in 4 cases. In the classification models built on the cursive List samples (L [C] and L+T [C]), the relevance of Tilt CV, with high values always steering the classification towards the MCI group, was coherently associated with the presence of other kinematic indicators, both in the fluency and in the frequency domains. Such an outcome could open new perspectives on the evaluation of the handwriting performance of patients with MCI, traditionally focused on execution time, product quality, and orthographic accuracy. Researchers and clinicians should also consider the handwriting process kinematics that underlie the generation of the written trace. The results of the SHAP analysis were the basis for the inspection of misclassification patterns in the cursive-based models. These were homogeneous in FNs, for whom Tilt CV was the main reason why the models predicted them as belonging to the HC group. When this was not the case, the cause was still found in the handwriting kinematics. The situation was variable for FPs. In L (C), three archetypes emerged: (1) the participants who experience alterations in the variability of pen inclination; (2) the participants with excessive amplitude in rotational movements; (3) a heterogeneous group, with difficulties both in kinematics and in-air durations. The same number of patterns showed up in T (C). The first was again characterized by abnormal values of Tilt CV, while the second exhibited the opposite behavior, with no impairments in Tilt CV. The last archetype was the most anticipated one, where increased times were the sole reason for misclassifications. When combining List and Text in cursive, 2 main patterns were highlighted. Once again, the most prevalent had Tilt CV_T as the cause of FPs occurrence, while the other one was the entropy of the acceleration signal. Overall, the analysis revealed the multifaceted nature of the handwriting process in older adults, highlighting the importance of the participant-specific exploration of the model predictions. This would allow one to unveil one’s peculiar pattern of handwriting impairments, thus helping the clinician in identifying the motor and cognitive aspects that need a thorough assessment.

It is important to highlight the limitations found in this work. Regarding the recruitment, depression and anxiety disorders, relevant for various aspects of cognitive motor processes, were not formally measured during patients’ assessment. However, patients with a history of anxiety disorders or depression were excluded from the sample. So, caution should be placed when trying to generalize the presented findings to the population with MCI. As for the inclusion of control participants, their selection was solely based on their MMSE score. While they did not report any neurological diagnoses, they did not undergo any thorough clinical examination aimed at assessing their cognitive status. Thus, the presence of cognitively impaired participants in the control group cannot be excluded. In the statistical analysis of the available data, the percentage of recruited patients who adopted the BL allograph (around 1/3 of the total) was limited with respect to cursive. This translated into a reduced number of statistical significances in the reliability analysis, while the binary classification models trained on BL samples were characterized by greater SDs of the classification metrics across the 10 folds of the stratified repeated cross-validation, with respect to their cursive counterparts. Despite the imbalance in the adopted allograph, we do believe that the performance difference between BL-trained and cursive-trained classification models does not imply a reduced discriminative power of the former writing style. Although additional BL samples are needed to prove this hypothesis, this allograph should be better suited for the binary classification problem based on the proposed tool and indicators. Indeed, being the allograph characterized by a highly standardized motor pattern of stroke generation, the main source of variability between MCI and HC participants should be found in the indicators measuring the in-air performance, which should reflect longer processing times and hesitations in the former group. When dealing with cursive, the participants’ personal handwriting style comes into play, making the classification task more challenging. Just as an example, greater times on-sheet could indicate either slowness of movement or a writing style characterized by longer, interconnected strokes. In general, the choice of the allograph was left free to the participant to guarantee an ecological execution of the task and to align with the PnP test instructions, which do not specify how the participant should write, but the presented findings should be confirmed by increasing the BL sample size. The free choice turned out to be a liability in a limited number of cases, where patients adopted different allographs in the test and retest acquisitions, thus making them unsuitable for the reliability analysis. Given the impossibility of characterizing specific cognitive domains with the battery of clinical tests used in this work, the inclusion of more specific clinical tests for different cognitive domains is planned as a next step to refine the correlation analysis. In this regard, the positive results obtained with the words and nonwords writing test, which is the only instrument investigating a specific cognitive function in the study, indicate that such a research objective seems promising and feasible. With respect to the digitizers, the used sensorized ink pen cannot directly provide information on the trace made on paper. Future work should focus on proper methods for the extraction of product-related handwriting information to reduce this gap, particularly for the cursive allograph. Other promising directions can be found in multimodal approaches for the classification of patients with MCI. For example, the authors in [21] demonstrated that the fusion of handwriting and EEG parameters can improve the classification performance with respect to considering a single data source alone. Of course, the feasibility and user friendliness of the protocol should be carefully designed in such a case. The results also found the basis for additional research: a longitudinal study could be conducted, possibly in an unsupervised scenario. Last, the clinical community would benefit from comparative investigations on the discriminative efficacy of various handwriting-based screening methods. They should also consider their implementation complexity to provide an all-around informed documentation regarding the utility and feasibility of their integration into clinical practice.

Conclusions

The relevance of handwriting in the population with MCI was confirmed in this work, which described a novel approach for its quantitative characterization. The analyses revealed 2 potential complementary scenarios for its fruitful application. On one side, information related to the participant’s clinical conditions can be quickly and noninvasively extracted, both through the PnP test and unconstrained handwriting. On the other hand, the handwriting tool emerges as a viable solution to track old adults’ conditions in a transparent and ecological way, fostering MCI remote monitoring in a home setting.