The data used in this study were collected in accordance with the Twitter Terms of Service. The Institutional Review Board of the University of Pennsylvania reviewed this study (protocol number: 828972) and deemed it exempt human subjects research under 45 CFR §46.101(b)(4) for publicly available data sources.

Between May 4 and May 20, 2021, we collected 67,060 publicly available tweets from the Twitter streaming application programming interface (API) that are in English, are not retweets, and include both a dementia-related keyword (eg, dementia, youngdementia, #yod, #ftd, alzheimer’s, alz, alzheimersdisease, mild cognitive impairment) and a linguistic marker that potentially indicates a diagnosis (eg, diagnosed, diagnosis, has, got, developed, with, from). The full list of API search terms is available in Multimedia Appendix 1. We then searched these tweets for references to select familial relationships (Multimedia Appendix 2), identifying 10,733 (16%) of the 67,060 tweets. We randomly sampled 1 tweet per user—8846 (82%) of the 10,733 tweets—and developed annotation guidelines (Multimedia Appendix 3) to help 3 annotators distinguish tweets that indicate having a family member with dementia from those that do not. Among the 8846 annotated tweets, 8346 (94%) were dual annotated, and 500 (6%) were annotated by all 3 annotators. Interannotator agreement, based on the 500 tweets annotated by all 3 annotators, was 0.82 (Fleiss kappa). Upon resolving the disagreements, it was determined that 5946 (67%) of the tweets indicate that the user has a family member with dementia, and 2900 (33%) of the tweets do not.

We performed benchmark supervised machine learning experiments to assess the utility of the annotated data set for automatically identifying Twitter users who have a family member with dementia. For the classifiers, we used the LibSVM [25] implementation of support vector machine (SVM) in Weka and SVM and 6 deep neural network classifiers based on BERT (bidirectional encoder representations from transformers): the BERT-Base-Uncased [26], DistilBERT-Base-Uncased [27], RoBERTa-Large [28], BioBERT-Large-Cased [29], Bio+ClinicalBERT [30], and BERTweet-Large [31] pretrained models in the Flair Python library. We split the 8846 tweets into 80% (7077 tweets) and 20% (1769 tweets) random sets as training data (Multimedia Appendix 4) and held-out test data, respectively, stratified based on the distribution of the binary annotated classes. For the SVM classifier, we preprocessed the tweets by normalizing URLs, usernames, digits, and keywords related to dementia (Multimedia Appendix 1) and familial relationships (Multimedia Appendix 2), removing nonalphanumeric characters and extra spaces, and lowercasing and stemming [32] the text. We used the Weka NGram Tokenizer to extract n-grams (n=1-3) as features in a bag-of-words representation. We used the radial basis function kernel and set the cost at c=32. For the BERT-based classifiers, we preprocessed the tweets by normalizing URLs and usernames and lowercasing the text. For training, we used stochastic gradient descent optimization, a batch size of 8, 15 epochs, and a learning rate of 0.001. During training, we fine-tuned all layers of the transformer model with our annotated tweets. To optimize performance, the model was evaluated after each epoch on a 5% split of the training set. To assess the scalability of our approach, we then deployed automatic classification on 198,674 unlabeled tweets, posted by 119,640 users, that were continuously collected from the Twitter streaming API (Multimedia Appendix 1) between May 4, 2021, and March 9, 2022, and mentioned a select familial relationship (Multimedia Appendix 2).

The benchmark performance of automatic classification demonstrates that our annotated data set has utility for accurately identifying Twitter users who have a family member with dementia, and deploying automatic classification on unlabeled tweets demonstrates that a large cohort of users can be identified. Therefore, our annotated data set enables the use of Twitter to scale up accessible, internet-based interventions directly targeted at family caregivers of people with dementia. Because our approach involves identifying tweets that mention a familial relationship, it would also enable interventions to be tailored to the care recipient.

Our approach to identifying family caregivers assumes that having “close” relatives with dementia would likely imply the users’ involvement in caregiving; however, the users identified in this study may not necessarily be caregivers or may have been caregivers but are no longer. We took this approach because we believe that limiting our identification of caregivers to users who explicitly state that they are providing ongoing care would underutilize the potential of Twitter for reaching caregivers on a large scale.

This paper presented an annotated data set and benchmark classification models for automatically identifying Twitter users who have a family member with dementia, enabling the use of Twitter on a large scale to not only explore family caregivers’ experiences among their tweets but also directly target interventions at these users.