This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Aging, is properly cited. The complete bibliographic information, a link to the original publication on https://aging.jmir.org, as well as this copyright and license information must be included.
Background
More than 6 million people in the United States have Alzheimer disease and related dementias, receiving help from more than 11 million family or other informal caregivers. A range of traditional interventions has been developed to support family caregivers; however, most of them have not been implemented in practice and remain largely inaccessible. While recent studies have shown that family caregivers of people with dementia use Twitter to discuss their experiences, methods have not been developed to enable the use of Twitter for interventions.
Objective
The objective of this study is to develop an annotated data set and benchmark classification models for automatically identifying a cohort of Twitter users who have a family member with dementia.
Methods
Between May 4 and May 20, 2021, we collected 10,733 tweets, posted by 8846 users, that mention a dementia-related keyword, a linguistic marker that potentially indicates a diagnosis, and a select familial relationship. Three annotators annotated 1 random tweet per user to distinguish those that indicate having a family member with dementia from those that do not. Interannotator agreement was 0.82 (Fleiss kappa). We used the annotated tweets to train and evaluate support vector machine and deep neural network classifiers. To assess the scalability of our approach, we then deployed automatic classification on unlabeled tweets that were continuously collected between May 4, 2021, and March 9, 2022.
Results
A deep neural network classifier based on a BERT (bidirectional encoder representations from transformers) model pretrained on tweets achieved the highest F1-score of 0.962 (precision=0.946 and recall=0.979) for the class of tweets indicating that the user has a family member with dementia. The classifier detected 128,838 tweets that indicate having a family member with dementia, posted by 74,290 users between May 4, 2021, and March 9, 2022—that is, approximately 7500 users per month.
Conclusions
Our annotated data set can be used to automatically identify Twitter users who have a family member with dementia, enabling the use of Twitter on a large scale to not only explore family caregivers’ experiences but also directly target interventions at these users.
natural language processingsocial mediadata miningdementiaAlzheimer diseasecaregiversIntroduction
More than 6 million people in the United States have Alzheimer disease and related dementias, and the burden is projected to double by 2060 [1]. Alzheimer disease is the sixth leading cause of death in the United States [2], and only 8% of people with dementia do not receive help from family members or other informal care providers [3], amounting to more than 11 million family or other unpaid caregivers in 2020 [4]. Caregivers of people with dementia are impacted physically, cognitively, socially, mentally, and financially. For instance, compared with noncaregivers, they are more vulnerable to disease due to chronic stress [5] and have lower durations and quality of sleep [6]. Compared with non–dementia caregivers, they are more likely to experience a decline in cognition [7] and social network size [8]. They are also more likely to experience depression compared with noncaregivers [9] and non–dementia caregivers [10], and depressive symptoms in dementia caregivers are associated with increased health care use and costs [11]. In addition to the increased costs of their personal health care, family caregivers of people with dementia pay for much of the recipient’s total care costs, with the costs being significantly higher for people with dementia than without dementia [12].
A range of traditional interventions has been developed to support family caregivers of people with dementia [13]; however, most of them have not been implemented in practice and remain largely inaccessible [14]. Recent systematic reviews have concluded that internet-based interventions are valued by family caregivers of people with dementia for their easy access [15] and can have beneficial effects on caregivers’ health [16]. While recent studies [17-23] have shown that family caregivers of people with dementia use Twitter to discuss their experiences, to the best of our knowledge, methods have not been developed to enable the use of Twitter as a platform for internet-based interventions. Given that nearly 1 of every 4 adults in the United States uses Twitter [24], Twitter may present a novel opportunity to reach family caregivers on a large scale, such as through user-targeted advertisements providing information about dementia, caregiving, resources, or services. The objective of this study was to develop an annotated data set and benchmark classification models for automatically identifying a cohort of Twitter users who have a family member with dementia.
MethodsEthical Considerations
The data used in this study were collected in accordance with the Twitter Terms of Service. The Institutional Review Board of the University of Pennsylvania reviewed this study (protocol number: 828972) and deemed it exempt human subjects research under 45 CFR §46.101(b)(4) for publicly available data sources.
Data Collection and Annotation
Between May 4 and May 20, 2021, we collected 67,060 publicly available tweets from the Twitter streaming application programming interface (API) that are in English, are not retweets, and include both a dementia-related keyword (eg, dementia, youngdementia, #yod, #ftd, alzheimer’s, alz, alzheimersdisease, mild cognitive impairment) and a linguistic marker that potentially indicates a diagnosis (eg, diagnosed, diagnosis, has, got, developed, with, from). The full list of API search terms is available in Multimedia Appendix 1. We then searched these tweets for references to select familial relationships (Multimedia Appendix 2), identifying 10,733 (16%) of the 67,060 tweets. We randomly sampled 1 tweet per user—8846 (82%) of the 10,733 tweets—and developed annotation guidelines (Multimedia Appendix 3) to help 3 annotators distinguish tweets that indicate having a family member with dementia from those that do not. Among the 8846 annotated tweets, 8346 (94%) were dual annotated, and 500 (6%) were annotated by all 3 annotators. Interannotator agreement, based on the 500 tweets annotated by all 3 annotators, was 0.82 (Fleiss kappa). Upon resolving the disagreements, it was determined that 5946 (67%) of the tweets indicate that the user has a family member with dementia, and 2900 (33%) of the tweets do not.
Automatic Classification
We performed benchmark supervised machine learning experiments to assess the utility of the annotated data set for automatically identifying Twitter users who have a family member with dementia. For the classifiers, we used the LibSVM [25] implementation of support vector machine (SVM) in Weka and SVM and 6 deep neural network classifiers based on BERT (bidirectional encoder representations from transformers): the BERT-Base-Uncased [26], DistilBERT-Base-Uncased [27], RoBERTa-Large [28], BioBERT-Large-Cased [29], Bio+ClinicalBERT [30], and BERTweet-Large [31] pretrained models in the Flair Python library. We split the 8846 tweets into 80% (7077 tweets) and 20% (1769 tweets) random sets as training data (Multimedia Appendix 4) and held-out test data, respectively, stratified based on the distribution of the binary annotated classes. For the SVM classifier, we preprocessed the tweets by normalizing URLs, usernames, digits, and keywords related to dementia (Multimedia Appendix 1) and familial relationships (Multimedia Appendix 2), removing nonalphanumeric characters and extra spaces, and lowercasing and stemming [32] the text. We used the Weka NGram Tokenizer to extract n-grams (n=1-3) as features in a bag-of-words representation. We used the radial basis function kernel and set the cost at c=32. For the BERT-based classifiers, we preprocessed the tweets by normalizing URLs and usernames and lowercasing the text. For training, we used stochastic gradient descent optimization, a batch size of 8, 15 epochs, and a learning rate of 0.001. During training, we fine-tuned all layers of the transformer model with our annotated tweets. To optimize performance, the model was evaluated after each epoch on a 5% split of the training set. To assess the scalability of our approach, we then deployed automatic classification on 198,674 unlabeled tweets, posted by 119,640 users, that were continuously collected from the Twitter streaming API (Multimedia Appendix 1) between May 4, 2021, and March 9, 2022, and mentioned a select familial relationship (Multimedia Appendix 2).
Results
Table 1 presents the precision, recall, and F1-scores of SVM and 6 deep neural network classifiers for the class of tweets indicating that the user has a family member with dementia, evaluated on a held-out test set of 1769 (20%) of the 8846 manually annotated tweets. The classifier based on a model pretrained on tweets (BERTweet-Large) achieved the highest F1-score: 0.962 (precision=0.946 and recall=0.979). When deployed on 198,674 unlabeled tweets, posted by 119,640 users, between May 4, 2021, and March 9, 2022, the BERTweet classifier detected 128,838 tweets indicating that the user has a family member with dementia, posted by 74,290 users—that is, approximately 7500 users per month.
Table 2 presents examples of false positives and false negatives of the BERTweet classifier in the test set. Among the 68 false positives, 36 (47%) refer to people with dementia who are not or may not be select family members (Tweet 1), 8 (12%) report that a family member has a condition other than dementia (Tweet 2), and 5 (7%) merely speculate that a family member has dementia (Tweet 3). Another 8 (12%) of the 68 false positives were a result of manual annotation errors. Among the 25 false negatives, 14 (56%) use deixis or anaphora, requiring additional context in the tweet to understand that a non–first person determiner (eg, “their” in Tweet 4) actually refers to the user, or that a personal pronoun (eg, “she” in Tweet 5) refers to a select family member with dementia. Furthermore, 12 (86%) of these 14 tweets also include references to people who are not family members or do not have dementia. Another 4 (16%) of the 25 false negatives were a result of manual annotation errors.
Precision, recall, and F1-scores of classifiers for detecting tweets indicating that the user has a family member with dementia.
Classifier
Precision
Recall
F1-score
SVMa
0.884
0.939
0.910
BERTb-Base-Uncased
0.924
0.954
0.938
DistilBERT-Base-Uncased
0.930
0.942
0.936
RoBERTa-Large
0.918
0.982
0.949
BioBERT-Large-Cased
0.907
0.978
0.941
Bio+ClinicalBERT
0.903
0.958
0.930
BERTweet-Large
0.946
0.979
0.962
aSVM: support vector machine.
bBERT: bidirectional encoder representations from transformers.
Sample false positives and false negatives of a BERTweet classifier for detecting tweets indicating that the user has a select family member with dementia.
Tweet number
Tweet
Actual
Predicted
1
Evelyn has dementia, I know. But when she asked me today how my dad was doing... it still hurt.
–
+
2
We really don't have a clue about what causes Alzheimer's. We don't have a clue about Parkinson's, which is what got my dad, either.
–
+
3
I just listened to the Everywhere at The End of Time, by The Caretaker, and thought about my grandmother. The songs are about dementia, something my grandma wasn't clearly diagnosed with, but it hit hard.
–
+
4
If someone tells u their parent has Alzheimer's please don’t say your grandparent or great aunt did too. I appreciate that u can relate to the experience but it is so different. Tell me a different time.
+
–
5
I have a family member who is vulnerable and two children in their late 20s. I didn’t want to risk passing virus to her or from her to my family member. My sister made a bubble with her and her carers. She has dementia so she probably hasn’t missed me!
+
–
DiscussionPrincipal Findings
The benchmark performance of automatic classification demonstrates that our annotated data set has utility for accurately identifying Twitter users who have a family member with dementia, and deploying automatic classification on unlabeled tweets demonstrates that a large cohort of users can be identified. Therefore, our annotated data set enables the use of Twitter to scale up accessible, internet-based interventions directly targeted at family caregivers of people with dementia. Because our approach involves identifying tweets that mention a familial relationship, it would also enable interventions to be tailored to the care recipient.
Limitations
Our approach to identifying family caregivers assumes that having “close” relatives with dementia would likely imply the users’ involvement in caregiving; however, the users identified in this study may not necessarily be caregivers or may have been caregivers but are no longer. We took this approach because we believe that limiting our identification of caregivers to users who explicitly state that they are providing ongoing care would underutilize the potential of Twitter for reaching caregivers on a large scale.
Conclusions
This paper presented an annotated data set and benchmark classification models for automatically identifying Twitter users who have a family member with dementia, enabling the use of Twitter on a large scale to not only explore family caregivers’ experiences among their tweets but also directly target interventions at these users.
bidirectional encoder representations from transformers
SVM
support vector machine
This work was supported by the National Library of Medicine (R01LM011176). The authors thank Ivan Flores for contributing to software applications, and Alexis Upshur and Aiden McRobbie-Johnson for contributing to annotating the Twitter data.
AZK designed the data collection, edited the annotation guidelines, performed the support vector machine classification experiments, conducted the error analysis, and wrote the manuscript. AM performed the deep learning classification experiments, deployed the BERTweet classifier, and edited the manuscript. KO developed the annotation guidelines, annotated the Twitter data, and edited the manuscript. GGH conceptualized and guided the study and edited the manuscript.
None declared.
MatthewsKAXuWGagliotiAHHoltJBCroftJBMackDMcGuireLCRacial and ethnic estimates of Alzheimer's disease and related dementias in the United States (2015-2060) in adults aged ≥65 years20190119151172410.1016/j.jalz.2018.06.306330243772S1552-5260(18)33252-7PMC6333531KochanekKXuJAriasEMortality in the United States, 20192020123951833395387KasperJDFreedmanVASpillmanBCWolffJLThe disproportionate impact of dementia on family and unpaid caregiving to older adults20151034101642910.1377/hlthaff.2015.05362643873934/10/1642PMC4635557Alzheimer's Association2021 Alzheimer's disease facts and figures20210317332740610.1002/alz.1232833756057FonarevaIOkenBSPhysiological and functional consequences of caregiving for relatives with dementia2014052657254710.1017/S104161021400003924507463S1041610214000039PMC3975665GaoCChapagainNYScullinMKSleep duration and sleep quality in caregivers of patients with dementia: a systematic review and meta-analysis2019080228e19989110.1001/jamanetworkopen.2019.9891314419382748661PMC6714015DasselKCarrDVitalianoPDoes caring for a spouse with dementia accelerate cognitive decline? Findings from the health and retirement study2017040157231932810.1093/geront/gnv14826582383gnv148LiuCFabiusCDHowardVJHaleyWERothDLChange in social engagement among incident caregivers and controls: findings from the caregiving transitions study20210123331-211412410.1177/089826432096194632962491MaMDorstynDWardLPrenticeSAlzheimers' disease and caregiving: a meta-analytic review comparing the mental health of primary carers to controls2018110522111395140510.1080/13607863.2017.137068928871796SheehanOHaleyWHowardVHuangJRhodesJRothDStress, burden, and well-being in dementia and nondementia caregivers: insights from the caregiving transitions study2021071361567067910.1093/geront/gnaa108328160145894888PMC8276607ZhuCWScarmeasNOrnsteinKAlbertMBrandtJBlackerDSanoMSternYHealth-care use and cost in dementia caregivers: Longitudinal results from the Predictors Caregiver Study201504171144445410.1016/j.jalz.2013.12.01824637299S1552-5260(14)00007-7PMC4164583KelleyASMcGarryKBollens-LundERahmanOHusainMFerreiraKBSkinnerJSResidential setting and the cumulative financial burden of dementia in the 7 years before death202006186861319132410.1111/jgs.1641432187655PMC7957824GauglerJEPotterTPruinelliLPartnering with caregivers20140830349351510.1016/j.cger.2014.04.00325037292S0749-0690(14)00038-XGitlinLMarxKStanleyIHodgsonNTranslating evidence based dementia caregiving interventions into practice: state-of-the-science and next steps2015045522102610.1093/geront/gnu12326035597gnu123PMC4542834HopwoodJWalkerNMcDonaghLRaitGWaltersKIliffeSRossJDaviesNInternet-based interventions aimed at supporting family caregivers of people with dementia: systematic review20180612206e21610.2196/jmir.954829895512v20i6e216PMC6019848LengMZhaoYXiaoHLiCWangZInternet-based supportive interventions for family caregivers of people with dementia: systematic review and meta-analysis20200909229e1946810.2196/1946832902388v22i9e19468PMC7511858YoonSWhat can we learn about mental health needs from tweets mentioning dementia on World Alzheimer's Day?2016110122649850310.1177/10783903166636902780326222/6/498PMC5337405DanilovichMTsayJAl-BahraniRChoudharyAAgrawalA#Alzheimer’s and dementia: expressions of memory loss on Twitter2018341485310.1097/TGR.0000000000000173ChengTYLiuLWooBKAnalyzing Twitter as a platform for Alzheimer-related dementia awareness: thematic analyses of Tweets2018121012e1154210.2196/1154231518232v1i2e11542PMC6715397YoonSLuceroRMittelmanMSLuchsingerJABakkenSMining Twitter to inform the design of online interventions for Hispanic Alzheimer's disease and related dementias caregivers2020092418313814310.1177/154041531988277731646904MehtaNZhuLLamKStallNMSavageRReadSHWuWPopPFaulknerCBronskillSERochonPAHealth forums and Twitter for dementia research: opportunities and considerations2020120768122881288910.1111/jgs.1679032894780BacsuJO'ConnellMECammerAAziziMGrewalKPooleLGreenSSivananthanSSpiteriRJUsing Twitter to understand the COVID-19 experiences of people with dementia: infodemiology study20210203232e2625410.2196/2625433468449v23i2e26254PMC7861035YoonSBroadwellPAlcantaraCDavisNLeeHBristolATipianiDNhoJMittelmanMAnalyzing topics and sentiments from Twitter to gain insights to refine interventions for family caregivers of persons with Alzheimer's disease and related dementias (ADRD) during COVID-19 pandemic2022011428917017310.3233/SHTI21088635062119SHTI210886PMC8830611AuxierBAndersonMSocial media use in 2021202104072022-02-25https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/ChangCCLinCJLIBSVM: a library for support vector machines2011042312710.1145/1961189.1961199DevlinJChengMLeeKToutanovaKBERT: pre-training of deep bidirectional transformers for language understanding2019Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)June 2-7, 2019Minneapolis, Minnesota, US4171418610.18653/v1/N19-1423SanhVDebutLChaumondJWolfTDistilBERT, a distilled version of BERTmaller, faster, cheaper and lighter2019Fifth Workshop on Energy Efficient Machine Learning and Cognitive ComputingDecember 13, 2019Vancouver, CanadaLiuYOttMGoyalNDuJJoshiMChenDLevyOLewisMZettlemoyerMStoyanovVRoBERTa: a robustly optimized BERT pretraining approach Preprint posted online July 26, 2019.
LeeJYoonWKimSKimDKimSSoCKangJBioBERT: a pre-trained biomedical language representation model for biomedical text mining202002153641234124010.1093/bioinformatics/btz682315018855566506PMC7703786AlsentzerEMurphyJBoagWWengWJindiDNaumannTMcDermottMPublicly available clinical BERT embeddings2019Proceedings of the 2nd Clinical Natural Language Processing WorkshopJune 7, 2019Minneapolis, Minnesota, USA727810.18653/v1/w19-1909NguyenDVuTNguyenABERTweet: a pre-trained language model for English tweets2020Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System DemonstrationsNovember 16-20, 2020Online91410.18653/v1/2020.emnlp-demos.2PorterMFAn algorithm for suffix stripping198014313013710.1108/eb046814