Principal Findings
In this study, we developed and successfully implemented a rule-based algorithm to identify the severity of ADRD from unstructured EHR data. We detailed the steps to be taken for extracting the relevant information from EHR data and highlighted the challenges associated with it due to heterogeneity in textual representation. We find a lack of access to specialty facilities may impede timely diagnosis and the possibility of treatment at early stages of ADRD progression. As the severity of dementia is critical for health care providers to prescribe appropriate treatment and link resources to patients and their caregivers, our 2-pronged approach to search for relevant information presents a parsimonious yet effective way to make the disease severity information readily available across disciplines and care settings.
Similar to previous research [21-24], we developed the algorithm using a rule-based approach. Starting with an initial list of keywords or phrases based on previous work and contextual clinical knowledge, the list is refined iteratively to identify the target information. This step is followed by sampling the reports from the data set and dividing them into training and testing sets for gold-standard comparison and evaluation. Previous research applied a rule-based approach to identify caregiver availability [21], a record of mild cognitive impairment or Alzheimer disease [22], documentation of cognitive tests [23], and social determinants of health for patients with ADRD [24]. Unlike previous work, where either the rule definition step specific to the medical system [21], included ICD (International Classification of Diseases) codes [22], or had additional biomarker and cognitive tests information [23]. In comparison, our method shows robustness by using common occurring keywords and points toward the need for defining a minimum number of umbrella rules that have the potential to be generalizable for the entire data set and have better performance. Our developed algorithm is independent of the health care systems and provides clinicians with the flexibility to either use it without any modification or adapt it to their needs. A recent systematic review paper has suggested that rule-based NLP algorithms had similar performance compared with those using more sophisticated methods when the information is scarce in the EHR [25]. In our case, information on the severity of ADRD was presented using a few words, in less than half of the data. To evaluate the performance of our algorithm, we divided the tests into two categories that are (1) identifying the “presence of information on ADRD severity” and (2) “severity of ADRD.” We found our algorithm to be highly accurate in extracting documented information on ADRD severity from the EHR. The performance on sensitivity and specificity also indicates that our algorithm was able to correctly extract stage information where present and reduce false positive results. Taken together, these results support the clinical use of our simplified and generalizable approach to identify the severity of ADRD. Furthermore, compared with previous work, our algorithm showed an improved average F1-score in identifying the ADRD severity of the condition of the patient [26]. The better performance of our algorithm compared with previous work could be attributed to the use of unstructured EHR data instead of structured EHR and defining clear umbrella rules by identifying recurring patterns in our data set for ADRD severity categorization.
The performance of our algorithm is slightly diminished in identifying the severity of dementia from the records. Upon inspection of the wrongly labeled cases, it can be attributed to (1) the test scores not reflecting the correct severity compared with the clinician’s evaluation, (2) lack of contextual understanding of the sentence, and (3) noise in EHR note (irrelevant information), also noted in previous work [21]. A majority of the abovementioned issues were mitigated by defining subrules [27]. However, any further inclusion would have come at the cost of reduced performance and the need for increased clinician oversight, which limits its generalizability.
Although the rule-based algorithm worked considerably well in most of the cases, it is limited by the patterns and rules defined by the developer. For cases where the token default token delimiter, in our case “space,” is changed or missing, the algorithm fails to extract information. One approach to solve this could be to have an alternative copy of the algorithm that includes other common delimiters (eg, “;” “,” “:”) to identify word tokens. In our evaluation, we only found a few outliers not following the default way of describing texts with “space” as the delimiter. The urge to include all the stray cases would lead to the hard coding of the algorithm and give rise to new challenges with considerable false positive results, making it difficult to comb through. With the development of large language models (LLMs), 1 potential solution might be to use LLM to shape the EHR data in the same format, such as converting all patient records to have equal spacing, removing random commas, etc, without changing textual content and then follow a rule-based approach as presented in this study. Our design philosophy has been to keep the algorithm general while including common patterns. One possible criticism of our study could be dichotomizing the severity of ADRD. The rationale for dichotomizing the severity of ADRD into mild versus moderate-to-severe dementia is related to clinical decision-making. Given that there are several types of ADRD, such as Alzheimer disease, vascular dementia, and Lewy body dementia, to name a few, the differential treatment and care plans are limited as a considerable number of patients have mixed dementia, and the gold standard for a definitive differential diagnosis is still based on autopsy studies. Therefore, for this study, we only focused on a dichotomized version of the severity of ADRD. Future studies should further investigate a more comprehensive classification of ADRD severity and possibly include other tests for diagnosis of ADRD outside MoCA and MMSE.
Overall, we demonstrate the ability of our rule-based algorithm to identify the severity of ADRD, where present, in the EHR and narrow it down to the location of occurrence in the EHR. This not only allows us to comb through valuable unstructured data with ease, but the sequential nature of the algorithm provides us with contextual data that has a high probability of containing information about the severity of dementia. The extracted data can be used in future work to train a machine learning (ML) model with rich and high-quality data. We expect to enhance our method to further identify and predict the progression of ADRD over time. As the performance of an ML model depends on the quality of the data set, following a segmented approach of using a rule-based algorithm for extracting relevant paragraphs from the EHR can be used first to enrich the data set and reduce noise (ie, nonrelevant information from the EHR) followed by model training on the data. Previous research has found that combining structured and unstructured data might be a viable approach to classify patients. With structured data containing useful demographic information and unstructured data containing contextual, patient, and clinical notes, the path forward could be to leverage the qualities of both kinds of data [28] for the use of EHR.
Previous studies that used traditional ML models, such as logistic regression and support vector machine, often include only the structured EHR data due to the limited requirement for data management [6,12,13]. In recent years, increasing numbers of studies have applied deep learning approaches to classify patients for a given condition [6,29,30]. One of the strengths of the deep learning approach is its ability to incorporate relationships between words and a large amount of data in the analysis, which fits the need for using both structured and unstructured data together. Despite its strength, a common criticism against the implementation of deep learning approaches in clinical settings is the lack of interpretability [19]. The use of an interpretable rule-based approach has enabled us to highlight potential biases and pitfalls to be considered when using black-box deep learning models. One potential solution is to use rule-based pattern matching to highlight the trigger words and related neighborhood of words for added context and classify or label the patient record using deep learning techniques [31,32]. With the improvements in LLMs and their enhanced contextual and semantic understanding of texts, our rule-based method can be coupled with a pretrained LLM in pre- or postprocessing of the extracted texts [33]. Care must be taken while using LLMs due to the generative nature of text predictions in avoiding alteration of textual information and being limited to standardizing textual information. The proposed approach needs to be thoroughly evaluated through data privacy and model uncertainty lens before adoption.
We also found that the percentage of patients with missing information on the severity of ADRD is very high. Given the added significance of such information in recent times due to newly approved treatment, it is critical in current clinical practice to improve documentation of the severity of dementia to promote high-quality care. In addition, despite the missing information, our relatively simple algorithm approach has been successful in making previously inaccessible and hard-to-find information readily available to clinicians for a large number of patients. These patients would have otherwise not had this information available to their care team without our very practical approach.
Our study has a few limitations due to the algorithm of our choice and design decisions. First, as mentioned earlier, the rules have to be manually defined and fine-tuned based on a training set. This process, although simple in complexity, can be challenging as EHR can be very different based on health care systems. As the rules are manually defined, it has room for human error. On the other hand, this approach gives the researchers the flexibility to adapt the algorithm structure easily to their health care systems and needs to be fine-tuned. Second, rule-based algorithm matching studies are limited by a lack of contextual understanding between text groups and fail to recognize connotations in sentences. For example, the presence of a negative test result may confuse the algorithm pattern identification process unless explicitly included in the algorithm definition. Third, our approach has been able to only include rules for patterns that are common throughout the data set, such as for the ADRD Cognitive Score function, we include every score defined in the format of AB/30, AB, and a special type of definition as mentioned before. Therefore, we might have missed out on some of the cases straying away from the common patterns. For our instance, attempts to include every unique case led to the results being very irrelevant as it started capturing a lot of unimportant information and the algorithm became very rigid. Fourth, the study determines dementia severity based on cognitive test scores and trigger words for ADRD and does not include medical prescriptions being used by the patient which can be a future direction to make our approach holistic.
Finally, even though we have attempted to keep our method generalizable, the data were extracted from 1 health care system, and the developed rules may not directly apply to data from other health care systems. We have described in detail the steps taken in designing the algorithm with the aim of serving as a baseline approach for research in identifying ADRD severity from unstructured EHR data. The algorithm does the job well for the criteria chosen, and the choice of criteria to include can be a decision of the study designer.