This retrospective study utilized perioperative data collected from January 2018 to December 2019, as part of a federal study (R01 AG055337, CP) approved by the institutional review board of the University of Florida, with a waiver for Health Insurance Portability and Accountability Act compliance and an honest broker for medical record retrieval. The research adhered to the Declaration of Helsinki and followed all the institutional protocols. Participants provided written informed consent at the time of data collection.

The study extracted data on 14,000 patients from a deidentified historical database. A specific subset, the TAVR cohort, was identified using the Current Procedural Terminology code 33361, which narrowed the focus to 131 patients. All these patients had aortic stenosis as one of their symptoms.

A person is classified as frail if they meet 3 or more of the following conditions: (1) unintentional weight loss of 10 pounds or more over the past 6 months; (2) a sense of persistent exhaustion, characterized by a moderate feeling that normal activities were strenuous over the past week or a moderate inability to initiate activities; (3) a reduced pace in walking, as assessed by a nurse; (4) diminished grip strength, determined by a gender-specific T-score below −2.5, based on the highest grip strength recorded across 3 attempts using a Jamar hydraulic hand dynamometer (Model J00105, Lafayette Instrument Europe); and (5) a self-reported decrease in physical activity, gauged by the Duke Activity Status Index [24]. The scoring range is from 0 to 5; individuals meeting fewer than 3 of these criteria are considered nonfrail. This frailty assessment is documented in electronic health records (EHRs) to assist in planning perioperative care. In the preoperative anesthesia clinic, both the nursing staff and anesthesia professionals undergo training to perform frailty assessments, guided by published criteria [21,25], alongside documenting educational background and conducting cognitive evaluations.

Latent Dirichlet allocation (LDA) is a statistical model used in the field of natural language processing to uncover the hidden thematic structure in extensive collections of documents. We used LDA to generate 150 topics on clinical notes collected 60 days before surgery. A total of 543,520 clinical notes from 18,513 patients were collected. We removed preoperative evaluation notes that documented the results of the Fried frailty phenotype score to avoid information contamination. During data preprocessing, we found that some clinical notes contained similar information about the patients. To overcome this issue, we calculated the similarity between clinical notes that share the same NOTE_KEY using cosine distance and removed the notes with a similarity score greater than 80%. Special characters were removed, and all uppercase letters were converted to lowercase. Three types of words from the clinical notes were removed: (1) high-frequency words that were not closely related to medical symptoms, (2) words with a frequency less than 30, and (3) common stop words. We further performed stemming to reduce the words to their root form to simplify and standardize the vocabulary. After the data preprocessing, a total of 170,731 notes from 18,012 patients were included. Topic modeling was based on 50,000 randomly selected clinical notes. Topics of the remaining clinical notes were predicted and aggregated to the individual level to represent if the patients had a mention of a certain topic.

Preoperative attributes extracted from structured EHR data encompassed patient details available up to 6 months prior to their frailty evaluation. These included sociodemographic data (like age, gender, race, and years of education), medical history and severity (such as cancer history and the American Society of Anesthesiologists [ASA] physical status classification [26]), along with the latest biochemical markers typically recorded in preoperative settings (eg, hemoglobin and hematocrit levels).

Sociodemographic attributes captured information such as the patient’s age at frailty evaluation, gender, race, marital and employment status, and educational attainment. We divided age into 3 categories: under 70 years, between 70 and 79 years, and over 80 years. Education level was quantified by the total years of formal education [27], with each year of progression counted and repetitions not contributing to the total. For instance, completion of high school was marked as 12 years, a bachelor’s degree as 16 years, and so on. Medical history and acuity features included various medical conditions, the ASA physical status score, and the type of clinical encounter.

Medical conditions were binary indicators based on whether a patient had certain conditions, with the International Classification of Diseases and Related Health Problems, Tenth Revision, diagnosis codes condensed into Clinical Classifications Software categories for a more clinically relevant summarization. Conditions appearing in at least 10% of a broader cohort were selected as features for the ML models. The ASA score, ranging from 1 to 5, assessed patient health, with scores 1-2 indicating low-risk or mild systemic disease, and scores 3 and above signaling serious systemic illness.

Biochemical features included hematocrit, hemoglobin, and platelet count, with hemoglobin and hematocrit levels recommended for evaluation in older patients (65 years and above) undergoing significant surgery or those with a history of severe anemia undergoing minor surgery.

We manually reviewed topics generated from topic modeling and removed those that were not related to medical findings, resulting in 121 topic features in total. TAVR patients without clinical notes were removed (n=34), resulting in 97/131 patients in the training and evaluating ML models. Table S1 in Multimedia Appendix 1 compares the demographics between patients who were removed due to the absence of clinical notes and those who were used for modeling. After combining features generated from topic modeling and those from structured EHR data, we performed least absolute shrinkage and selection operator (LASSO) regression to reduce the feature space. Different parameters for regularization strength were examined, and the one that achieved the highest area under the receiver operating characteristic curve (AUC) for predicting frailty was selected for feature selection. We further applied Extreme Gradient Boosting (XGBoost) on features selected from LASSO to predict frailty. XGBoost is an ensemble learning method that uses decision tree frameworks. It creates models to improve the efficacy of preceding trees by diminishing errors through gradient descent. The resulting frailty index from XGBoost is a continuous score between 0 and 1, representing the probability of a patient being predicted as frail; high scores indicate a greater likelihood of frailty in TAVR patients.

To assess the additive value of topic features derived from clinical notes, we also evaluated the performance of XGBoost on predicting frailty using only features from structured EHR data. We used the k-nearest neighbor imputation technique to impute missing values in numerical features with the mean value from the k-nearest neighbors (k=5) found in the training dataset. Missing values in categorical features were imputed as a new label (unknown). Numerical features were scaled to 0-1, and categorical features were encoded before training the ML model. Figure 1 illustrates the overview of the analytic pipeline.

We employed a 5×5 nested cross-validation technique to evaluate our ML models. This technique consisted of 5 outer and 5 inner folds. During each outer fold, we set aside one-fifth of the patient records as an independent testing set, while the remaining four-fifths formed the training set. This outer training subset was then split into 5 inner folds for further validation. Each inner fold served as a stand-alone validation set, while the other 4 acted as the training set for the inner loop. The inner loop was responsible for training the models and fine-tuning the hyperparameters, taking a methodical approach in searching for the model’s ideal hyperparameter settings. Meanwhile, the outer loop estimated errors and evaluated generalization capabilities. To optimize hyperparameters, we utilized a grid search strategy to systematically explore various combinations of predefined hyperparameters to train our models. We calculated and reported the mean and SD for metrics like AUC, accuracy, sensitivity, and specificity across the 5 outer folds. This thorough approach boosts confidence in the models’ generalizability and scalability.

We used the Shapley Additive Explanations (SHAP) method to explain how our trained ML models work. SHAP is a widely used model-agnostic explanatory approach that helps in understanding the outputs of ML models. We created a SHAP summary plot to demonstrate the importance of features and their impact on the outcome. This impact is shown through a sign and magnitude: the SHAP value’s sign indicates the direction of the feature’s impact on the outcome (eg, a positive SHAP value indicates that the feature in question increases the likelihood of frailty), while its magnitude reflects the feature’s predictive influence.

Categorical features between the frail and nonfrail cohorts were compared using a χ² test for features with expected cell counts greater than 5 and a Fisher exact test for features with expected cell count less than or equal to 5 (ie, race). Age differences between the frail and nonfrail cohorts were compared using a Wilcoxson rank sum test. The significance level was set at P<.05.

This study developed a frailty index using ML techniques by combining structured and unstructured real-world data. Incorporating the features obtained from unstructured clinical notes and structured EHR data significantly improved the model’s performance in identifying frailty. SHAP revealed that cardiovascular conditions, specifically CHF and AFib, as well as musculoskeletal disorders, were the primary factors that predict frailty among patients. This is consistent with the mainstream of clinical evidence [28-30]. The prominence of age as a predictor supports the well-established association between advancing age and increased vulnerability to frailty and limited quality of life improvement post TAVR [31]. The impact of years of formal education on frailty prediction was relatively lower (ranked 10th in importance), indicating that while socioeconomic factors contribute to health outcomes, clinical characteristics are more determinative in the context of frailty. This clinical ML application can easily be integrated into pre-TAVR settings to guide perioperative management, heart team discussions, and resource utilization. Our model provides actionable and semiquantitative insights and can potentially highlight targets for optimization (eg, anemia). In addition, the data processing and derivation pipeline we practically demonstrate allows ML model retraining to accommodate updates in practice and interinstitutional context differences.

Structured real-world data that contain rich historical and current information across various domains (eg, diagnoses, procedures, demographics) have been leveraged to predict frailty. Segal et al [32] constructed ML models using medical claims data to predict the Fried frailty phenotype, achieving an AUC of 0.76 with a gradient boosting machines algorithm. Similarly, Le Pogam et al’s [33] exploration using inpatient discharge data obtained an AUC of 0.71, while Bai et al [34] harnessed structured EHR data to predict preoperative frailty, achieving an AUC of 0.74 using XGBoost. Notably, a considerable amount of information contained within EHRs exists as unstructured text. Recent advancements in ML and natural language processing have enabled researchers to harness this valuable resource, which traditionally posed substantial challenges for manual analysis. For instance, Shao et al [35] used topic modeling to extract frailty-related topics from clinical notes and defined frailty using the number of frailty-related topics present. This frailty index was associated with a 2-fold increased risk of the composite outcome combining 1-year hospitalization and mortality. Further, Chen et al [36] and Martin et al [37] developed deep learning models to predict sentences indicating geriatric syndromes or actionable frailty aspects and achieved an F₁-score of 0.61 and a scaled Brier score of 0.52. Despite these advances, a gap remains in the literature concerning the integration of structured and unstructured real-world data for frailty prediction. Moreover, the development of frailty indices tailored to preoperative contexts, especially for TAVR patients, remains unexplored.

Our results demonstrated that clinical notes serve as a complement to structured EHR data in predicting preoperative frailty. Incorporating features extracted from clinical notes significantly improved the model’s performance over using only structured EHR data, with the integrated model using only 12 features achieving an AUC of 0.82 compared to 0.64 for the EHR-only model in the test set. This performance is superior to the one-size-fits-all statistical models such as the Society of Thoracic Surgeons score, which has an AUC of 0.64-0.7 in risk stratifying patients [38-40].

Identifying the key topics and associated terms provides actionable insights into understanding and managing frailty-associated factors. For instance, knowing that the patient has heart failure requiring cardiac medications and contributing to frailty may trigger putting the patient on a more intensive pre- and post-TAVR optimization pathway, such as right heart catheterization, to optimize heart failure therapy preprocedures. Similar approaches can be applied to other cardiovascular conditions or musculoskeletal disorders that are partially correctable with focused pre- or postprocedural conditioning. The topic of CHF treatment and management, with high-weight terms like “Carvedilol,” can highlight high systemic vascular resistance hemodynamic phenotypes, systolic heart failure, or arrhythmia requiring beta blockade. Similarly, the topic focusing on AFib management, with terms like “Eliquis” and “Apixaban,” highlights the need to evaluate and manage arrhythmias, which are risk factors for stroke and also are associated with obesity, hypoventilation, and chronically low physical and cardiopulmonary conditioning. The specificity of terms and their weights in each topic provides a framework for health care professionals to identify and prioritize areas of concern in pre-TAVR patients, ensuring a comprehensive and tailored approach to frailty assessment. Identifying “high-risk candidates” is crucial in determining a patient’s readiness for TAVR and optimizing postprocedural outcomes by addressing the multifaceted aspects of frailty and medical complexity in this vulnerable patient population [41,42]. A considerable advantage to using ML models is that they are translatable to new TAVR techniques, adaptable to clinical settings and automated facilitating implementation, and updatable to accommodate shifts in populations and practices.

Compared to the traditional frailty measurements, a key advantage of our developed method is its applicability to newly admitted patients with severe aortic stenosis being considered for TAVR. The model can process a patient’s available information, including both structured EHR data and unstructured clinical notes, to generate a frailty index score once the patient is admitted. This assessment provides an objective measure of a patient’s frailty status, which has the potential to enhance the clinical shared decision-making between physicians and the patient. By offering an early and timely indication of a patient’s frailty status, this approach allows for a more informed discussion about the risks and benefits of TAVR, potential alternative treatments, and the need for additional preoperative interventions and postoperative care plans, ultimately improving the care of TAVR patients.

This study, while insightful, acknowledges specific limitations. It utilizes retrospective EHR data, a method that provides a valuable foundation for analysis despite its inherent limitations, such as the potential for minor reporting inconsistencies, recall variations, and lead time considerations. The study’s focus on frailty, although a topic of ongoing discussion due to its varied definitions and intersection with medical complexity, enriches the discourse by offering a nuanced view of its implications. Additionally, while our ML approach effectively integrates structured and unstructured real-world data to predict frailty, it does not extensively address potential confounding factors such as comorbidities, medications, or socioeconomic status. This focus on predictive modeling, rather than causal analysis, may limit the ability to fully account for these variables. Furthermore, this study has not undergone external validation; however, this aspect is mitigated by the rigorous internal review processes, which reduce the risk of overfitting and ensure the model’s preliminary reliability. Rather than significantly limiting, these considerations underscore the study’s contribution to ongoing research and its potential as a springboard for further investigation and external validation efforts.

This study demonstrates the effectiveness of combining unstructured clinical notes with structured EHR data to assess frailty in TAVR patients. This approach can potentially improve risk stratification for TAVR, adding insight into targets for perioperative optimization and potentially improving TAVR outcomes. Future efforts should integrate these models into clinical settings to improve TAVR planning and patient care, emphasizing continuous medical education and innovation.