University of Chicago Medicine (UCM) is an urban academic medical center in Chicago, IL, serving a predominantly Black patient population. Study participants were recruited from UCM’s primary care clinic from June 2018 to December 2019 to participate in the MDG pilot, which was a delayed comparator study that creates personalized goals using patient responses and a predictive model. All participants were ≥65 years of age, had a confirmed diagnosis of type 2 diabetes mellitus in the medical record, had attended at least 1 outpatient primary care clinic visit within the year prior to recruitment, and had an active online portal account through UCM. Throughout the MDG pilot, patients were surveyed via the portal about their health status. All participants who completed the initial survey, regardless of the MDG randomization arm to which they were allocated, were included in this study.

The study protocol was approved by the institutional review board at UCM (number 18-0425), and all participants provided written informed consent.

The mortality index deployed during this pilot study (hereafter referred to as the Lee Index) was developed and validated using data from the Health and Retirement Study [17]. The index has been widely used in other geriatric studies [18-22]. The index incorporates demographic characteristics (age, sex, and BMI), information about specific self-reported disease diagnoses (presence of cancer or malignant tumor, excluding skin cancer; presence of heart failure; an activity-limiting lung condition or home oxygen use; and recent cigarette smoker), and self-reported performance on a series of functional limitations (difficulty bathing or showering, difficulty managing money, difficulty ambulating several blocks, and difficulty pushing or pulling large objects). Points assigned to various answers to each question range from 1 to 7 out of a total of 26. The Lee Index calculation was performed twice for each patient: first using the information provided by the patient in their online portal survey and then using the data obtained retrospectively from the chart review described below. The scores were categorized into the following American Diabetes Association (ADA) classes for older adults with diabetes: participants with scores 0 to 7 were classified into class 1 with an HbA_1c goal <7.5%, scores 8 to 11 into class 2 with an HbA_1c goal <8.0%, and scores >12 into class 3 with an HbA_1c goal <8.5% [23,24]. These score cutoffs correspond with life expectancies of >10 years, 5 to 10 years, and <5 years.

As part of the enrollment and intervention for the larger MDG study, a comprehensive survey was sent to all study participants via an online patient portal in the EHR with questions about their individual health, diabetes management, treatment preferences, and adverse events. A subset of survey answers corresponded to the mortality index questions that were used to calculate the survey score.

A research assistant performed a retrospective chart review in UCM’s EHR for all study participants over 6 months prior to and through enrollment. Relevant data were systematically obtained. The medical history and problem list sections were reviewed for relevant diagnoses. Demographic data were obtained from registration forms. Body measures were found in encounter flowsheets. Functional conditions, limitations, and any assessments or clarification of diagnoses were found by manually searching for keywords in a note search feature. All data obtained were recorded and coded into an electronic database. Differences between all survey responses and chart review were compared and frequencies were summed to determine which questions had the greatest diferences between the survey and the chart review.

Life expectancy predictions were calculated from the Gompertz-Predicted Median Life Expectancy data based on the Lee Index score calculated both from the survey and the chart review [11].

All quantitative outcomes were summarized with descriptive statistics. The McNemar test was used to evaluate the differences in patients’ individual data inputs and ADA classes between chart abstraction and the survey. The Wilcoxon signed-rank test was used to evaluate the differences in patients’ mortality index scores and life expectancy between chart abstraction and the survey. Linear regression was also used to model the correlation between the mortality index scores from chart abstraction and the survey. RStudio (version 4.1.0; Posit PBC) was used for data analysis.

There were 75 participants who completed the survey through the larger MDG study. The majority of participants were female (n=50, 66.7%), identified as Black (n=49, 65.3%), and had a mean age of 72.5 (SD 5.3) years. Additionally, most participants (n=59, 78.7%) had an HbA_1c level ≤8%, while approximately a quarter (n=17, 22.7%) had an HbA_1c level <6.5% prior to enrollment (Table 1).

Individual responses to questions were compared between the 2 methods and frequencies were summed to show notable differences across all questions except for demographics (Figure 1). Across questions, the largest differences in available data were related to physical function. A total of 31 (41.3%) individuals noted difficulty completing a task (ie, “bathing or showering,” “walking a few blocks,” and “pushing or pulling large objects”) in the survey that was not captured in the chart review. A total of 23 (30.7%) participants had differences in their ability to push and pull; of these, 22 (95.7%) reported difficulties not found in the chart and only 1 had a reported difficulty in the chart that was not found in the survey. The last functional question, regarding difficulty with finances, differed for 5 participants where more disability was identified in the chart review compared to the survey.

Additionally, 10 participants had a “cancer or a malignant tumor” identified by the chart review that was not reported in the survey, and 8 participants had “a lung condition that limits [their] usual activities” identified by their chart review that was not reported in the survey.

The average mortality index score for all patients based on information from the self-reported MDG survey was 7.23 (SD 3.04, IQR 4), while the score calculated from the chart abstraction was 7.07 (SD 2.66, IQR 4). The difference between the Lee Index score assignment comparing survey and chart review was not statistically significant (P=.82). Comparing individual participants’ scores, there was a significant positive correlation between the scores (R=0.77) (Figure 2).

Using the mortality scores, the mean life expectancy prediction for all patients based on information from the MDG survey was 13.37 (SD 6.03, IQR 8.9) years compared to 13.40 (SD 5.53, IQR 8.9) years from the chart abstraction (P=.82).

Of the 75 participants, 15 (20%) were placed in different ADA classes based on comparison of the survey and the chart review (P=.54; Table 2). The class distribution of the population was similar and not significantly different between the 2 class calculations (P=.74; Figure 3). From the survey, 60% (n=45) were in class 1, 31% (n=23) were in class 2, and 9% (n=7) were in class 3. The chart review percentages were 65% (n=49), 28% (n=21), and 7% (n=5), respectively.

In this small exploratory study, we found differences in the reported functional status and disease history of patients when comparing data from chart reviews and surveys. Despite these differences in data inputs, the overall model life expectancy prediction was not significantly different across the chart review and the patient survey. It is not clear from this exploratory study whether the differences in data availability from the chart and the patient survey would actually lead to clinically significant differences in glucose control recommendations for older patients at scale. An alternative interpretation is that available data in the EHR performed reasonably well despite data limitations.

With the exception of demographics, other domains had differences between the chart review and the survey, the largest difference being in the functional status domains. The functional question related to having trouble pushing or pulling showed the largest difference between the 2 abstraction methods. The same question had the least predictive power in the calculations so it did not affect the overall score and subsequent class calculations significantly in this model. In models for other clinical conditions such as frailty, questions regarding functional status could have greater impact on the accuracy of predictions [25].

An additional area of difference was related to diagnoses. We found that many participants underreported temporally distant and low-stage cancers or early-stage chronic medical conditions such as heart failure or lung disease compared to what was documented in the chart.

A notable limitation in this study is the study size, which likely leaves the study underpowered to identify significant differences. Another is that our original study required patients to be able to use our patient portal application, MyChart. Thus, patients were excluded from recruitment if they did not have an active MyChart account. It is possible that patients who have an active patient portal are more engaged in care and have a more complete EHR than patients without an active portal.

We continued to see differences in several health domains as has been noted previously [15], with the largest differences observed in questions related to functional assessments. Since functional status is clinically important for an array of decisions, our study suggests the need to systematically capture functional status data in the EHR. As a health industry, we already collect responses to many social and behavior questions, like smoking status, to use in various models such as the ASCVD (Atherosclerotic Cardiovascular Disease) Risk Calculator [26], and effort should be made to collect more of these PROs. Our study shows that prediction models, such as the Lee Index, are incomplete without them. The overall impact of this incompleteness would be much more important for populations with significant functional impairments.

The future of health care will likely continue toward an increasing use of prediction models. As health care systems incorporate these models, there will be questions about whether or not to incorporate models that include PROs as inputs. Our study suggests that some of these models may actually already perform reasonably well with available EHR data despite some degree of missingness. Whether or not the degree of missing data leads to clinically important differences will require further study of the value of systematic PRO data collection. Tools such as MyChart open the door for routine collection of such patient-reported data at a reasonable cost to health systems. As systems become more experienced with systematically collecting PROs, these data become part of the EHR. Beyond the utility for predicting future outcomes, eliciting PROs may also have a benefit for patient engagement and health behavior change.