Background: Prediction models are being increasingly used in clinical practice, with some requiring patient-reported outcomes (PROs). The optimal approach to collecting the needed inputs is unknown.
Objective: Our objective was to compare mortality prediction model inputs and scores based on electronic health record (EHR) abstraction versus patient survey.
Methods: Older patients aged ≥65 years with type 2 diabetes at an urban primary care practice in Chicago were recruited to participate in a care management trial. All participants completed a survey via an electronic portal that included items on the presence of comorbid conditions and functional status, which are needed to complete a mortality prediction model. We compared the individual data inputs and the overall model performance based on the data gathered from the survey compared to the chart review.
Results: For individual data inputs, we found the largest differences in questions regarding functional status such as pushing/pulling, where 41.4% (31/75) of participants reported difficulties that were not captured in the chart with smaller differences for comorbid conditions. For the overall mortality score, we saw nonsignificant differences (P=.82) when comparing survey and chart-abstracted data. When allocating participants to life expectancy subgroups (<5 years, 5-10 years, >10 years), differences in survey and chart review data resulted in 20% having different subgroup assignments and, therefore, discordant glucose control recommendations.
Conclusions: In this small exploratory study, we found that, despite differences in data inputs regarding functional status, the overall performance of a mortality prediction model was similar when using survey and chart-abstracted data. Larger studies comparing patient survey and chart data are needed to assess whether these findings are reproduceable and clinically important.
Prediction models are being increasingly used in many aspects of clinical practice to identify distinct patient subpopulations and to guide the selection of therapies for chronic disease management [, ]. For instance, among older adults with diabetes, life expectancy prediction has become integral for determining individualized glycated hemoglobin (HbA1c) goals [ ]. The benefits of intensive glucose control (eg, HbA1c level <7.0%) are not realized for 9 to 10 years, and the risks, such as hypoglycemia and falls, in patients with limited life expectancy generally outweigh potential benefits [ - ]. Based on this comparison of life expectancy and time to benefit from intensive glucose control, multiple diabetes care guidelines have recommended the individualization of HbA1c goals by health status [ ].
Many prediction models are designed to rely on readily available data from electronic health records (EHRs) and insurance claims [, - ]. However, major life expectancy prediction models for older patients require self-reporting of their ability to perform basic and instrumental activities of daily living [ , ]. Some of these variables may already exist in various structured or unstructured elements of the EHR, but prior studies have demonstrated that patient-reported outcomes (PROs) obtained via survey may differ from data obtained from the patient’s EHR [ - ].
Despite increasing calls to use prediction models, the optimal approach to collecting the inputs for these models as part of clinical practice is unclear. An important question is whether we can rely solely on existing EHR data to populate these models or if we should expend additional resources to systematically collect PRO data. EHRs can be readily available for data extraction and analysis. In addition, EHRs have been designed to include fields for PRO data during the course of routine care. However, systematically collected PROs may reduce rates of missing data, have the benefit of coming directly from the patient, and can be tailored to the needs of the survey or model. Collecting the data often requires more time and effort due to tool development, management of patient follow-up times and nonresponse rates, and addressing of patient difficulties in responding due to cognitive, physical, technological, or other reasons . Thus, it is important to determine the value of systematic data collection for PROs.
While conducting the My Diabetes GOAL (MDG) pilot trial, we systematically collected data for a prediction model via an electronic survey from older patients with diabetes. This created an opportunity to compare results from a life expectancy model using data from chart abstraction and data from patient surveys. The aims of this study were to (1) characterize the magnitude of variation in the availability of data from chart abstraction or survey for individual variables and (2) determine any difference in mortality risk score or life expectancy prediction for patients based on surrogate information from chart abstraction compared to survey.
University of Chicago Medicine (UCM) is an urban academic medical center in Chicago, IL, serving a predominantly Black patient population. Study participants were recruited from UCM’s primary care clinic from June 2018 to December 2019 to participate in the MDG pilot, which was a delayed comparator study that creates personalized goals using patient responses and a predictive model. All participants were ≥65 years of age, had a confirmed diagnosis of type 2 diabetes mellitus in the medical record, had attended at least 1 outpatient primary care clinic visit within the year prior to recruitment, and had an active online portal account through UCM. Throughout the MDG pilot, patients were surveyed via the portal about their health status. All participants who completed the initial survey, regardless of the MDG randomization arm to which they were allocated, were included in this study.
The study protocol was approved by the institutional review board at UCM (number 18-0425), and all participants provided written informed consent.
Mortality Index Score
The mortality index deployed during this pilot study (hereafter referred to as the Lee Index) was developed and validated using data from the Health and Retirement Study . The index has been widely used in other geriatric studies [ - ]. The index incorporates demographic characteristics (age, sex, and BMI), information about specific self-reported disease diagnoses (presence of cancer or malignant tumor, excluding skin cancer; presence of heart failure; an activity-limiting lung condition or home oxygen use; and recent cigarette smoker), and self-reported performance on a series of functional limitations (difficulty bathing or showering, difficulty managing money, difficulty ambulating several blocks, and difficulty pushing or pulling large objects). Points assigned to various answers to each question range from 1 to 7 out of a total of 26. The Lee Index calculation was performed twice for each patient: first using the information provided by the patient in their online portal survey and then using the data obtained retrospectively from the chart review described below. The scores were categorized into the following American Diabetes Association (ADA) classes for older adults with diabetes: participants with scores 0 to 7 were classified into class 1 with an HbA1c goal <7.5%, scores 8 to 11 into class 2 with an HbA1c goal <8.0%, and scores >12 into class 3 with an HbA1c goal <8.5% [ , ]. These score cutoffs correspond with life expectancies of >10 years, 5 to 10 years, and <5 years.
As part of the enrollment and intervention for the larger MDG study, a comprehensive survey was sent to all study participants via an online patient portal in the EHR with questions about their individual health, diabetes management, treatment preferences, and adverse events. A subset of survey answers corresponded to the mortality index questions that were used to calculate the survey score.
A research assistant performed a retrospective chart review in UCM’s EHR for all study participants over 6 months prior to and through enrollment. Relevant data were systematically obtained. The medical history and problem list sections were reviewed for relevant diagnoses. Demographic data were obtained from registration forms. Body measures were found in encounter flowsheets. Functional conditions, limitations, and any assessments or clarification of diagnoses were found by manually searching for keywords in a note search feature. All data obtained were recorded and coded into an electronic database. Differences between all survey responses and chart review were compared and frequencies were summed to determine which questions had the greatest diferences between the survey and the chart review.
Life Expectancy Prediction
Life expectancy predictions were calculated from the Gompertz-Predicted Median Life Expectancy data based on the Lee Index score calculated both from the survey and the chart review .
All quantitative outcomes were summarized with descriptive statistics. The McNemar test was used to evaluate the differences in patients’ individual data inputs and ADA classes between chart abstraction and the survey. The Wilcoxon signed-rank test was used to evaluate the differences in patients’ mortality index scores and life expectancy between chart abstraction and the survey. Linear regression was also used to model the correlation between the mortality index scores from chart abstraction and the survey. RStudio (version 4.1.0; Posit PBC) was used for data analysis.
There were 75 participants who completed the survey through the larger MDG study. The majority of participants were female (n=50, 66.7%), identified as Black (n=49, 65.3%), and had a mean age of 72.5 (SD 5.3) years. Additionally, most participants (n=59, 78.7%) had an HbA1c level ≤8%, while approximately a quarter (n=17, 22.7%) had an HbA1c level <6.5% prior to enrollment ().
|Mean (SD)||72.5 (5.3)|
|Median (IQR)||72 (6)|
|Sex, n (%)|
|Race, n (%)|
|Mean (SD)||7.1 (1.0)|
|Median (IQR)||6.9 (1.2)|
|Level, n (%)|
|Blood pressure (mm Hg)|
|Mean (SD)||131 (16.7)|
|Median (IQR)||130 (21)|
|Mean (SD)||68 (10.2)|
|Median (IQR)||67 (12)|
|Mean (SD)||32 (7.1)|
|Median (IQR)||31 (8.7)|
|<25 kg/m2, n (%)||11 (14.7)|
|Use of diabetic medications, n (%)|
|Other characteristics, n (%)|
|Statin use||63 (84)|
aParticipants identified as “Asian Indian” (n=3) and “More than one race; Hispanic” (n=1).
bHbA1c: glycated hemoglobin.
cNo recent HbA1c (n=8).
dGLP-1: glucagon-like peptide 1.
eSGLT2: sodium-glucose cotransporter-2.
Individual Survey Data Inputs
Individual responses to questions were compared between the 2 methods and frequencies were summed to show notable differences across all questions except for demographics (). Across questions, the largest differences in available data were related to physical function. A total of 31 (41.3%) individuals noted difficulty completing a task (ie, “bathing or showering,” “walking a few blocks,” and “pushing or pulling large objects”) in the survey that was not captured in the chart review. A total of 23 (30.7%) participants had differences in their ability to push and pull; of these, 22 (95.7%) reported difficulties not found in the chart and only 1 had a reported difficulty in the chart that was not found in the survey. The last functional question, regarding difficulty with finances, differed for 5 participants where more disability was identified in the chart review compared to the survey.
Additionally, 10 participants had a “cancer or a malignant tumor” identified by the chart review that was not reported in the survey, and 8 participants had “a lung condition that limits [their] usual activities” identified by their chart review that was not reported in the survey.
Mortality Index Score
The average mortality index score for all patients based on information from the self-reported MDG survey was 7.23 (SD 3.04, IQR 4), while the score calculated from the chart abstraction was 7.07 (SD 2.66, IQR 4). The difference between the Lee Index score assignment comparing survey and chart review was not statistically significant (P=.82). Comparing individual participants’ scores, there was a significant positive correlation between the scores (R=0.77) ().
Life Expectancy Prediction
Using the mortality scores, the mean life expectancy prediction for all patients based on information from the MDG survey was 13.37 (SD 6.03, IQR 8.9) years compared to 13.40 (SD 5.53, IQR 8.9) years from the chart abstraction (P=.82).
Of the 75 participants, 15 (20%) were placed in different ADA classes based on comparison of the survey and the chart review (P=.54;). The class distribution of the population was similar and not significantly different between the 2 class calculations (P=.74; ). From the survey, 60% (n=45) were in class 1, 31% (n=23) were in class 2, and 9% (n=7) were in class 3. The chart review percentages were 65% (n=49), 28% (n=21), and 7% (n=5), respectively.
|Class assignment||Survey class 1, n||Survey class 2, n||Survey class 3, n|
|Chart class 1||41||7||1|
|Chart class 2||4||15||2|
|Chart class 3||0||1||4|
In this small exploratory study, we found differences in the reported functional status and disease history of patients when comparing data from chart reviews and surveys. Despite these differences in data inputs, the overall model life expectancy prediction was not significantly different across the chart review and the patient survey. It is not clear from this exploratory study whether the differences in data availability from the chart and the patient survey would actually lead to clinically significant differences in glucose control recommendations for older patients at scale. An alternative interpretation is that available data in the EHR performed reasonably well despite data limitations.
With the exception of demographics, other domains had differences between the chart review and the survey, the largest difference being in the functional status domains. The functional question related to having trouble pushing or pulling showed the largest difference between the 2 abstraction methods. The same question had the least predictive power in the calculations so it did not affect the overall score and subsequent class calculations significantly in this model. In models for other clinical conditions such as frailty, questions regarding functional status could have greater impact on the accuracy of predictions .
An additional area of difference was related to diagnoses. We found that many participants underreported temporally distant and low-stage cancers or early-stage chronic medical conditions such as heart failure or lung disease compared to what was documented in the chart.
A notable limitation in this study is the study size, which likely leaves the study underpowered to identify significant differences. Another is that our original study required patients to be able to use our patient portal application, MyChart. Thus, patients were excluded from recruitment if they did not have an active MyChart account. It is possible that patients who have an active patient portal are more engaged in care and have a more complete EHR than patients without an active portal.
We continued to see differences in several health domains as has been noted previously , with the largest differences observed in questions related to functional assessments. Since functional status is clinically important for an array of decisions, our study suggests the need to systematically capture functional status data in the EHR. As a health industry, we already collect responses to many social and behavior questions, like smoking status, to use in various models such as the ASCVD (Atherosclerotic Cardiovascular Disease) Risk Calculator [ ], and effort should be made to collect more of these PROs. Our study shows that prediction models, such as the Lee Index, are incomplete without them. The overall impact of this incompleteness would be much more important for populations with significant functional impairments.
The future of health care will likely continue toward an increasing use of prediction models. As health care systems incorporate these models, there will be questions about whether or not to incorporate models that include PROs as inputs. Our study suggests that some of these models may actually already perform reasonably well with available EHR data despite some degree of missingness. Whether or not the degree of missing data leads to clinically important differences will require further study of the value of systematic PRO data collection. Tools such as MyChart open the door for routine collection of such patient-reported data at a reasonable cost to health systems. As systems become more experienced with systematically collecting PROs, these data become part of the EHR. Beyond the utility for predicting future outcomes, eliciting PROs may also have a benefit for patient engagement and health behavior change.
The study was partially funded by grants K24AG069080 (National Institute on Aging), P30 DK092949 (National Institute of Diabetes and Digestive and Kidney Diseases), and R01DK127961 (National Institute of Diabetes and Digestive and Kidney Diseases), as well as by the University of Chicago Medicine Innovation Award.
Conflicts of Interest
- Dinh A, Miertschin S, Young A, Mohanty SD. A data-driven approach to predicting diabetes and cardiovascular disease with machine learning. BMC Med Inform Decis Mak 2019 Nov 6;19(1):211 [CrossRef] [Medline]
- Gill TM. The central role of prognosis in clinical decision making. JAMA 2012 Jan 11;307(2):199-200 [CrossRef] [Medline]
- Lee AK, Steinman MA, Lee SJ. Improving the American Diabetes Association Framework for individualizing treatment in older adults: evaluating life expectancy. BMJ Open Diabetes Res Care 2020 Sep;8(1):e001624 [CrossRef] [Medline]
- American Diabetes Association. Older adults: standards of medical care in diabetes—2020. Diabetes Care 2020 Jan;43(Suppl 1):S152-S162 [CrossRef] [Medline]
- Lipska KJ, Krumholz H, Soones T, Lee SJ. Polypharmacy in the aging patient: a review of glycemic control in older adults with type 2 diabetes. JAMA 2016 Mar 8;315(10):1034-1045 [CrossRef] [Medline]
- Huang ES. Management of diabetes mellitus in older people with comorbidities. BMJ 2016 Jun 15;353:i2200 [CrossRef] [Medline]
- McCoy RG, Nori VS, Smith SA, Hane CA. Development and validation of HealthImpact: an incident diabetes prediction model based on administrative data. Health Serv Res 2016 Oct;51(5):1896-1918 [CrossRef] [Medline]
- Dunkler D, Gao P, Lee SF, Heinze G, Clase CM, Tobe S, et al. Risk prediction for early CKD in type 2 diabetes. Clin J Am Soc Nephrol 2015 Aug 7;10(8):1371-1379 [CrossRef] [Medline]
- Basu S, Narayanaswamy R. A prediction model for uncontrolled type 2 diabetes mellitus incorporating area-level social determinants of health. Med Care 2019 Aug;57(8):592-600 [CrossRef] [Medline]
- Chudasama YV, Khunti KK, Zaccardi F, Rowlands AV, Yates T, Gillies CL, et al. Physical activity, multimorbidity, and life expectancy: a UK Biobank longitudinal study. BMC Med 2019 Jun 12;17(1):108 [CrossRef] [Medline]
- Lee SJ, Boscardin WJ, Kirby KA, Covinsky KE, Berthold HK. Individualizing life expectancy estimates for older adults using the Gompertz law of human mortality. PLoS ONE 2014 Sep 29;9(9):e108540 [CrossRef] [Medline]
- An Q, Chronister K, Song R, Pearson M, Pan Y, Yang B, et al. Comparison of self-reported HIV testing data with medical records data in Houston, TX 2012-2013. Ann Epidemiol 2016 Mar 23;26(4):255-260 [CrossRef] [Medline]
- Hoggatt KJ, Hepner KA. Assessing brief intervention for unhealthy alcohol use: a comparison of electronic health record documentation and patient self-report. J Stud Alcohol Drugs 2018 Sep;79(5):697-701 [CrossRef] [Medline]
- Wallace E, Moriarty F, McGarrigle C, Smith SM, Kenny RA, Fahey T. Self-report versus electronic medical record recorded healthcare utilisation in older community-dwelling adults: comparison of two prospective cohort studies. PLoS ONE 2018 Oct 26;13(10):e0206201 [CrossRef] [Medline]
- Tisnado DM, Adams JL, Liu H, Damberg CL, Chen WP, Hu FA, et al. What is the concordance between the medical record and patient self-report as data sources for ambulatory care? Med Care 2006 Feb;44(2):132-140 [CrossRef] [Medline]
- Basch E, Barbera L, Kerrigan CL, Velikova G. Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 2018 May 23;38:122-134 [CrossRef] [Medline]
- Lee SJ, Lindquist K, Segal MR, Covinsky KE. Development and validation of a prognostic index for 4-year mortality in older adults. JAMA 2006 Feb 15;295(7):801-808 [CrossRef] [Medline]
- Pilla SJ, Schoenborn NL, Maruthur NM, Huang ES. Approaches to risk assessment among older patients with diabetes. Curr Diab Rep 2019 Jul 19;19(8):59 [CrossRef] [Medline]
- Huang ES, Nathan AG, Cooper JM, Lee SM, Shin N, John PM, et al. Impact and feasibility of personalized decision support for older patients with diabetes: a pilot randomized trial. Med Decis Making 2017 Jul;37(5):611-617 [CrossRef] [Medline]
- Kobayashi LC, Jackson SE, Lee SJ, Wardle J, Steptoe A. The development and validation of an index to predict 10-year mortality risk in a longitudinal cohort of older English adults. Age Ageing 2017 May 1;46(3):427-432 [CrossRef] [Medline]
- Gitlin LN, Hauck WW, Dennis MP, Winter L, Hodgson N, Schinfeld S. Long-term effect on mortality of a home intervention that reduces functional difficulties in older adults: results from a randomized trial. J Am Geriatr Soc 2009 Mar;57(3):476-481 [CrossRef] [Medline]
- Huang ES, Zhang Q, Gandra N, Chin MH, Meltzer DO. The effect of comorbid illness and functional status on the expected benefits of intensive glucose control in older patients with type 2 diabetes: a decision analysis. Ann Intern Med 2008 Jul 1;149(1):11-19 [CrossRef] [Medline]
- LeRoith D, Biessels GJ, Braithwaite SS, Casanueva FF, Draznin B, Halter JB, et al. Treatment of diabetes in older adults: an Endocrine Society* clinical practice guideline. J Clin Endocrinol Metab 2019 May 1;104(5):1520-1574 [CrossRef] [Medline]
- Kirkman MS, Briscoe VJ, Clark N, Florez H, Haas LB, Halter JB, et al. Diabetes in older adults. Diabetes Care 2012 Dec;35(12):2650-2664 [CrossRef] [Medline]
- Perna S, Francis MD, Bologna C, Moncaglieri F, Riva A, Morazzoni P, et al. Performance of Edmonton Frail Scale on frailty assessment: its association with multi-dimensional geriatric conditions assessed with specific screening tools. BMC Geriatr 2017 Jan 4;17(1):2 [CrossRef] [Medline]
- Goff DC, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014 Jun 24;129(25 Suppl 2):S49-S73 [CrossRef] [Medline]
|ADA: American Diabetes Association|
|ASCVD: Atherosclerotic Cardiovascular Disease|
|EHR: electronic health record|
|HbA1c: glycated hemoglobin|
|MDG: My Diabetes GOAL|
|PRO: patient-reported outcome|
|UCM: University of Chicago Medicine|
Edited by Nicole Ruggiano; submitted 03.11.22; peer-reviewed by Nestoras Mathioudakis, Takashi Kadowaki; final revised version received 13.06.23; accepted 19.09.23; published 09.11.23Copyright
© Sean Bernstein, Sarah Gilson, Mengqi Zhu, Aviva G Nathan, Michael Cui, Valerie G Press, Sachin Shah, Parmida Zarei, Neda Laiteerapong, Elbert S Huang. Originally published in JMIR Aging (https://aging.jmir.org), 9.11.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Aging, is properly cited. The complete bibliographic information, a link to the original publication on https://aging.jmir.org, as well as this copyright and license information must be included.