This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Aging, is properly cited. The complete bibliographic information, a link to the original publication on https://aging.jmir.org, as well as this copyright and license information must be included.
Background
eHealth technologies for self-management can improve quality of life, but little is known about whether the benefits gained outweigh their costs. The electronic patient-reported outcome (ePRO) mobile app and portal system supports patients with multiple chronic conditions to collaborate with primary health care providers to set and monitor health-related goals.
Objective
This study aims to estimate the cost of ePRO and the cost utility of the ePRO intervention compared with usual care provided to patients with multiple chronic conditions and complex needs living in the community, from the perspective of the publicly funded health care payer in Ontario, Canada.
Methods
We developed a decision tree model to estimate the incremental cost per quality-adjusted life year (QALY) gained for the ePRO tool versus usual care over a time horizon of 15 months. Resource utilization and effectiveness of the ePRO tool were drawn from a randomized clinical trial with 6 family health teams involving 45 participants. Unit costs associated with health care utilization (adjusted to 2020 Canadian dollars) were drawn from literature and publicly available sources. A series of sensitivity analyses were conducted to assess the robustness of the findings.
Results
The total cost of the ePRO tool was CAD $79,467 (~US $ 63,581; CAD $1733 [~US $1386] per person). Compared with standard care, the ePRO intervention was associated with higher costs (CAD $1710 [~US $1368]) and fewer QALYs (–0.03). The findings were consistent with the clinical evidence, suggesting no statistical difference in health-related quality of life between ePRO and usual care groups. However, the tool would be considered a cost-effective option if it could improve by at least 0.03 QALYs. The probability that the ePRO is cost-effective was 17.3% at a willingness-to-pay (WTP) threshold of CAD $50,000 (~US $40,000)/QALY.
Conclusions
The ePRO tool is not a cost-effective technology at the commonly used WTP value of CAD $50,000 (~US $40,000)/QALY, but long-term and the societal impacts of ePRO were not included in this analysis. Further research is needed to better understand its impact on long-term outcomes and in real-world settings. The present findings add to the growing evidence about eHealth interventions’ capacity to respond to complex aging populations within finite-resourced health systems.
Community-dwelling older adults (≥ 65 years old) with multiple chronic conditions and complex care consume a substantial amount of health care resources [1,2]. Existing evidence has shown that patients with multimorbidity have more frequent hospital admissions, longer hospital stays, and that health care costs exponentially increase with the greater number of health conditions, placing a high economic burden on health systems [3,4]. Provision of care is particularly challenging for this population due to the lack of specific assessment tools for multimorbidity, and the more complex management and coordination of care, which involves different professionals and clinical settings. The difficult management of multimorbidity—with guidelines that focus on single conditions, multiple therapies, and medications—can reduce treatment adherence and patients’ health-related quality of life (HRQoL) [2,4,5]. Moreover, older adults are at a higher risk of poor health outcomes given the complexity of social, environmental, and other contextual issues that they face within and outside the health system, such as social frailty and access to home and community care that meet their needs [6,7].
To address these challenges, there is growing interest in person-centered, integrated, and holistic care approaches that may help coordinate personalized and comprehensive care involving older adults, their caregivers, and health care providers [4,8]. Additionally, self-management programs have created efficiency gains, yielding improvements in health status and reductions in unnecessary health care utilizations [8]. However, there are few existing digital tools to enable these person-centered approaches for older adults in primary care settings [9]. The electronic patient-reported outcome tool (henceforth, called the ePRO tool) is one of such digital tools, which can facilitate collaborative care based on individualized goals created by older adults and providers, also known as goal-oriented care. The tool is delivered through the internet and mobile devices, and can be useful for complex care given their ability to improve access, continuity and efficiency of care, patient self-management, and communication [10].
A randomized trial had shown that ePRO plus usual care did not significantly improve the HRQoL in older adults with complex needs, partly due to recruitment challenges [10]. However, ethnographic data collected as part of the trial highlighted the importance of the coherence or meaningfulness of the intervention to the end users (ie, patients and providers) and uncovered the challenge to align coherence across diverse groups. When coherence was well aligned, users were more likely to see the value of the technology and use it more over time. In addition to assessing perceived value, there is a need to examine whether the challenges to improve clinical outcomes balance the additional investment in provider and technology costs associated with administering the ePRO tool within a clinical setting. While previous studies have shown eHealth interventions to be cost-effective [11], the cost-effectiveness of the ePRO tool, which was implemented in the community setting, has not been formally evaluated. This study was therefore conducted to estimate the cost of ePRO tool and examine whether the benefits gained from the tool outweighed its costs from the perspective of Canada’s publicly funded health care system.
MethodsStudy Design and Population
We performed a cost-utility analysis of the ePRO compared with standard care. The analysis was based on data from a pragmatic, stepped-wedge, cluster randomized trial with patients from 6 comprehensive primary care practices—called family health teams (FHTs)—across Ontario, Canada. FHTs provide integrated primary care, led by a physician or a nurse practitioner, and assisted by other professionals such as registered nurses, social workers, and dietitians [12]. A usual care pathway for older adults with multiple chronic conditions may include routine visits to their health care providers with or without their caregivers.
All FHT sites started in the control period, during which all recruited patients received usual care, and were randomly assigned to either the early or late intervention groups, with an initial control period of 3 and 6 months, respectively. The FHTs were then switched to the intervention period, during which patients and providers used ePRO as part of the primary care, for 12 months in the early intervention group and 9 months in the late intervention group. Enrollment occurred from January to August 2018, and the trial from April 2018 to June 2019.
Consistent with the trial, the study population for this cost-utility study was community-dwelling individuals aged 60 years or older with complex chronic conditions, defined as diagnosed with 2 or more chronic conditions and 10 or more visits to their primary health care provider within the past year. This number of visits has been identified as an indicator of complexity [10]. Chronic conditions were identified through the FHTs electronic medical records. Additional eligibility criteria included the perceived willingness to engage in goal-oriented care conversations, ability to use a smartphone or tablet, capable of providing consent to participate, and willing to complete surveys until completion of the trial. Detailed information on the trial can be found elsewhere [10].
Ethics Approval
Research ethics approval was granted by the University of Toronto’s Health Sciences Research Ethics Board (approval number 33944) and the ethics committees of all participating practices.
Intervention and Comparator
The development and usability of the ePRO tool were grounded in user-centered co-design, with a 4-phased approach [13-16]. The ePRO tool has 2 key features: (1) My Goals, which allows patients, caregivers, and providers to create goal-oriented patient care plans using a mobile device during a 15-30-minute care planning appointment. Specified-measurable-attainable-realistic-time–specific goal principles were used to guide goal setup and include free-form text to write down general feelings on progress; and (2) Outcome Measures, which helps patients, caregivers, and providers to monitor patient measures and outcomes (daily, weekly, or monthly) through validated and reliable health status scales such as Patient-Reported Outcomes Measurement Information System (PROMIS), Global Health Scale (GHS), Health Assessment Questionnaire (HAQ), 9-item Patient Health Questionnaire (PHQ-9) and Generalized Anxiety Disorder 7-Item (GAD-7) scale [9,17-19].
Given that eHealth tools to support self-management are an emerging class of technology, there are no comparable interventions identified for the analysis. Therefore, the standard care comparator in this analysis is multidisciplinary primary care provided by FHTs.
Time Horizon
The cost-utility analysis compared the costs and outcomes over 15 months of follow-up, which is consistent with the length of the stepped-wedge follow-up period. Costs and health outcomes were not discounted.
Measurement of Effectiveness
The ePRO effectiveness is measured as the effect of the intervention on HRQoL compared with usual care after adjusting for the family practice sites and multiple measurements at baseline, 3, 6, 9, 12, and 15 months. The trial measured QoL using the Assessment of Quality of Life 4-Dimension (AQoL-4D), a generic instrument that uses multiattribute utility theory, which makes it suitable for cost-utility evaluations [20]. It has also demonstrated validity in chronically ill community-dwelling populations [20]. At each time point, the individual responses of the AQoL-4D questionnaire were converted to weighted multiattributable utility values (ranging from 1.00 [full health] to 0.00 [death equivalent health states]). We used an area under the curve method to estimate quality-adjusted life years (QALYs) and a mixed effect linear regression to estimate the effect of ePRO on QALYs compared with usual care, while controlling for baseline utility values [21], the age of participants, sex, and number of comorbidities and baseline utilities. The baseline QALYs were informed by the observed data during the initial period of the stepped-wedge trial, when all patients received multidisciplinary primary care by Ontario FHTs for 3 or 6 months.
Resources and CostsOverview
We estimated costs from the perspective of a publicly funded health care system in the province of Ontario, Canada, and considered costs associated with ePRO and health care utilization.
Cost of the ePRO Tool
The cost of ePRO tool consisted of technology costs and training costs. Technology-related costs of the intervention were based on real-world costs incurred during the clinical trial. We excluded costs related to trial co-ordination and included any recurrent program costs borne to the government in future adoption. Cost sheets provided by the technology and research partners were stratified by different activities, the quantity used, and the price of each unit. The technology program costs comprised technology support, technology training, licensing, communication, onboarding management, app modification, new feature development, and professional services support costs (Table 1).
Input parameters of the model.
Parameter
Base estimate (2020 CAD $a)
Probability distribution (SD)
Data source
Professional services support
10,284
N/Ab
ePROc clinical trial
Technical support (before onboarding)
2971
N/Ab
ePRO clinical trial
Technical support (after onboarding)
2971
N/Ab
ePRO clinical trial
License
9182
N/Ab
ePRO clinical trial
Communication with health care teams
5942
N/Ab
ePRO clinical trial
Management (before onboarding)
8912
N/Ab
ePRO clinical trial
Management (after onboarding)
3714
N/Ab
ePRO clinical trial
Modifications
12,855
N/Ab
ePRO clinical trial
New features
15,405
N/Ab
ePRO clinical trial
Total technology costs
72,234
Gamma (11,131)
ePRO clinical trial
Health services utilization per person/year
5500
Gamma (541)
[22]
Total training costs
7233
Gamma (852)
Ministry of Health and Long-Term Care Report [23,24]
Reduction of health services utilization in the standard care group
0.03
Normal (0.003)
[25]
Reduction of health services utilization in the ePRO group
0.04
Normal (0.004)
[25]
Number of patients in the trial
45
N/A
ePRO clinical trial
aCAD $1=US $0.80.
bN/A: not applicable (used based estimates only).
cePRO: electronic patient reported outcome.
The training costs were included in the analysis as they were considered an opportunity cost. Although they did not represent an additional cost to the public payer, for the time of the training, the clinicians were not in their usual care routine. The training was offered to 5 FHT professionals involved in the delivery of the intervention: family physicians, registered nurses, nurse practitioners, registered social workers, and diabetes educators [26]. The training was administered in 2-hour sessions and was calculated using the hourly wages of the health care providers obtained from published provincial resources. Training costs, together with their sources, for the ePRO intervention are provided in Multimedia Appendix 1.
Cost Associated With Health Care Utilization
We obtained the cost associated with health care utilization among patients with complex needs from a retrospective population-based study reporting the health system costs associated with multimorbidity in Ontario [3]. Based on this study, we calculated the total health system costs for older adults living in community with 2-5 comorbidities; these costs were inflated to 2020 values using the health care–specific Consumer Price Index reported by Statistics Canada (Multimedia Appendix 2).
We derived the impact of ePRO on health care system costs based on the 13-item Patient Activation Measure (PAM), a validated tool that provides a weighted score on a scale of 0-100, with 4 associated activation levels, and is used to compare self-management capabilities [27]. In level 1, patients are aware of the importance of their role; in level 2, they have the confidence and knowledge necessary to take action regarding their treatment; in level 3, they actually take action and gain independence; and in level 4, they are able to maintain the behavior even under stress [27]. A score of less than 47.0 places a patient in level 1, 47.1-55.1 level 2, 55.2-72.4 level 3, and more than 72.5 in level 4. Self-management capacity levels, measured using PAM, are associated with reduced utilization across primary and secondary care [25,28,29]. In the absence of administrative and reliable self-reported utilization data, PAM level changes between pre- and postintervention were used to quantify changes in resource use cost from the baseline [25]. Research has demonstrated an inverse correlation between increasing PAM scores and reduction in cost, even after controlling for confounding demographic and comorbidity factors; therefore, it can be used as a credible proxy for future cost savings [25]. We based the percentage of reduction in health services utilization on literature findings of an 8% cost reduction per 1 increase in PAM level change [25]. The PAM questionnaire was administered to patient participants of the study at baseline, 3, 6, 9, 12, and 15 months. To calculate the cost reduction in each group, we multiplied the 8% reduction per level by the percentage of participants in each group that had their PAM levels increased while in that state. For example, if in one of the groups 10% of patients achieved a 1-level increase and 20% achieved a 2-level increase over the course of the trial, the total health system costs for that group were reduced by 24% (0.08 × 0.1 + 0.16 × 0.2).
Analysis
We used a decision tree to estimate the expected total costs and outcomes associated with ePRO and usual care after consultation with clinical partners. The tree splits into 2 branches, ePRO tool intervention and usual care. The incremental cost-effectiveness ratio of ePRO versus usual care was calculated by dividing the differences in total costs by the difference in QALYs. We conducted sensitivity analyses to address the uncertainties of our model and to better understand the impact of model assumptions on cost-effectiveness results, including a tornado diagram for the incremental net monetary benefit at a CAD $50,000 (~US $40,000)/QALY willingness-to-pay (WTP) threshold, and a probabilistic sensitivity analysis with 10,000 Monte Carlo simulations, sampled using the related distribution of the parameters (Table 1).
ResultsBase Case Analysis
The total cost of the ePRO tool was CAD $79,467 (~US $63,581; CAD $1733 [~US $1386] per person); of these, the technology component accounted for 90.89% (CAD $72,234 [~US $57,794]) of the total costs. The trial reported that ePRO combined with usual multidisciplinary care did not significantly impact patient HRQoL (P=.24) or patient activation (P=.17) [10]. Our base case analysis showed that, compared with the standard care, the addition of the ePRO intervention was associated with higher costs (CAD $7133 [US $5707] vs. CAD $5423 [US $4338] per patient) and slightly fewer QALYs (0.42 vs. 0.45) than usual care (Table 2). The technology cost was a key driver of an incremental cost. Although the ePRO intervention could reduce health system costs compared with standard care (CAD $5258 [US $4206] vs. CAD $5324 [US $4259], respectively), this saving was insufficient to offset the added technology cost.
Base case results.
Strategy
Mean costs (CAD $a)
Incremental costs (CAD $)
Mean QALYsb
Incremental QALYs
ICERc (CAD $/QALY)
Usual care
5423
—
0.45
—
—
ePROd
7133
1710
0.42
–0.03
Dominatede
aCAD $1=US $0.80.
bQALY: quality-adjusted life year.
cICER: incremental cost-effectiveness ratio.
dePRO: electronic patient-reported outcome.
eePRO was more costly and produced fewer QALYs than usual care.
Sensitivity Analyses
Results of the deterministic sensitivity analysis showed that the effectiveness of ePRO is the most influential driver of the cost-effectiveness findings. ePRO would be considered a cost-effective option, that is, having a positive incremental net monetary benefit, if it could improve by at least 0.03 QALYs. Other key determinants included the technology and the training costs for the implementation of the tool. However, individual variation in none of these costs could change the cost-effectiveness results (Figure 1). The probabilistic sensitivity analysis results showed that ePRO has a 17.3% probability of being cost-effective at the WTP threshold of CAD $50,000 (~US $40,000)/QALY (Figures 2 and 3); this probability increased to 25.1% if the WTP threshold increased to CAD $100,000 (~US $80,000)/QALY.
One-way sensitivity analysis (tornado diagram). ePRO: electronic patient-reported outcome; EV: expected value; IT: information technology; NMB: net monetary benefit; QALY: quality-adjusted life year;.
Our study showed the ePRO tool was not cost-effective at a commonly used WTP threshold of CAD $50,000 (~US $40,000)/QALY. The main driver of our cost-effectiveness results is the effectiveness of the ePRO tool on QALYs. Our results were robust to changes in input parameters and model assumptions with the probability of being cost-effective of 17.3% at the WTP values of CAD $50,000 (~US $40,000)/QALY.
Previous studies assessing the cost-effectiveness of eHealth interventions reported mixed findings, where some interventions were considered cost-effective or cost-saving, whereas other studies reported that eHealth interventions were not cost-effective [11,30]. Some economic evaluations of these technologies also reported comparable HRQoL among patients receiving eHealth and those receiving usual care [31-33]. Consistent with these results, our study showed that the addition of ePRO to usual care was not cost-effective because the tool generated fewer QALYs than usual care. Lower QALYs observed in this cost-utility analysis mirrored insignificant changes in HRQoL that might be due to the lack of statistical power resulting from recruitment challenges and low response rates to the surveys [31-33].
Some aspects of the ePRO tool must be considered when interpreting this economic evaluation. The ePRO tool is not targeted to patients with a specific disease, but rather to a heterogeneous population with a different number and variety of conditions. Hence, effectiveness of the ePRO tool may be different for patients with different conditions, given their complex needs and individual preferences [11]. Accessibility and digital literacy also have a direct impact on the effectiveness of eHealth interventions, as they can increase access and be convenient for some patients, but may also be inappropriate or inaccessible to others [7].
Furthermore, being a tool that is mainly focused on improving the patient’s engagement in the treatment and changing health behaviors, a trial-based cost-effectiveness analysis may not be the ideal design as it may take more than 10 years for health interventions to change patients’ behaviors and respective health outcomes [9,11,34]. With prolonged use of the tool, more significant effects may be detected, especially in chronic conditions that are mostly affected by changes in the behavior of the patients. Additional outcomes, such as patient and provider satisfaction, disease management, and self-care activities, with a longer time horizon could be useful in identifying whether ePRO had any other relevant benefits. Previous economic evaluations of eHealth technologies with no impact in clinical outcomes concluded that they were cost-effective due to their low cost and effectiveness when considering outcomes beyond HRQoL [32,35]. eHealth interventions can also have gains in efficiency with the increase in the number of users and sharing of data systems and infrastructure [31].
Another relevant consideration is that with prolonged use of the app and in less controlled conditions, patient adherence may decrease [36]. Data collected during the stepped-wedge trial indicated that adherence to the tool was moderate; however, this does not necessarily translate into fidelity to the model of care—a shift from the more classic, passive behavior of the patient toward their health to a model with higher engagement of both providers and patients in goal-setting conversations and oriented care [37,38]. It is possible that even with patients continually using the tool, the expected effects on health outcomes may not fully materialize if this shift in the model of care does not occur.
Study Limitations
This analysis had some important limitations. First, our economic evaluation was based on a single stepped-wedge trial, which limits the generalizability of our study for the real-world setting and a more heterogeneous population. Despite being a pragmatic trial that intended to approximate as much as possible the usual care conditions, the trial faced recruitment challenges and low response rates to outcome measures. Moreover, the trial was conducted within the context of FHTs, but ePRO could also be applicable to other models of primary care. The use of health care payer perspective in this analysis excludes some benefits that can add value to the ePRO tool. A societal perspective would allow the inclusion of indirect costs such as informal and unpaid time in caregiving, increased poverty, and loss of income due to time away from work [6]. For an intervention focused on primary care, out-of-pockets costs may be relevant, especially in health systems similar to Canada, where hospital-level costs are publicly funded, but many outpatient and nonphysician services are not funded. Some eHealth interventions were found to be cost-effective due to the inclusion of costs pertinent to a societal perspective [39,40].
Lastly, we used the PAM scores as a proxy to calculate the reduction in health care service utilization costs of the patients in the trial. Although an increase in patient activation levels has been associated with a reduction in these costs, these values can vary, and this was not tested in the Canadian setting [25,29]. However, our sensitivity analysis showed that our results were robust to a change in the effect of the ePRO tool on health care utilization.
Conclusion
Our cost-utility analysis highlighted that the ePRO tool is not a cost-effective technological solution for community-dwelling older adults with multiple chronic conditions when compared with usual care. However, the tool would become a cost-effectiveness option if it could improve QALYs by at least 0.03 unit. This study highlights the minimal effectiveness of eHealth solutions required to make the solutions cost-effective in response to the rising trends of complex, aging populations within finite-resourced health systems. Fidelity and adherence to the eHealth tools could improve their effectiveness, and this relationship should be investigated in future studies. Pragmatic trials with larger number of participants, fewer missing data, and longer follow-up time could help inform the implementation of the eHealth intervention, such as ePRO tool, in a finite-resourced setting.
Family health team provider training costs.
Health care utilization costs (person-level costing using administrative databases in Ontario).
AbbreviationsAQoL-4D
The Assessment of Quality of Life 4-Dimension
ePRO
electronic patient-reported outcome
FHT
family health team
GAD-7
Generalized Anxiety Disorder 7-Item scale
GHS
Global Health Scale
HAQ
Health Assessment Questionnaire
HRQoL
health-related quality of life
ICER
incremental cost-effectiveness ratio
PAM
Patient Activation Measure
PHQ-9
9-item Patient Health Questionnaire
PROMIS
Patient-Reported Outcomes Measurement Information System
QALY
quality-adjusted life year
WTP
willingness to pay
The co-authors acknowledge and thank the providers, administrators, patients, and research staff who participated in the clinical trial. Each of them gave their time, enthusiasm, and energy for which we are deeply grateful. We also thank and acknowledge research support from Phat Edward Chau, Jasvinei Srinatharan, Tujuanna Austin, Karen Paik, Candis LePage, Janelle Gravesande, Zaki Hyatt-Shaw, and Parminder Hans for their coordination support throughout this study. We also acknowledge the contribution of collaborators Drs Ross Upshur, Cheryl Cott, and Renee Lyons who offered early guidance in this work. Funding support for this project came from the Canadian Institutes of Health Research eHealth Innovation Partnership Program operating grant (number FN-143559).
Funding for this study is through a Canadian Federal Grant (CIHR; FN-143559), with the salaries of authors RHH, BCFC, CSG, WPW, and KT being funded through their academic and scientific positions at their respective institutions.
WodchisWPAustinPCHenryDAA 3-year study of high-cost users of health careCMAJ20160216188318218810.1503/cmaj.15006426755672cmaj.150064PMC4754179NavickasRPetricVFeiglABSeychellMMultimorbidity: What do we know? What should we do?J Comorb201602176141110.15256/joc.2016.6.7229090166joc.2016.6.72PMC5556462ThavornKMaxwellCJGruneirABronskillSEBaiYKoné PefoyoAnna JPetrosyanYWodchisWPEffect of socio-demographic factors on the association between multimorbidity and healthcare costs: a population-based, retrospective cohort studyBMJ Open20171006710e01726410.1136/bmjopen-2017-01726428988178bmjopen-2017-017264PMC5640069Ioakeim-SkoufaIPoblador-PlouBCarmona-PírezJonásDíez-ManglanoJesúsNavickasRGimeno-FeliuLAGonzález-RubioFranciscaJurevicieneEDambrauskasLPrados-TorresAGimeno-MiguelAMultimorbidity Patterns in the General Population: Results from the EpiChron Cohort StudyInt J Environ Res Public Health202006141712424210.3390/ijerph1712424232545876ijerph17124242PMC7345112GriffithLEGruneirAFisherKPanjwaniDGafniAPattersonCMarkle-ReidMPloegJInsights on multimorbidity and associated health service use and costs from three population-based studies of older adults in Ontario with diabetes, dementia and strokeBMC Health Serv Res2019051619131310.1186/s12913-019-4149-33109698910.1186/s12913-019-4149-3PMC6524233SchainkAKKuluskiKLyonsRFFortinMJadadARUpshurRWodchisWPA scoping review and thematic classification of patient complexity: offering a unifying frameworkJ Comorb20121010211910.15256/joc.2012.2.1529090137PMC5556402BaylissEBosworthHNoelPWolffJDamushTMciverLSupporting self-management for patients with complex medical needs: recommendations of a working groupChronic Illn20070629321677510.1177/1742395307081501180836713/2/167BurgersJSVoermanGEGrolRFaberMJSchneiderECQuality and coordination of care for patients with multiple conditions: results from an international survey of patient experienceEval Health Prof201009273333436410.1177/0163278710375695208019760163278710375695Steele GrayCWodchisWPUpshurRCottCMcKinstryBMercerSPalenTERamsayTThavornKProject Collaborators And Technology Partner‚ QoC Health IncSupporting Goal-Oriented Primary Health Care for Seniors with Complex Care Needs Using Mobile Technology: Evaluation and Implementation of the Health System Performance Research Network, Bridgepoint Electronic Patient Reported Outcome ToolJMIR Res Protoc2016062452e12610.2196/resprot.575627341765v5i2e126PMC4938886Steele GrayCChauETahsinFHarveySLoganathanMMcKinstryBMercerSWNieJXPalenTERamsayTThavornKUpshurRWodchisWPAssessing the Implementation and Effectiveness of the Electronic Patient-Reported Outcome Tool for Older Adults With Complex Care Needs: Mixed Methods StudyJ Med Internet Res202112022312e2907110.2196/2907134860675v23i12e29071PMC8726765IribarrenSJCatoKFalzonLStonePWWhat is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutionsPLoS One201722122e017058110.1371/journal.pone.017058128152012PONE-D-16-26479PMC5289471Ontario Ministry of HealthFamily Health TeamsOntario Ministry of Health2021-07-23https://www.health.gov.on.ca/en/pro/programs/fht/Steele GrayCGravesandeJHansPKNieJXSharpeSLoganathanMLyonsRCottCUsing Exploratory Trials to Identify Relevant Contexts and Mechanisms in Complex Electronic Health Interventions: Evaluating the Electronic Patient-Reported Outcome ToolJMIR Form Res2019022731e1195010.2196/1195030810532v3i1e11950PMC6414821Steele GrayCKhanAIKuluskiKMcKillopISharpeSBiermanASLyonsRFCottCImproving Patient Experience and Primary Care Quality for Patients With Complex Chronic Disease Using the Electronic Patient-Reported Outcomes Tool: Adopting Qualitative Methods Into a User-Centered Design ApproachJMIR Res Protoc2016021851e2810.2196/resprot.520426892952v5i1e28PMC4777883Steele GrayCGillAKhanAIHansPKKuluskiKCottCThe Electronic Patient Reported Outcome Tool: Testing Usability and Feasibility of a Mobile App and Portal to Support Care for Patients With Complex Chronic Disease and Disability in Primary Care SettingsJMIR Mhealth Uhealth2016060242e5810.2196/mhealth.533127256035v4i2e58PMC4911509Steele GrayCMillerDKuluskiKCottCTying eHealth Tools to Patient Needs: Exploring the Use of eHealth for Community-Dwelling Patients With Complex Chronic Disease and DisabilityJMIR Res Protoc2014112634e6710.2196/resprot.350025428028v3i4e67PMC4260075Steele GrayCIrfan KhanAMcKillopISharpeSCottCUser-centred co-design with multiple user groups: The case of the electronic Patient Reported Outcome (ePRO) mobile application and portalInt J Integr Care2019080819443910.5334/ijic.s3439Steele GrayCGillAKhanAIHansPKKuluskiKCottCThe Electronic Patient Reported Outcome Tool: Testing Usability and Feasibility of a Mobile App and Portal to Support Care for Patients With Complex Chronic Disease and Disability in Primary Care SettingsJMIR Mhealth Uhealth2016060242e5810.2196/mhealth.533127256035v4i2e58PMC4911509HansPKGrayCSGillATiessenJThe provider perspective: investigating the effect of the Electronic Patient-Reported Outcome (ePRO) mobile application and portal on primary care provider workflowPrim Health Care Res Dev2017913190215116410.1017/s1463423617000573OsborneRQuality of life assessment in the community-dwelling elderly Validation of the Assessment of Quality of Life (AQoL) Instrument and comparison with the SF-36Journal of Clinical Epidemiology20030256213814710.1016/s0895-4356(02)00601-7MancaAHawkinsNSculpherMJEstimating mean QALYs in trial-based cost-effectiveness analysis: the importance of controlling for baseline utilityHealth Econ2005051454879610.1002/hec.94415497198ThavornKWodchisWMaxwellCKonéAGruneirABronskillSEconomic Burden of Multimorbidity in Ontario's Health Care SystemCanadian Association for Health Services and Policy Research (CAHSPR)2013Legislative Assembly of Ontario2016 Annual Report of the Office of the Auditor General of OntarioPhysician Billing2016Toronto, ON, CanadaQueen’s Printer for OntarioMinistry of Health and Long-Term CareSalary Benchmarks for Interdisciplinary Health Care ProvidersFamily Health Teams: Guide to Interdisciplinary Provider Compensation20102021-07-23https://www.rtso.ca/wp-content/uploads/2015/06/MOHLTC-fht_inter_provider-Oct-2013.pdfLindsayAHibbardJHBoothroydDBGlaseroffAAschSMPatient Activation Changes as a Potential Signal for Changes in Health Care Costs: Cohort Study of US High-Cost PatientsJ Gen Intern Med201812533122106211210.1007/s11606-018-4657-63029160410.1007/s11606-018-4657-6PMC6258627GlazierRHRedelmeierDonald ABuilding the patient-centered medical home in OntarioJAMA20100602303212186710.1001/jama.2010.75320516421303/21/2186HibbardJStockardJMahoneyETuslerMDevelopment of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumersHealth Serv Res200408394 Pt 110052610.1111/j.1475-6773.2004.00269.x15230939HESR269PMC1361049BarkerISteventonAWilliamsonRDeenySRSelf-management capability in patients with long-term conditions is associated with reduced healthcare utilisation across a whole health economy: cross-sectional analysis of electronic health recordsBMJ Qual Saf20181223271298999910.1136/bmjqs-2017-00763530139822bmjqs-2017-007635PMC6288702GreeneJHibbardJHSacksROvertonVParrottaCDWhen patient activation levels change, health outcomes and costs change, tooHealth Aff (Millwood)201503343431710.1377/hlthaff.2014.04522573249334/3/431RyanDPriceDMusgraveSDMalhotraSLeeAJAyansinaDSheikhATarassenkoLPagliariCPinnockHClinical and cost effectiveness of mobile phone supported self monitoring of asthma: multicentre randomised controlled trialBMJ20120323344mar23 1e1756e175610.1136/bmj.e175622446569PMC3311462LuxtonDDHansenRNStanfillKMobile app self-care versus in-office care for stress reduction: a cost minimization analysisJ Telemed Telecare2014101420843143510.1177/1357633x14555616MaddisonRPfaeffliLWhittakerRStewartRKerrAJiangYKiraGLeungWDalleckLCarterKRawstornJA mobile phone intervention increases physical activity in people with cardiovascular disease: Results from the HEART randomized controlled trialEur J Prev Cardiol20150609226701910.1177/2047487314535076248176942047487314535076HagbergLABrekkeHKBertzFWinkvistACost-utility analysis of a randomized controlled weight loss trial among lactating overweight/obese womenBMC Public Health201401151413810.1186/1471-2458-14-38244288021471-2458-14-38PMC3910238SculpherMJClaxtonKDrummondMMcCabeCWhither trial-based economic evaluation for health care decision making?Health Econ2006071576778710.1002/hec.109316491461NundySDickJJChouCNoconRSChinMHPeekMEMobile phone diabetes project led to improved glycemic control and net savings for Chicago plan participantsHealth Aff (Millwood)2014023322657210.1377/hlthaff.2013.05892449377033/2/265PMC4034376KuluskiKGillANaganathanGUpshurRJaakkimainenRLWodchisWPA qualitative descriptive study on the alignment of care goals between older persons with multi-morbidities, their family physicians and informal caregiversBMC Fam Pract2013090814113310.1186/1471-2296-14-133240105231471-2296-14-133PMC3844424ReubenDBTinettiMEGoal-Oriented Patient Care — An Alternative Health Outcomes ParadigmN Engl J Med201203366977777910.1056/nejmp1113631MaddiganSLMajumdarSRJohnsonJAUnderstanding the complex associations between patient-provider relationships, self-care behaviours, and health-related quality of life in type 2 diabetes: a structural equation modeling approachQual Life Res200508146148950010.1007/s11136-005-0586-z16110929ArmstrongKASempleJLCoytePCReplacing ambulatory surgical follow-up visits with mobile app home monitoring: modeling cost-effective scenariosJ Med Internet Res20140922169e21310.2196/jmir.352825245774v16i9e213PMC4211032WittenbornJSClemonsTRegilloCRayessNLiffmann KrugerDReinDEconomic Evaluation of a Home-Based Age-Related Macular Degeneration Monitoring SystemJAMA Ophthalmol20170501135545245910.1001/jamaophthalmol.2017.0255283589482613353PMC5470421