This program evaluation used a qualitative longitudinal design, collecting data across different stages of development and implementation of the tele-GMH teams over a 3-year period. Interviews were conducted between February 2022 and September 2024 via convenience sampling. The initial study design was guided by CFIR 1.0 [29]; however, the study design was updated, and all analysis was conducted based on CFIR 2.0 [28]. Findings reported are based on CFIR 2.0.

We used a convenience sampling approach, focused on trying to reach all the clinicians from the 8 CRHs that provided GMH services or specialty geriatrics services between October 2021 and September 2024 to participate in one or more interviews. We invited 26 of the 27 clinicians; we were unable to invite one clinician because they left their VHA position before the invitation was sent. All clinicians served rural veterans, as identified using the rural-urban commuting area codes used by the VHA Office of Rural Health [1]. We refer to the services as tele-GMH, as the majority of clinicians and services are focused on GMH.

Led by the project director (CEG), the analysis team (MBH, ADP, and RBT) developed semistructured interview guides to be administered to clinicians at 3 time points corresponding to each service’s startup (time point 1), development (time point 2), and sustainment (time point 3) stages. Interview guides are available from the first author (CEG) by request; the overarching purpose of these guides was related to the multiyear evaluation with varying data collected at each time point.

During the time point 1 interviews, clinicians were asked about factors related to the development of their service, reasons why they joined the CRH, services provided, common reasons for referral and referral sources, regularly used measures, team composition, and team collaboration. Overall, time point 1 focused on CFIR domains related to the innovation and characteristics of the inner setting.

Time point 2 interviews asked clinicians about planning and implementation of services, facilitators and barriers of implementation, service impacts on rural veterans and their caregivers, service impacts on referring clinicians and others in the health care system, outcomes to track, and reflections on the sustainment of the service. This interview guide included questions that mapped onto constructs in CFIR 1.0. Time point 2 interviews gathered information about changes to the innovation and the inner setting and the initial implementation process.

Time point 3 interviews focused on relationships with referral sources, service impacts on referring clinicians and rural veterans, barriers and facilitators to service delivery, and geriatric medicine needs [30] for rural veterans. This interview guide asked questions about any changes to innovation, the inner setting, and the implementation process.

We aimed to interview all clinicians at time point 1, then used more purposive sampling at time points 2 and 3 to ensure that perspectives were gathered from multiple disciplines and from every CRH team. This purposive approach for time points 2 and 3 allowed for gathering perspectives from every region with services while minimizing time burden on the clinicians delivering services.

The project was reviewed by the Stanford University IRB, the IRB of record for VA Palo Alto Health Care System, and determined that this work did not meet the definition of human subjects research (review numbers: IRB-65150, IRB-70700, and IRB-73787) and program evaluation was conducted to inform VA operations. Union concurrence was obtained prior to conducting interviews. All participants provided verbal consent prior to their virtual interviews.

In accordance with VHA policy, interview guides were sent to the unions for review and concurrence. Participants were emailed an invitation to engage in interviews approximately annually. One reminder email was sent to participants who did not respond to the first invitation. Interviews were conducted and recorded via videoconferencing technology. All interviewers were trained in qualitative interviewing by the qualitative methodologist (ADP).

The evaluation team consisted of a geropsychologist/project director (CEG), project coordinators (CC and MFL), a qualitative analyst (MBH), a qualitative methodologist (ADP), geropsychologist (AL), a health psychologist (RBT), and a geropsychology postdoctoral fellow (LP). Interviews were conducted by 5 authors (CEG, MBH, CC, MFL, and LP). All evaluation team members participated in the analysis (see Multimedia Appendix 1 for reflexivity statements from the evaluation team). Biases were managed through discussion, weekly qualitative analysis meetings, and multiple authors reviewing narrative summaries.

Interviews were auto-transcribed using Microsoft Teams software. Team members proofread transcriptions against the audio recordings to ensure that transcripts were verbatim. Led by the project director (CEG), the team developed templates to summarize transcripts into domains (key topics of interest) that corresponded to the interview guides. Using rapid qualitative template-based analysis [31], one individual summarized each transcript, and a second individual reviewed the transcript summary; then both met to review the summary and resolve any discrepancies by consensus. Data saturation was determined through review of the transcript summaries and the written domain narratives that synthesized data across each summary. Review occurred in pairs; the full team discussed findings during twice-monthly meetings. Domain narratives with similar content were combined. This analysis focused on 8 domain summaries, including joining CRH; name of service; planning process; advertising; determinants; relationships with referral sources; services/challenges; and outcomes, sustainability, and service impacts. Two authors (CEG and MBH) used a matrix template based on CFIR 2.0 [28,32] to code the domain summaries to identify the key components of CFIR underlying our findings. These 2 authors also used the 8 domain narratives to generate the implementation roadmap. The roadmap was reviewed and refined during qualitative team meetings. The finalized implementation roadmap was reviewed with the project advisory team, which included 2 CRH geriatric psychiatrists, a CRH director, and a national leader in GMH.

The evaluation team developed materials summarizing key aspects related to the development and implementation of tele-GMH services in the CRHs, outlining steps sites may consider in the planning and implementation process. These materials were reviewed with CRH clinicians during monthly consultation meetings and were distributed to CRH clinicians via email. Minor changes (focused primarily on CRH-specific processes) were made based on feedback. This process enabled the evaluation team to engage in member checking [33] through sharing our understanding of implementation processes with the interviewees themselves.

Forty-six interviews were conducted, representing perspectives from 25 clinicians across 8 CRHs. Most clinicians were geriatric psychiatrists (10/25, 40%) or geropsychologists (6/25, 24%). Other clinicians included neuropsychologists, clinical pharmacists, and a geriatrician (see Table 1). The CRH services primarily encompassed tele-GMH services, with later expansions including geriatric medicine. All models (n=23) included at least one GMH clinician and delivered synchronous telehealth services to home or clinic (see Table S1 in Multimedia Appendix 1). Teams provided services for veterans at spoke sites, including community-based outpatient clinics, smaller VHA medical centers, and video to home, including for patients receiving home-based primary care (HBPC) services. Two teams also provided telehealth services to community living centers (CLCs), VHA’s nursing home equivalent.

We identified the key steps taken to implement tele-GMH services, organized into three main phases: (1) preimplementation, (2) implementation scale-up and expansion, and (3) sustainment. As seen in Figure 1, the steps were cyclical with iterative changes over time. The following describes the steps and relevant determinants of implementation in each phase, with particular attention to the implementation of care for rural veterans.

The scale-up and expansion of the tele-GMH services was an iterative process, as represented by the cyclical phase of Figure 1. See Table 3 for examples of the CFIR constructs related to the implementation process domain.

The maintenance and sustainment phase encompassed stability in the interprofessional teams and focused on collaboration with local clinicians. Other key aspects of the maintenance phase were reflections on the impact of the service on patients, caregivers, referring clinicians, and an overall increased access to geriatrics specialty care, as displayed in Table 4, and found within the CFIR construct of reflecting and evaluating.

Key facilitators to maintenance of services were positive interprofessional relationships, including mentoring new team members. One geropsychologist (ID 7) said of the founding tele-GMH team member:

Similarly, a clinical pharmacy practitioner (ID 12) felt their team nurse was “fantastic… she’s been instrumental for some success” because the nurse holds multiple roles, like helping psychology with overflow work, triaging patients, and contacting veterans and facilitating connecting them with tech help. Hiring of support staff, such as a dedicated medical support assistant to assist with scheduling, was deemed to be very supportive, enabling the teams’ ongoing success.

Using qualitative methods with multiple interviews conducted with tele-GMH clinicians across a 3-year period, we identified the key activities and determinants of success across three phases: (1) preimplementation, (2) implementation scale-up and expansion, and (3) sustainment of tele-GMH services for older veterans.

The preimplementation phase encompassed the crucial steps of engaging leaders, securing funding/hiring, and defining services. A key determinant of success during this phase was the use of adaptations to the tele-GMH model to meet local needs and priorities. Also important were the characteristics of the individuals involved in developing and delivering the tele-GMH services. The CFIR domains of capability (ie, expertise in GMH) and motivation to care for rural and underserved older veterans were factors that led clinicians to these CRH positions. Expanding virtual positions may be one way to address the persistent geriatrics workforce shortage and enable specialist clinicians to serve the population they trained to serve. Moreover, remote work has been found to be associated with low burnout, particularly for female health care workers [34]. Thus, expanding virtual positions may help individual clinicians while also filling the geographic need for geriatrics care in rural regions [35].

The implementation scale-up and expansion phase was cyclical and iterative, focused on building a referral network and expanding or maintaining services offered to older veterans. Key determinants in this phase included building and maintaining relationships with local clinicians, teaming (or intentional collaboration), doing (making iterative changes to processes), tailoring services for sites, and adapting services in response to feedback. This fits with findings on the importance of partnership alignment, communication, and feedback as determinants of implementing comprehensive geriatrics assessment [36]. A key determinant to both preimplementation and implementation was having the tele-GMH team embedded within CRH, which allowed the clinicians to take advantage of CRH operational knowledge to initiate services. This likely allowed the tele-GMH clinicians to focus on building relationships and designing their program while CRH staff guided them in the logistics of telehealth service delivery. This echoes previous work that identified the critical role of administrative staff in supporting virtual services [37].

Tele-GMH clinicians described periods of maintenance followed by refinement and growth in the third phase. Barriers to implementation, including patient-level factors, virtual care obstacles, and unfamiliarity with local aging resources, were addressed using refinement to processes and engaging nurse case managers, social workers, and telehealth clinical technicians. As found in our previous work with surveys of referring clinicians [22], several barriers impede the uptake of services, including lack of clarity about consultation process logistics (ie, who can place consultations and from which settings), perceptions that older veterans may not benefit from telemedicine, and not following tele-GMH clinician recommendations. Facilitators of the maintenance phase include clear messaging, teaming, relationship building within teams, and care coordination. The maintenance phase warrants future study to better understand the mechanisms underlying high-functioning remote health care teams and how these teams communicate with local VHA and community clinicians. This communication and care coordination is critical as many rural veterans use both VHA and community care [38]. Furthermore, there is a need to understand patient perspectives as they pertain to working with remote teams. A mixed methods study found unclear and overlapping health care clinician roles and limited care coordination to be barriers to the effective virtual primary care for patients with chronic diseases [39].

The growing shortage of both mental health and geriatric clinicians demands innovative and scalable solutions and programs to address the needs of older adults. This work dovetails with calls for integrated geriatric clinical service lines rather than disease-focused service lines [35]. However, due to workforce shortages [2,3] and the limited availability of geriatrics clinicians in rural areas, virtual care models must be considered. Tele-GMH has the potential to be such a program due to the adaptability of the model, sustainability of service delivery with embedded administrative support, and attraction of specialist clinicians to serve older veterans. Strengths of our study include using CFIR to identify determinants of success and barriers to implementation that can be applied in similar program development in other health care systems. Although the tele-GMH service may be adapted to care for veterans with complex needs, broader initiatives to transform health care, such as “Age Friendly Health Systems,” must be pursued to improve care for all older adults [40].

Several limitations to this qualitative evaluation warrant consideration. First, the findings are limited to perspectives on VHA and thus may not directly apply to other health care systems. Second, we do not include quantitative outcomes or perspectives from patients or caregivers, which limits the interpretation of the acceptability of the findings and impact on service recipients. Third, our findings rely on tele-GMH clinician self-report, which may not fully capture all VHA perspectives in this report (ie, referring clinician, VHA leadership), although those perspectives have been reported elsewhere [21-23]. Fourth, we did not interview every tele-GMH clinician at each time point, potentially biasing our findings. Fifth, our findings may be biased by sampling of clinicians who chose to work in these roles in the CRH. Sixth, the rapid analytic approach fit with our team-based synthesis; however, it does not allow for the line-by-line analytic depth of traditional qualitative coding approaches.

Our findings provide evidence of implementation factors related to virtual GMH care, which was noted as a main weakness in the broader literature on geriatrics care models [41]. Similar processes emerged across regions in the development and sustainment of tele-GMH services, which allowed for the creation of a unified roadmap. The roadmap provides a pathway that other regions may follow to create tele-GMH services to enhance access to specialty care for aging rural veterans. Our findings underscore the role of VHA as a learning health system [42] where experts and innovators can pilot and refine new models of care.