Monitoring Intervention Fidelity and Adaptations
Intervention Adaptation Strategies and Examples
Allison Cuthel, MPH
Liz Wing, MA
This section describes strategies to anticipate how to work with health systems that could potentially adapt the ePCT intervention, and includes real-world case studies from the Collaboratory’s Demonstration Projects.
Common Strategies to Anticipate Changes and Monitor Adherence and Adaptations
|Build and nurture relationships with health system partners||Consider using different communication strategies and communicate often. For example, ask a chief medical officer to write an e-mail to explain the importance of the intervention. Conduct debriefing meetings with site teams to understand what unplanned changes have occurred or might be planned in the near future.|
|Document aspects of the intervention that are essential features||Assess how much room there is to modify the intervention from the target.|
|Request, monitor, and act on data regularly (eg, monthly, quarterly)||Detect and assess changes made within the healthcare setting and to the intervention. For example, if the researcher has data on orders (eg, time frame for a dialysis session), they could assess if and when they were changed.|
|Periodically verify that the software or EHR is functioning as expected||For example, to assess whether manual changes were made by a physician.|
|Offer training to staff to encourage fidelity||Training can be in-person or virtual, half-day or full-day. Repeat training sessions can be offered as a booster.|
|Conduct brief qualitative check-in meetings or site visits||Use ethnographic methods to observe, document, and learn how the intervention is being implemented and which variations and different levels of intensity are being used. Document anecdotes and stories that go along with the intervention over time. This information could inform your results and dissemination products like toolkits that a healthcare system could use after the trial is over.|
Intervention Adaptation Scenarios from the Demonstration Projects
“Begin with the mindset that there will be some adaptations during the trial. There will be multiple time points along the way for adaptations, including those during program delivery and at the end (eg, post-lessons learned) and before finalizing toolkits (eg, implementation and adaptation guides).” –Russell Glasgow, PhD (Nudge Study Dissemination & Implementation Workgroup)
Next we describe a few real-world scenarios from the Collaboratory Demonstration Projects. All these trials are PCTs embedded in healthcare systems. Scenarios include considerations for whether the study team has an a priori plan to anticipate and handle changes; whether the team uses a particular framework or method to track modifications; and how the team plans to work with their stakeholders to adapt the intervention.
Case Study: PRIM-ER
Background: The PRIM-ER (Primary Palliative Care for Emergency Medicine) Demonstration Project is an ongoing ePCT evaluating the effect of a primary palliative care program on outcomes for older adults with serious illness in diverse emergency department (ED) settings. Using a pragmatic, cluster-randomized, stepped-wedge design, the study is being conducted across a diverse group of 35 EDs that vary in specialty geriatric and palliative care capacity, geographic region, payer mix, and demographics. The intervention includes evidence-based, multidisciplinary primary palliative care education; simulation-based workshops on communication in serious illness; clinical decision support; and provider audit and feedback. The hypothesis is that older adult visitors with serious, life-limiting illness cared for by providers with primary palliative care skills will be less likely to be admitted to an inpatient setting; more likely to be discharged home or to palliative care service; and will have higher home health and hospice use, fewer inpatient days, ICU admissions at 6 months, and longer survival than patients seen before implementation.
Intervention monitoring plan: Over the study period, the team will document and evaluate changes as they occur within the participating health systems. The team plans to use the FRAME model to track modifications at each site to better understand their impact on intervention fidelity. And, as described in the trial’s protocol publication (Grudzen et al. 2019), the RE-AIM framework will be used to analyze the quality of the implementation.
A complex intervention such as PRIM-ER consists of core features and processes that promote the core features. Before the full implementation phase of the trial, the study team identified the core features that need to be standardized across all intervention sites as well as the components and processes of the features that may be adapted. The team remains flexible and transparent in communicating with key stakeholders about the components that can and cannot be tailored so that the study retains a level of standardization and integrity in design. For maximum buy-in from stakeholders, the study team is encouraging sites to adapt, when possible, the processes for each intervention component to their local site’s context in order to carry out the deliverables of the intervention. The team will collaborate with each site and assess and approve any new processes that arise to ensure the fidelity of the intervention while allowing adaptation to local context. The team will also share suggestions with stakeholders about what worked at other sites that have completed the intervention.
If substantial changes occur in leadership, the PRIM-ER study team will organize a teleconference with the new and existing leaders in the health system. During the call, expectations and potential strategies to maximize continued engagement and partnership will be discussed. Upon completion of each site’s 3-week intervention period, the program manager will complete post-intervention reflection notes that identify the impact of system changes on the intervention.
Other Examples of Monitoring Plans from the Collaboratory Demonstration Projects
Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications
|Tests effectiveness of automated mobile phone text reminders sent at scale based on pharmacy data for patients to refill medications with the goal of improving adherence and outcomes||The team is monitoring for system changes that affect the intervention through monthly meetings with all site PIs. Quarterly stakeholder meetings with patients, providers, and health system leaders are held to keep them apprised of study updates and to obtain feedback about study issues.
Modifications of study procedures are being tracked and evaluated for their impact on intervention fidelity and outcomes. The team plans to use the RE-AIM framework to evaluate the implementation, short-term sustainability, and dissemination of the intervention.
Lumbar Imaging with Reporting of Epidemiology
|Tests effectiveness of a simple intervention whereby epidemiologic benchmarks are inserted into lumbar spine imaging reports||Although the team did not use a formal framework, they tracked EHR data every 6 months to verify whether the intervention text was being implemented as designed. Troubleshooting by the site PI uncovered instances where it was not being delivered. A switch in the radiology reporting system led to a break in the automatic insertion of the intervention text.|
Pragmatic Trial of Video Education in Nursing Homes
|Offers and shows an advance care planning video to patients admitted to the nursing home within partner nursing home systems||The team did not use a formal framework but did employ a structured “real-time” monitoring of the video intervention fidelity at 119 nursing homes. They worked with the facilities to integrate a novel report in the EHR that provided documentation about whether the video was offered and shown to patients as per the implementation protocol.
Facility-level adherence reports (proportion of enrolled patients offered and shown a video) were created by the research team that were provided as feedback to the program implementation site leader at each facility. At monthly telephone meetings, the PROVEN implementation team reviewed these reports with the site leaders and discussed strategies to improve adherence.
Facility site leaders were asked every 6 months about existing or new initiatives external to the trial that focused on advance care planning or reduction in hospital transfers.
Improving Chronic Disease Management with Pieces
|Uses a novel technology platform to enable the use of EHR data to improve care of patients with the triad of chronic kidney disease, diabetes, and hypertension within primary care practices or medical homes in the community||The team did not use a framework for monitoring but did conduct regular calls with their stakeholders to learn about and document changes to the implementation. The team had planned to monitor fidelity through automatic detection of order sets, but had to subsequently add manual efforts.
The Data and Safety Monitoring Board (DSMB) asked the team to come up with ways to measure fidelity through tracking of various metrics related to implementation of the intervention and monitor for separation between control and experimental groups. The study team is blinded to this comparison, but the DSMB has access to and reviews these data.
To document the changes sites made, every week the practice facilitators reviewed lists of patients that should have been flagged and enrolled compared with those who were not enrolled. Discrepancies were communicated with the clinic by providing updated patient lists or by phone. Manual reminders for who was eligible to clinic staff were also successful to supplement the automated process.
Grudzen CR, Brody AA, Chung FR, et al. Primary Palliative Care for Emergency Medicine (PRIM-ER): Protocol for a pragmatic, cluster-randomised, stepped wedge design to test the effectiveness of primary palliative care education, training and technical support for emergency medicine. BMJ Open. 2019;9:e030099. doi: 10.1136/bmjopen-2019-030099.