Monitoring Intervention Fidelity and Adaptations
Section 5
Intervention Adaptation Strategies and Examples
This section describes strategies to anticipate how to work with health systems that could potentially adapt the ePCT intervention, and includes real-world case studies from the NIH Collaboratory Trials.
Common Strategies to Anticipate Changes and Monitor Adherence and Adaptations
| Build and nurture relationships with health system partners | Consider using different communication strategies and communicate often. For example, ask a chief medical officer to write an e-mail to explain the importance of the intervention. Conduct debriefing meetings with site teams to understand what unplanned changes have occurred or might be planned in the near future. |
| Document aspects of the intervention that are essential features | Assess how much room there is to modify the intervention from the target. For more, see the PCORI methodology standards for studies of complex interventions |
| Request, monitor, and act on data regularly (eg, monthly, quarterly) | Detect and assess changes made within the healthcare setting and to the intervention. For example, if the researcher has data on orders (eg, time frame for a dialysis session), they could assess if and when they were changed. |
| Monitor for change over time | External changes, including changes to guidelines, reimbursement policy, the electronic health record, and changes to public health may all impact standard-of-care and/or the implementation of the intervention. |
| Periodically verify that the software or EHR is functioning as expected | For example, to assess whether manual changes were made by a physician. |
| Offer training to staff to encourage fidelity | Training can be in-person or virtual, half-day or full-day. Repeat training sessions can be offered as a booster. |
| Conduct brief qualitative check-in meetings or site visits | Use ethnographic methods to observe, document, and learn how the intervention is being implemented and which variations and different levels of intensity are being used. Document anecdotes and stories that go along with the intervention over time. This information could inform your results and dissemination products like toolkits that a healthcare system could use after the trial is over. |
Intervention Adaptation Scenarios from the NIH Collaboratory Trials
“Begin with the mindset that there will be some adaptations during the trial. There will be multiple time points along the way for adaptations, including those during program delivery and at the end (eg, post-lessons learned) and before finalizing toolkits (eg, implementation and adaptation guides).” –Russell Glasgow, PhD (Nudge Study Dissemination & Implementation Workgroup)
Next we describe a few real-world scenarios from the NIH Collaboratory Trials. All these trials are PCTs embedded in healthcare systems. Scenarios include considerations for whether the study team has an a priori plan to anticipate and handle changes; whether the team uses a particular framework or method to track modifications; and how the team plans to work with their partners to adapt the intervention.
Case Study: PRIM-ER
Background: The PRIM-ER (Primary Palliative Care for Emergency Medicine) NIH Collaboratory Trial is an ongoing ePCT evaluating the effect of a primary palliative care program on outcomes for older adults with serious illness in diverse emergency department (ED) settings. Using a pragmatic, cluster-randomized, stepped-wedge design, the study is being conducted across a diverse group of 35 EDs that vary in specialty geriatric and palliative care capacity, geographic region, payer mix, and demographics. The intervention includes evidence-based, multidisciplinary primary palliative care education; simulation-based workshops on communication in serious illness; clinical decision support; and provider audit and feedback. The hypothesis is that older adult visitors with serious, life-limiting illness cared for by providers with primary palliative care skills will be less likely to be admitted to an inpatient setting; more likely to be discharged home or to palliative care service; and will have higher home health and hospice use, fewer inpatient days, ICU admissions at 6 months, and longer survival than patients seen before implementation.
Intervention monitoring plan: Over the study period, the team documented and evaluated changes as they occurred within the participating health systems. The team tracked modifications at each site to better understand their impact on intervention fidelity. And, as described in the trial’s protocol publication (Grudzen et al. 2019), the RE-AIM framework will be used to analyze the quality of the implementation.
A complex intervention such as PRIM-ER consists of the core features (i.e., functions) and the components and processes that promote the core features (i.e., forms). Before the full implementation phase of the trial, the study team identified the core functions that needed to be standardized across all intervention sites as well as the forms that could be adapted. The team remains flexible and transparent in communicating with key stakeholders about the components that can and cannot be tailored so that the study retains a level of standardization and integrity in design. For maximum buy-in from stakeholders, the study team is encouraging sites to adapt, when possible, the processes for each intervention component to their local site’s context in order to carry out the deliverables of the intervention. The team will collaborate with each site and assess and approve any new processes that arise to ensure the fidelity of the intervention while allowing adaptation to local context. The team will also share suggestions with stakeholders about what worked at other sites that have completed the intervention.
If substantial changes occurred in leadership, the PRIM-ER study team organized a teleconference with the new and existing leaders in the health system. During the call, expectations and potential strategies to maximize continued engagement and partnership were discussed. Upon completion of each site’s 3-week intervention period, the program manager completed post-intervention reflection notes that identified the impact of system changes on the intervention.
The PRIM-ER study team also used the RE-AIM framework to evaluate specific components of the intervention. For example, the PRIM-ER team evaluated the clinical decision support (CDS) tool at the delivery-site level through mixed-methods analysis. This included a 12 month-post intervention site level 30 minute quantitative survey and qualitative interviews of Principal Investigators and physician champions. The post-implementation survey was developed using the RE-AIM framework and it served multiple purposes. It was an opportunity to 1) ensure the clinical support tool screenshots and detailed information collected during implementation were accurate, 2) understand barriers and facilitators in implementation and maintenance 3) assess if changes had been made post-intervention to the CDS and the drivers for change and 4) understand COVID-19 impacts if any as half of the sites implemented the intervention pre-COVID. The qualitative interviews were conducted as part of a larger assessment to understand the best practice alert implementation adoption, effectiveness, and maintenance processes. Results of this evaluation is currently under peer review.
“The application of theory and delineation of forms and functions, as well prospective adaptation monitoring of large complex interventions can support the balance of fidelity with adaptability to encourage successful interventions among a variety of clinical environments.” (Hill et al. 2020)
Other Examples of Monitoring Plans from the NIH Collaboratory Trials
| Study | Intervention | Monitoring Plan |
| Nudge
Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications |
Tests effectiveness of automated mobile phone text reminders sent at scale based on pharmacy data for patients to refill medications with the goal of improving adherence and outcomes | The team is monitored for system changes that affected the intervention through monthly meetings with all site PIs. Quarterly partner meetings with patients, providers, and health system leaders were held to keep them apprised of study updates and to obtain feedback about study issues.
Modifications of study procedures were tracked and evaluated for their impact on intervention fidelity and outcomes. The team used the RE-AIM framework to evaluate the implementation, short-term sustainability, and dissemination of the intervention. |
| LIRE
Lumbar Imaging with Reporting of Epidemiology |
Tests effectiveness of a simple intervention whereby epidemiologic benchmarks are inserted into lumbar spine imaging reports | Although the team did not use a formal framework, they tracked EHR data every 6 months to verify whether the intervention text was being implemented as designed. Troubleshooting by the site PI uncovered instances where it was not being delivered. A switch in the radiology reporting system led to a break in the automatic insertion of the intervention text. |
| PROVEN
Pragmatic Trial of Video Education in Nursing Homes |
Offers and shows an advance care planning video to patients admitted to the nursing home within partner nursing home systems | The team did not use a formal framework but did employ a structured “real-time” monitoring of the video intervention fidelity at 119 nursing homes. They worked with the facilities to integrate a novel report in the EHR that provided documentation about whether the video was offered and shown to patients as per the implementation protocol.
Facility-level adherence reports (proportion of enrolled patients offered and shown a video) were created by the research team that were provided as feedback to the program implementation site leader at each facility. At monthly telephone meetings, the PROVEN implementation team reviewed these reports with the site leaders and discussed strategies to improve adherence. Facility site leaders were asked every 6 months about existing or new initiatives external to the trial that focused on advance care planning or reduction in hospital transfers. |
| ICD-Pieces
Improving Chronic Disease Management with Pieces |
Uses a novel technology platform to enable the use of EHR data to improve care of patients with the triad of chronic kidney disease, diabetes, and hypertension within primary care practices or medical homes in the community | The team did not use a framework for monitoring but did conduct regular calls with their partners to learn about and document changes to the implementation. The team had planned to monitor fidelity through automatic detection of order sets, but had to subsequently add manual efforts.
The Data and Safety Monitoring Board (DSMB) asked the team to come up with ways to measure fidelity through tracking of various metrics related to implementation of the intervention and monitor for separation between control and experimental groups. The study team was blinded to this comparison, but the DSMB had access to and reviewed these data. To document the changes sites made, every week the practice facilitators reviewed lists of patients that should have been flagged and enrolled compared with those who were not enrolled. Discrepancies were communicated with the clinic by providing updated patient lists or by phone. Manual reminders for who was eligible to clinic staff were also successful to supplement the automated process. |
SECTIONS
REFERENCES
Grudzen CR, Brody AA, Chung FR, et al. Primary Palliative Care for Emergency Medicine (PRIM-ER): Protocol for a pragmatic, cluster-randomised, stepped wedge design to test the effectiveness of primary palliative care education, training and technical support for emergency medicine. BMJ Open. 2019;9:e030099. doi: 10.1136/bmjopen-2019-030099.
Hill J, Cuthel AM, Lin P, Grudzen CR. 2020. Primary Palliative Care for Emergency Medicine (PRIM-ER): Applying form and function to a theory-based complex intervention. Contemporary Clinical Trials Communications. 18:100570. doi:https://doi.org/10.1016/j.conctc.2020.100570.