“Fidelity to the intervention is not necessarily paramount—adaptability and adoption are important too.” –Eric Larson, NIH Collaboratory Health Care Systems Interactions Core
Whether study teams choose to use a formal framework to monitor fidelity and adaptations, tracking such modifications will be key to ensuring that you understand the internal validity of the study. Next, we introduce a few frameworks and resources for study teams to consider.
The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) is an approach developed, and recently expanded, by Wiltsey Stirman and colleagues to help study teams identify and report modifications to interventions or implementation strategies—planned and unplanned (Stirman 2013; Wiltsey Stirman 2019). FRAME, an update and synthesis of earlier adaptation research and models (Glasgow et al. 1999), helps researchers compare the impact of a modification with the impact on fidelity to the intervention. The FRAME approach can be used to track and document aspects of the intervention’s implementation such as why, when, and where the change occurred; the nature of the change; the target of the change; and, importantly, the goal of the change—for example, to improve effectiveness, increase reach or engagement, or reduce cost.
The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was designed to help study teams evaluate “essential program elements including external validity that can improve the sustainable adoption and implementation of effective, generalizable, evidence-based interventions” (Glasgow et al. 1999; www.re-aim.org). RE-AIM has the goal of improving robustness, translatability, and public health impact of an intervention in real-world settings (Glasgow et al. 2019). RE-AIM encompasses both individual- and setting-level impacts. Recent advances to the RE-AIM framework focus on pragmatic aspects including measuring feasibility and context in both planning and evaluation of an intervention. RE-AIM can be used iteratively to guide adaptations during the conduct of a study.
The Practical Robust Implementation and Sustainability Model (PRISM) is a framework that integrates with the RE-AIM model. PRISM can be used to assess how contextual factors such as the healthcare organization, incentives, policies, or workflow affect the conduct of a health intervention and RE-AIM outcomes at critical milestones of the study. Especially relevant is the PRISM component of “implementation and sustainability infrastructure” to support program or policy implementation and maintenance. The developers of PRISM intend the framework to provide “a practical, actionable model that could be used by both practitioners and researchers to plan and guide interventions, implementation strategies, adaptations, and factors related to sustainability” (McCreight et al. 2019).
The approach called Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) uses ethnographic methods to collect and analyze qualitative data about a clinical intervention in a relatively short period (Palinkas and Zatzick 2019). Designed especially to evaluate hybrid effectiveness-implementation trials, RAPICE comprises a series of steps involving participant observations, field notes, key informant interviews, and thematic content analysis leading to a more comprehensive interpretation of the intervention’s implementation processes and outcomes.
The Living Textbook identifies other considerations related to potential changes to the intervention:
- The section on designing the analysis plan discusses unanticipated changes, such as changes to the study population, organizational structure, or clinical practice and standards that may have an impact on the analysis of your study.
- The section on using electronic health record (EHR) data in clinical research discusses how EHR data can support the process of monitoring the fidelity of the intervention.
- If the ePCT intervention proves effective, the section on dissemination provides information on frameworks for dissemination and implementation.
Glasgow RE, Vogt TM, Boles SM. 1999. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 89:1322-1327. doi:10.2105/ajph.89.9.1322. PMID: 10474547.
Glasgow RE, Harden SM, Gaglio B, et al. 2019. RE-AIM Planning and Evaluation Framework: adapting to new science and practice with a 20-year review. Front Public Health. 7:64-64. doi:10.3389/fpubh.2019.00064. PMID: 30984733.
McCreight MS, Rabin BA, Glasgow RE, et al. 2019. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. doi:10.1093/tbm/ibz085.
Palinkas LA, Zatzick D. 2019. Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) in pragmatic clinical trials of mental health services implementation: methods and applied case study. Adm Policy Ment Health. 46:255-270. doi:10.1007/s10488-018-0909-3. PMID: 30488143.
Stirman SW, Miller CJ, Toder K, Calloway A. 2013. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 8:65. doi:10.1186/1748-5908-8-65. PMID: 23758995.