Assessing Feasibility
Section 5
Pilot Testing
Pilot testing involves an assessment of the readiness of the embedded intervention before launching its full implementation. Such assessment will involve evaluating the context, capabilities, and challenges of the partner healthcare system as well as testing key elements of the intervention and collection and transfer of data from the EHR.
Readiness Assessment for Pragmatic Trials (RAPT)
RAPT is the first model to help interventionists and funders assess the extent to which interventions are ready for PCTs. Scoring efficacious interventions using RAPT can inform research team discussions regarding whether or not to advance an intervention to effectiveness testing using a PCT and how to design that PCT. (Baier et al. 2019)
RAPT is a recently developed framework for study teams in the pilot phase to assess readiness of their embedded intervention before advancing to the full implementation phase (Baier et al. 2019). RAPT delineates 9 readiness criteria to evaluate the intervention, from low to high readiness:
RAPT Domain | |
Implementation protocol | Is the protocol sufficiently detailed to be replicated? |
Evidence | To what extent does the evidence base support the intervention’s efficacy? |
Risk | Is it known how safe the intervention is? |
Feasibility | To what extent can the intervention be implemented under existing conditions? |
Measurement | To what extent can the intervention’s outcomes be captured? |
Cost | How likely is the intervention to be economically viable? |
Acceptability | How willing are providers likely to be to adopt the intervention? |
Alignment | To what extent does the intervention align with external stakeholders’ priorities? |
Impact | How useful will the intervention’s results be? |
For details on how to use the RAPT tool, visit the RAPT Model website.
Challenges During the Pilot Phase
Feasibility assessment specific to embedded PCTs may be associated with the following challenges.
Biostatistical Issues
Study teams should work early on with their statistician to anticipate gaps or issues around cluster randomization, including sample size and potential for contamination, intraclass correlation, varying cluster size, the need for stratification or matching, and potential for missing follow-up data.
The Collaboratory’s Biostatistics and Study Design Core provides resources to help address challenges related to:
- Statistical lessons learned
- Constrained randomization
- Small-sample robust variance correction
- Frailty models
- Unequal cluster sizes
- Pair-matching versus stratification
- Intraclass correlation coefficient
- Extracting usable data from EHRs
- Analyses in the presence of noncompliance and study dropout
- Mixed effects models
Secondary Use of EHR Data
Using EHR data for research is fundamentally different from using prospectively collected data. Several aspects of EHR data drive these differences, including the lack of control over data definitions and data collection processes in healthcare facilities, procedures for access to the data, frequent dependence on record linkage, the need for computable definitions for cohorts and outcomes of interest, and the intricacies of demonstrating that data are of adequate quality to support research conclusions.
Consider how the intervention will use existing EHR data for cohort identification, recruitment, sample size estimates, population screening, collection of embedded patient-reported outcome (PRO) data, and so on. If there will be different EHR systems, how will they be linked? Pilot test the data collection procedures and any web-based tools developed specifically for the study.
The Collaboratory’s Electronic Health Records Core offers PCT-specific guidance for study teams on:
Capabilities and Readiness of the Partner Healthcare System
Consider whether there is a gap in existing services for the target population, or whether the healthcare system has had difficulties successfully addressing patient needs. It is important to evaluate how the embedded intervention will align with the goals of the healthcare system to advance their practice. Also consider the best timing for embedding the PCT intervention if there are expected EHR platform changes or pending external contextual or policy changes that could impact the system’s capacity to implement the PCT. Allow sufficient time for build up and/or development of new EHR tools and forms.
Consider testing the intervention across partner systems that vary in capacity or context; for example, a VA health system and a regional or community health system. Evaluate how effective the system or clinic will be as a research partner. One way is for the study team to pilot the programming capacity of a site (e.g., giving the site sample programming code, or asking sites to send the code that they will use to the study coordinating center) to determine if the site is able to fully participate. Researchers could devise a set of criteria for site inclusion. For example, in a trial in which epidemiological benchmarks are inserted into radiology imaging reports, the study team wanted to verify at each site that the intervention text could be successfully included in the report based on a specific CPT code, modality, patient age, and date. Another way to gauge the effectiveness of a partnership is to confirm the healthcare system’s commitment to identifying the clinical staff available to carry out the study intervention.
Integrating the Study Into the Clinical Workflow
Test how smoothly the embedded intervention will be incorporated into the site’s existing system infrastructure. Sites have different workflows and available resources, which can complicate the process. In pragmatic research, it is important that study procedures mimic routine practice. Consider how individual healthcare providers will be engaged in the intervention and how burdensome it may be for them. Make an effort to simplify and streamline the study to make integration as user-friendly as possible. Be open to adapting the study based on stakeholder input or changes in the care delivery system. Study teams are also encouraged to pilot the effectiveness of trial-related materials such as informed consent forms and processes, training videos, brochures and placards, and toolkits.
- What are 2-5 aspects of your trial that are essential to pilot?
SECTIONS
sections
- Introduction
- Developing the Trial Documentation
- Establishing Close Partnerships With Participating Healthcare System Leaders and Staff
- Delineating the Roles of All Stakeholders to Determine Training Needs
- Pilot Testing
- Feasibility Assessment Scenarios From the NIH Collaboratory Trials
- Spotlight on NIH Collaboratory Trials
- Additional Resources
Resources
Readiness Assessment for Pragmatic Trials (RAPT) (Baier RR et al. 2019)
Diffusion of innovations in service organizations: systematic review and recommendations (Greenhalgh T et al. 2004)
A guide to research partnerships for pragmatic clinical trials (Johnson KE et al. 2014)
Trials without tribulations: minimizing the burden of pragmatic research on healthcare systems (Larson EB et al. 2015)
current section : Pilot Testing
- Introduction
- Developing the Trial Documentation
- Establishing Close Partnerships With Participating Healthcare System Leaders and Staff
- Delineating the Roles of All Stakeholders to Determine Training Needs
- Pilot Testing
- Feasibility Assessment Scenarios From the NIH Collaboratory Trials
- Spotlight on NIH Collaboratory Trials
- Additional Resources