Pilot testing involves an assessment of the readiness of the embedded intervention before launching its full implementation. Such assessment will involve evaluating the context, capabilities, and challenges of the partner healthcare system as well as testing key elements of the intervention and collection and transfer of data from the EHR.

Readiness Assessment for Pragmatic Trials (RAPT)

RAPT is the first model to help interventionists and funders assess the extent to which interventions are ready for PCTs. Scoring efficacious interventions using RAPT can inform research team discussions regarding whether or not to advance an intervention to effectiveness testing using a PCT and how to design that PCT. (Baier et al. 2019)

RAPT is a recently developed framework for study teams in the pilot phase to assess readiness of their embedded intervention before advancing to the full implementation phase (Baier et al. 2019). RAPT delineates 9 readiness criteria to evaluate the intervention, from low to high readiness:

For details on how to use the RAPT tool, visit the RAPT Model website.

Challenges During the Pilot Phase

Feasibility assessment specific to embedded PCTs may be associated with the following challenges.

Biostatistical Issues

Study teams should work early on with their statistician to anticipate gaps or issues around cluster randomization, including sample size and potential for contamination, intraclass correlation, varying cluster size, the need for stratification or matching, and potential for missing follow-up data.

The Collaboratory’s Biostatistics and Study Design Core provides resources to help address challenges related to:

• Statistical lessons learned
• Constrained randomization
• Small-sample robust variance correction
• Frailty models
• Unequal cluster sizes
• Pair-matching versus stratification
• Intraclass correlation coefficient
• Extracting usable data from EHRs
• Analyses in the presence of noncompliance and study dropout
• Mixed effects models

Secondary Use of EHR Data

Using EHR data for research is fundamentally different from using prospectively collected data. Several aspects of EHR data drive these differences, including the lack of control over data definitions and data collection processes in healthcare facilities, procedures for access to the data, frequent dependence on record linkage, the need for computable definitions for cohorts and outcomes of interest, and the intricacies of demonstrating that data are of adequate quality to support research conclusions.

Consider how the intervention will use existing EHR data for cohort identification, recruitment, sample size estimates, population screening, collection of embedded patient-reported outcome (PRO) data, and so on. If there will be different EHR systems, how will they be linked? Pilot test the data collection procedures and any web-based tools developed specifically for the study.

The Collaboratory’s Electronic Health Records Core offers PCT-specific guidance for study teams on:

• Acquiring and using EHR data
• EHR-based phenotyping
• NIH Collaboratory Distributed Research Network

Capabilities and Readiness of the Partner Healthcare System

Consider whether there is a gap in existing services for the target population, or whether the healthcare system has had difficulties successfully addressing patient needs. It is important to evaluate how the embedded intervention will align with the goals of the healthcare system to advance their practice. Also consider the best timing for embedding the PCT intervention if there are expected EHR platform changes or pending external contextual or policy changes that could impact the system’s capacity to implement the PCT. Allow sufficient time for build up and/or development of new EHR tools and forms.

Consider testing the intervention across partner systems that vary in capacity or context; for example, a VA health system and a regional or community health system. Evaluate how effective the system or clinic will be as a research partner. One way is for the study team to pilot the programming capacity of a site (e.g., giving the site sample programming code, or asking sites to send the code that they will use to the study coordinating center) to determine if the site is able to fully participate. Researchers could devise a set of criteria for site inclusion. For example, in a trial in which epidemiological benchmarks are inserted into radiology imaging reports, the study team wanted to verify at each site that the intervention text could be successfully included in the report based on a specific CPT code, modality, patient age, and date. Another way to gauge the effectiveness of a partnership is to confirm the healthcare system’s commitment to identifying the clinical staff available to carry out the study intervention.

Integrating the Study Into the Clinical Workflow

Test how smoothly the embedded intervention will be incorporated into the site’s existing system infrastructure. Sites have different workflows and available resources, which can complicate the process. In pragmatic research, it is important that study procedures mimic routine practice. Consider how individual healthcare providers will be engaged in the intervention and how burdensome it may be for them. Make an effort to simplify and streamline the study to make integration as user-friendly as possible. Be open to adapting the study based on interest holder input or changes in the care delivery system. Study teams are also encouraged to pilot the effectiveness of trial-related materials such as informed consent forms and processes, training videos, brochures and placards, and toolkits.

• What are 2-5 aspects of your trial that are essential to pilot?