Pilot Testing

Assessing Feasibility

Section 5

Pilot Testing


Lynn L. DeBar, PhD, MPH

Jeffrey G. Jarvik, MD, MPH

Leah Tuzzio, MPH

Miguel A. Vazquez, MD


Contributing Editor

Liz Wing, MA

This phase involves evaluating the context, capabilities, and challenges of the partner healthcare system and testing key elements of the intervention and collection and transfer of data from the EHR. Feasibility assessment specific to embedded PCTs may be associated with the following challenges.

Biostatistical Issues

Study teams should work early on with their statistician to anticipate gaps or issues around cluster randomization, including sample size and potential for contamination, intraclass correlation, varying cluster size, the need for stratification or matching, and potential for missing follow-up data.

The Collaboratory’s Biostatistics and Study Design Core provides resources to help address challenges related to:

Secondary Use of EHR Data

Using EHR data for research is fundamentally different from using prospectively collected data. Several aspects of EHR data drive these differences, including the lack of control over data definitions and data collection processes in healthcare facilities, procedures for access to the data, frequent dependence on record linkage, the need for computable definitions for cohorts and outcomes of interest, and the intricacies of demonstrating that data are of adequate quality to support research conclusions.

Consider how the intervention will use existing EHR data for cohort identification, recruitment, sample size estimates, population screening, collection of embedded patient-reported outcome (PRO) data, and so on. If there will be different EHR systems, how will they be linked? Pilot test the data collection procedures and any web-based tools developed specifically for the study.

The Collaboratory’s Phenotypes, Data Standards, and Data Quality Core and Electronic Health Records Core offer PCT-specific guidance for study teams on:

Capabilities and Readiness of the Partner Healthcare System

Consider whether there is a gap in existing services for the target population, or whether the system has had difficulties successfully addressing patient needs. It is important to evaluate how the embedded intervention will align with the goals of the healthcare system to achieve their practice metrics. Also consider the best timing for embedding the PCT intervention if there are expected EHR platform changes or pending external contextual or policy changes that could impact the system’s capacity to implement the PCT.

Evaluate how effective the system or clinic will be as a research partner. One way is for the study team to pilot the programming capacity of a site (giving it sample programming code) to determine if the site is able to fully participate. Researchers could devise a set of criteria for site inclusion. For example, in a trial in which epidemiological benchmarks are inserted into radiology imaging reports, the study team wanted to verify at each site that the intervention text could be successfully included in the report based on a specific CPT code, modality, patient age, and date. Another way to gauge the effectiveness of a partnership is to confirm the healthcare system’s commitment to identifying the clinical staff available to carry out the study intervention.

Integrating the Study Into the Clinical Workflow

Test how smoothly the embedded intervention will be incorporated into the site’s existing system infrastructure. Sites have different workflows and available resources, which can complicate the process. In pragmatic research, it is important that study procedures mimic routine practice. Consider how individual healthcare providers will be engaged in the intervention and how burdensome it may be for them. Make an effort to simplify and streamline the study to make integration as user-friendly as possible. Be open to adapting the study based on stakeholder input or changes in the care delivery system. Study teams are also encouraged to pilot the effectiveness of trial-related materials such as informed consent forms and processes, training videos, brochures and placards, and toolkits.




DeBar LL, Jarvik JG, Tuzzio L, Vazquez MA. Assessing Feasibility: Pilot Testing. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. Available at: http://rethinkingclinicaltrials.org/assessing-feasibility/pilot-testing/. Updated November 7, 2017.