Unanticipated Changes

Analysis Plan

Section 7

Unanticipated Changes


Elizabeth R. DeLong, PhD

For the NIH Health Care Systems Collaboratory Biostatistics and Study Design Core


Contributing Editors

Jonathan McCall, MS

Damon M. Seils, MA

Changes to Study Populations

One potential challenge for any clinical trial is an unanticipated change to the putative study population. These changes can arise for a variety of reasons, including shifts in health plan coverage, changes in demographic characteristics, changes in clinic or health system leadership or organization, and extrinsic factors (such as media reports) that affect the willingness of healthcare providers or patients to enroll in a clinical trial. When these changes occur, they can have differential effects on study arms. Below, we examine some of the factors that can affect a study population and address the importance of planning for, monitoring, and evaluating such changes.

Changes in Demographic Characteristics or Coverage Patterns

In some cases, the populations potentially available for study participation may change due to underlying demographic shifts. Changes to insurance plans and Medicare or Medicaid coverage may also strongly affect populations seen at the level of health systems, hospitals, or individual clinics and practices.

Case Example: STOP CRC Trial

The Strategies and Opportunities to Stop Colorectal Cancer (STOP CRC) trial, an NIH Collaboratory Demonstration Project, is a CRT designed to evaluate strategies for improving rates of colorectal cancer screening. The study, which is being conducted in a network of federally qualified health centers in Oregon and California, encountered unanticipated changes to potential study populations when the recent Medicaid expansion created volatility as coverage expanded.

Case Example: PPACT

The Collaborative Care for Chronic Pain in Primary Care Trial (PPACT), an NIH Collaboratory Demonstration Project, is a CRT comparing multimodal approaches to pain management to reduce reliance on opioids. The study was initiated before the current recognition of an epidemic of opioid misuse and abuse that has been widely covered in the news media. As a result, health systems have implemented some of the same strategies as PPACT, and the usual care arm has become more like the intervention arm. To the extent that these changes are beneficial, the study will not be able to test the overall effect of the intervention.

Patient/Participant Decisions

Extrinsic factors affecting decisions by patients or providers about whether to participate in a trial or to continue in it may also perturb the available study population in ways that are difficult to anticipate. For example, pre-existing conceptions about the study, media coverage of a particular therapy, and even coverage of clinical research more generally (whether positive or negative) may affect willingness to participate in a trial. In addition, positive or negative media coverage of reputational issues may affect willingness to seek treatment at a particular clinical or health system.

Leadership Decisions

Leadership changes in a hospital or health system may occur rapidly and frequently in an era of frequent consolidation and reorganization. Such changes may also affect the level of support available for a study, whether in the start-up, conduct, or follow-up phase, and in some cases can effectively stop an ongoing trial at a given site. Written agreements prior to study start should be a matter of course, specifying the level of participation and guaranteeing that the protocol will be followed for the duration of the study.

Regulations and Standards

Changes in regulations governing medical practice or research conduct may create challenges for ongoing studies or render a study in planning or development impracticable. Many newer PCT designs depend on facilitated access to electronic patient data for cohort identification through federated systems and distributed research networks. Changes in the regulations and practices affecting access to these data could have significant effects on PCTs.

Personnel Turnover and Training Issues

Because clinical trials, whether traditional or pragmatic, depend on a relatively uniform implementation of a research protocol across all sites, the experience, knowledge, training, and commitment of investigators and site personnel have major implications for the quality of the trial’s implementation and the validity of the data collected. Frequent turnover among investigators and/or support staff can cause significant disruptions to trial operations, especially if there is a lack of familiarity with the trial or if staff training has not been adequate.

Case Example: PROVEN Trial

The NIH Collaboratory’s Pragmatic Trial of Video Education in Nursing Homes (PROVEN) trial is a cluster-randomized study designed to test a video-assisted decision support intervention in advanced care planning for nursing home patients. Investigators designed and health care system staff integrated into system EHRs a “Video Status Report” to document staff offering the intervention to patients. Intervention monitoring reports from these records revealed considerable variation in video offer rates across facilities, indicating that some site staff needed additional training to ensure that patients were being offered the intervention appropriately.

Planning for and Responding to Unanticipated Changes

Planning for circumstances that could potentially affect study arms should begin at the time of initial study design. Although the specific causes of disruption may not be predictable, contingency planning should account for impacts such as the ones described above. Planning for challenges may include efforts to incorporate robustness and flexibility into the study design.


To ensure that unanticipated changes to study populations do not go undetected during the course of a trial, investigators and staff should create and put in place a plan for continuous monitoring of study implementation and progress. Depending on the nature of the study, measures chosen for monitoring may be quantitative, qualitative, or both.



Version History

January 16, 2019: Made nonsubstantive changes to the text and formatting as part of the annual content update (changes made by D. Seils).

Published August 25, 2017


DeLong ER. Analysis Plan: Unanticipated Changes. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. Available at: https://rethinkingclinicaltrials.org/chapters/design/analysis-plan-top/unanticipated-changes-to-study-populations/. Updated January 16, 2019. DOI: 10.28929/020.