Analysis Plan
Section 7
Unanticipated Changes
Conditions change during the course of any clinical trial. Most trials, especially those that span multiple years, must deal with the evolution of clinical practice and other changes in medical care. Pragmatic clinical trials face additional challenges because of their less structured approach and because they often are embedded within the workflows of large healthcare systems.
In this section, we discuss unanticipated changes during study implementation that investigators should consider in advance so that the impact of such changes can be minimized during the planning phase. The section addresses the following topics:
- Changes in the potential study population, including demographic characteristics and insurance coverage patterns, fundamental data issues, and patient/participant decisions
- Changes in participating healthcare systems, including healthcare system leadership, personnel turnover and training issues, site withdrawal due to burden, and structural changes such as reorganization
- Changes in clinical practice, regulations, and standards, including contamination of the control group with elements of the intervention, and new healthcare initiatives that focus on the same problem as the intervention
- Trial- or site-imposed data differences
- Planning for and responding to unanticipated changes
Changes in the Potential Study Population
One potential challenge for any clinical trial is an unanticipated change in the putative study population. Such a change may arise for a variety of reasons, including shifts in insurance coverage, changes in demographic characteristics, and extrinsic factors (such as media reports) that affect the willingness of patients, healthcare providers, or other participants to enroll in the trial. When these changes occur, they can have differential effects on the study arms.
Demographic Characteristics and Insurance Coverage Patterns
The population that is potentially available for study participation may change as a result of underlying demographic shifts. Changes in insurance plans and Medicare or Medicaid coverage may also strongly affect the populations seen at the level of healthcare systems, hospitals, or individual clinics and practices.
Case Example: STOP CRC Trial
The Strategies and Opportunities to Stop Colorectal Cancer (STOP CRC) trial, an NIH Pragmatic Trials Collaboratory Trial, was a cluster randomized trial designed to evaluate strategies for improving rates of colorectal cancer screening. The study, which was conducted in a network of federally qualified health centers in Oregon and Northern California, encountered unanticipated changes in the potential study population when implementation of the Affordable Care Act led to Medicaid expansion and the inclusion of colorectal cancer screening as a quality metric in Oregon’s new pay-for-performance Medicaid program. The changes increased the size of Oregon’s Medicaid-enrolled population that was age-eligible for colorectal cancer screening by 55% (Coronado et al 2015).
Fundamental Data Issues
Evolution of technology and advances in diagnostic codes can modify the eligible population and hence change the definition of the target population. For example, when troponin began to be used for diagnosis of myocardial infarction, some sites were ahead in its use while others lagged. Unless a study specified exact diagnosis and onset criteria in selecting the study population, it could be subject to differing eligibility across sites. Likewise, a study with myocardial infarction as the outcome would be subject to the same problem.
Patient/Participant Decisions
Extrinsic factors affecting decisions by patients or providers about whether to participate or continue in a trial may influence the available study population in ways that are difficult to anticipate. For example, preexisting conceptions about the study, media coverage of a particular therapy, and even coverage of clinical research generally (whether positive or negative) may affect willingness to participate in a trial. In addition, positive or negative media coverage of reputational issues may affect willingness to seek treatment at a particular clinic or healthcare system.
Changes in Participating Healthcare Systems
Healthcare System Leadership
Leadership changes in a hospital or healthcare system may occur rapidly and frequently in an era of frequent consolidation and reorganization. Such changes may also affect the level of support available for a study, whether in the start-up, conduct, or follow-up phase, and in some cases can effectively stop an ongoing trial at a given site. Written agreements prior to study start should be a matter of course, specifying the level of participation and guaranteeing that the protocol will be followed for the duration of the study.
Personnel Turnover and Training Issues
Because clinical trials, whether traditional or pragmatic, depend on relatively uniform implementation of a research protocol across all sites, the experience, knowledge, training, and commitment of investigators and site personnel have major implications for the quality of the trial’s implementation and the validity of the data collected. Frequent turnover among investigators and/or support staff can disrupt trial operations, especially if there is a lack of familiarity with the trial or if staff training has been inadequate.
For example, a trial might have the goal of increasing patient satisfaction by implementing a training program, supported by the hospital, for nurses and residents. Changes in leadership of either the hospital or the healthcare system might realign priorities, and the program could be halted or modified in some or all of the sites. Although this issue could affect both intervention and control sites, it could have a differential impact on the study groups. Statistical adjustment for such changes may not be possible. When planning the study, the study team should ensure that a clear memorandum of understanding documents the specific conduct and obligations of all partners.
Case Example: PROVEN Trial
The Pragmatic Trial of Video Education in Nursing Homes (PROVEN) trial, an NIH Collaboratory Demonstration Project, is a cluster randomized study designed to test a video-assisted decision support intervention in advance care planning for patients in nursing homes. Investigators designed a “video status report” that the healthcare system staff integrated into the EHR to document staff offering the intervention to patients. Intervention monitoring reports from these records revealed considerable variation in video offer rates across facilities, indicating that some site staff needed additional training to ensure that patients were offered the intervention appropriately.
Site Withdrawal Due to Burden
If sites, or the healthcare system that is responsible for the overall agreements, is not fully cognizant of the requirements of trial conduct, some of the conditions of participation could turn out to be onerous. A proactive memorandum of understanding that identifies site responsibilities, defines the eligible population, and clearly specifies activities to be performed by the site research team should prevent or at least lower the risk of site dropout. A pilot study or feasibility assessment can help work out the details.
Structural Changes, Such as Reorganization
As healthcare systems struggle to provide high-quality, cost-effective healthcare, changes in structure are inevitable. It is possible that one or more study sites will close or merge, or the healthcare delivery model will be revamped to address newer performance metrics. To quote Yogi Berra, “It’s tough to make predictions, especially about the future.” However, prior to implementing a study, consideration of the possibilities and discussions regarding the likelihood of changes that might affect the trial should occur.
Changes in Clinical Practice, Regulations, and Standards
Changes in practice, standards, and regulations may create challenges for ongoing studies or render a study in planning or development impracticable. Many newer pragmatic trial designs depend on facilitated access to electronic patient data for cohort identification through federated systems and distributed research networks. Changes in regulations and practices affecting access to these data could have significant effects on pragmatic trials.
For example, an NIH Collaboratory Trial that had the goal of reducing the use of pain medication was initiated before the growing awareness of the opioid epidemic. Greater recognition of the problem drove local efforts to address it. Again, anticipation and monitoring of a trend among control sites is important. If the intervention has not been adopted on a widespread basis, it may be possible to investigate differentially evolving trends in outcomes among controls that are adopters.
Case Example: PPACT
The Collaborative Care for Chronic Pain in Primary Care Trial (PPACT), an NIH Collaboratory Trial, was a cluster randomized trial comparing multimodal approaches to pain management to reduce reliance on opioids. The study was initiated before the current recognition of an epidemic of opioid misuse and abuse that has been widely covered in the news media. As a result, healthcare systems have implemented some of the same strategies as PPACT during the trial, and the usual care arm became more like the intervention arm. To the extent that these changes were beneficial, the study might not have been able to test the overall effect of the intervention.
Contamination of the Control Group With Elements of the Intervention
A trial that has the goal of increasing mental health screening may encounter unanticipated contamination of the control group with elements of the intervention. For example, this challenge may arise in a trial that has the goal of increasing mental health screening as more and more primary care providers spontaneously decide to use the Patient Health Questionnaire-9 (PHQ-9) during routine clinic visits. If the intervention is a program that promotes use of the PHQ-9 among intervention clinics and the outcome is the proportion of patients who were screened, this spontaneous use could dilute the effect of the intervention. In planning for such a trial, baseline rates over time, rather than at a single time point, should be calculated and the trial should include monitoring for a time effect. In any case, it is important to recognize and monitor intervention-like activities using the same instruments and procedures and to incorporate these considerations in the design and analysis.
New Healthcare Initiatives That Focus on the Same Problem as the Intervention
This issue is similar to the previous challenge, though it is possible in this case for all control sites to come under the umbrella of the increased efforts. In both cases, programs such as these vary in their approach and intensity. It may be possible to disentangle common elements from the unique aspects of the ongoing trial and to adjust the analysis accordingly, though sample size may become an issue. An attempt might be made in the design stage to craft an intervention that would be unique and unlikely to be implemented outside the trial. The more complicated and intense the intervention, the less likely it is to be duplicated. However, it also becomes less likely to be generalized and sustainable.
Trial- or Site-Imposed Data Differences
Several unforeseen site-specific situations may arise during the conduct of the trial. For example, it might become evident when comparing accrual rates among sites that a different ordering in check boxes for comorbid conditions had been created according to specialty, enabling the most likely to appear first on the checklist. In a pragmatic trial, complete standardization could challenge workflow and efficiency. In this case, stratification by specialty could have been considered.
A situation that likely would have a more pronounced impact would be disappointing accrual rates. In some cases, changes to operational aspects of the protocol are made for the purpose of enhancing enrollment. Such changes implicitly change the definition of the intervention. Again, careful planning, monitoring, and recording of situations such as these are necessary. The statistical analysis cannot be assumed to overcome evolving differences.
Some study modifications can have major effects on both the conduct and results of a trial. When accrual rates are disappointing, studies might alter the design by expanding the eligibility criteria, thereby redefining the target population. Doing so is likely to dilute the intervention effect and have a substantial impact on statistical power and sample size. The statistical analysis must accommodate recognition of these changes and account for them.
The opposite problem occurs when the overall event rate is lower than anticipated. In this case, studies sometimes enrich the population during the trial with patients at higher risk to obtain a higher baseline risk. Again, this kind of change creates a different target population and complicates the interpretation of the results. The statistical analysis may be able to account for these changes, but the trade-off between event rate and accrual rate is always complicated.
Planning for and Responding to Unanticipated Changes
Planning for circumstances that may affect the study arms should begin at the time of initial study design. Although the specific causes of disruption may not be predictable, contingency planning should account for impacts such as those described in this section. Planning for challenges may include efforts to incorporate robustness and flexibility into the study design.
To ensure that unanticipated changes to study populations do not go undetected during the course of a trial, investigators and staff should include a plan for continuous monitoring of study implementation and progress. Depending on the nature of the study, measures chosen for monitoring may be quantitative, qualitative, or both.
SECTIONS
sections
- Introduction
- Intraclass Correlation
- Unequal Cluster Sizes
- Accounting for Residual Confounding in the Analysis
- Missing Data and Intention-to-Treat Analyses
- Electronic Health Record Data Extraction
- Unanticipated Changes
- Interim Reassessment of Sample Size in Cluster Randomized Trials
- Case Study: STOP CRC Trial
Resources
Building Partnerships to Ensure a Successful Trial
Living Textbook chapter describing stakeholder partnerships throughout the ePCT life cycle. Healthcare system leaders can provide valuable advice regarding how to handle unexpected changes during the conduct of ePCTs.
Statistical Analysis Plan Checklist for Addressing COVID-19 Impacts
Tool from the NIH Pragmatic Trials Collaboratory to assist investigators in identifying impacts of the COVID-19 public health emergency on ongoing pragmatic clinical trials
REFERENCES
Coronado GD, Petrik AF, Bartelmann SE, Coyner LA, Coury J. 2015. Health policy to promote colorectal cancer screening: improving access and aligning federal and state incentives. Clin Res (Alex). 29:50-55. doi:10.14524/CR-14-0044. PMID: 27135047.
current section : Unanticipated Changes
- Introduction
- Intraclass Correlation
- Unequal Cluster Sizes
- Accounting for Residual Confounding in the Analysis
- Missing Data and Intention-to-Treat Analyses
- Electronic Health Record Data Extraction
- Unanticipated Changes
- Interim Reassessment of Sample Size in Cluster Randomized Trials
- Case Study: STOP CRC Trial