Traditional efficacy trials aim to ensure consistent delivery of and high adherence to study interventions. Study staff delivering interventions are expected to strictly follow study protocols and, whenever possible, remain masked to study treatment assignments. Clinicians’ experiences with delivering study treatments or conducting study assessments are considered potential sources of bias. In this model, research teams discover or create knowledge at a safe distance from frontline staff and then disseminate that knowledge once conclusions are settled.

See the Concealment and Masking section of the Living Textbook.

In contrast, pragmatic trials aim to understand the effectiveness of (often complex) interventions in real-world clinical or public health settings. Variable fidelity to protocol, incomplete adherence, and barriers to implementation are all signal rather than noise or sources of bias. Understanding the experience frontline clinical staff and health system leaders have with study interventions is essential to interpreting trial results.

Consequently, health system partners should not be insulated from analysis and interpretation of pragmatic trial findings. Instead, study teams should continue engaging with all levels (from frontline staff to executive leaders) of partner health systems throughout data analysis, sharing results as soon as they are available, exploring potential explanations and implications for practice or policy. In this collaborative process, study teams contribute greater expertise regarding study design, standardized outcome assessment, and analytic methods, while health system partners contribute greater expertise regarding barriers to implementation, variation in fidelity or adherence, and potential problems with quality or integrity of routinely collected clinical data.

Presentations of preliminary results to health system partners may differ from those for research audiences. For example, clinicians and clinical leaders will generally value metrics more directly related to clinical practice (such as the proportion of patients with significant clinical improvement) or directly relevant to business success (such as clinic-level performance on report card quality metrics) than they will value metrics more familiar to research audiences (such as regression model coefficients or estimates of effect size).

The experience of health system partners can be especially valuable for understanding or explaining findings that do not follow investigators’ hypotheses or expectations. Those include null findings, discovery of unexpected risks or adverse events, or high levels of variability in uptake or implementation of study interventions. Those scenarios may prompt additional secondary or subgroup analyses, not for the purpose of concealing or overruling analyses declared a priori but for the purpose of understanding why original hypotheses were not supported.

Reporting of trial results through clinical trial registries and journal publications should, of course, adhere to analytic plans declared and registered a priori, including specification of primary and secondary trial outcomes, primary analytic methods, thresholds for statistical significance, and assumptions used for power and sample size calculations. But, presentations for health system partners or leaders of similar health systems may consider a wider range of outcomes and include secondary or exploratory analyses informed by experience of health system partners. Primary scientific publications should address generalizable conclusions regarding effectiveness and potential risks. Presentations to health system leaders should instead address, “What does this mean for my health system and the patients we serve?”

Example of Collaborative Analysis and Interpretation: Suicide Prevention Outreach Trial (SPOT)

SPOT, an NIH Collaboratory Trial, was a 3-arm trial that evaluated whether outreach interventions could reduce risk of self-harm or suicide attempts among patients who reported frequent suicidal ideation on routinely administered PHQ-9 depression questionnaires (Simon et al 2022). Following a modified Zelen design, people reporting frequent suicidal thoughts were randomly assigned to continue in usual care or to be offered one of two low-intensity outreach interventions. Contrary to expectation, patients offered an interactive online program teaching dialectical behavior therapy skills experienced a higher risk of self-harm over the following 18 months. Trial investigators shared this finding with clinical partners in all 4 participating health systems as soon as analyses were completed. Clinical partners suggested additional analyses to examine 2 specific questions: Could offer of an online skills training program have led patients to reduce use of or disengage from traditional mental health care? How was the signal of increased risk related to level of engagement with the online skills training program? These secondary analyses informed interpretation of results and were reported in an appendix to the primary trial publication.