January 31, 2020: Living Textbook Grand Rounds Series Part 1: Pragmatic Clinical Trials: How Do I Start? (Lesley H. Curtis, PhD, Greg Simon, MD, MPH)


Greg Simon, MD, MPH
Senior Investigator
Kaiser Permanente Washington Health Research Institute

Lesley H. Curtis, PhD
Chair and Professor
Department of Population Health Sciences
Duke University School of Medicine
Interim Executive Director, Duke Clinical Research Institute


Pragmatic Clinical Trials: How Do I Start?


Pragmatic clinical trials; PRECIS-2; Real-world evidence; Health systems research; Stakeholders; Clinical workflow; Study team

Key Points

  • Embedded pragmatic clinical trials (ePCTs) are large, efficient studies conducted in the real world that provide evidence for adoption of an intervention into clinical practice.
  • ePCTs are conducted in partnership with healthcare systems, use streamlined procedures and existing infrastructure, and answer important medical questions. However, high relevance to real-world decision-making can sometimes come at the expense of trial efficiency.
  • The PRECIS-2 scores are not absolute virtues; rather, the tool helps researchers determine if their trial is fit for purpose based on their study question.
  • For greater generalizability, ePCTs should be conducted in a diverse range of patients, and study results should be reported transparently.

Discussion Themes

How might we support health systems that serve more diverse populations to participate in a pragmatic clinical trial?

What concerns might be voiced by health system leaders regarding potential reputational risk of a PCT, and perhaps downstream issues about the results publication?

The question “Can everyone do this study?” is different from “Can everyone believe the research results?”

To see upcoming topics in the Living Textbook Grand Rounds series, download the flyer and share with your colleagues and institution. To learn more about the fundamentals of designing and launching a successful ePCT visit the Living Textbook.

#pctGR, @Collaboratory1

January 30, 2020: Meeting Materials from the NIH Collaboratory PRISM Kickoff Meeting

The Collaboratory has made available all the presentations from their recent PRISM Kickoff meeting held in Bethesda November 19-20, 2019.

PRISM Demonstration Project Investigators
Left to right: Kathleen Sluka, PT, PhD (FM TIPS); Andrea Cheville, MD (NOHARM); Karen Sherman, PhD, MPH (AcuOA); Jon Tilburt, MD (NOHARM); Lynn DeBar, PhD, MPH (AcuOA); Leslie Crofford, MD (FM TIPS); and Natalia Morone, MD (OPTIMUM).

The PRISM (Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing) program is a component of the NIH HEAL Initiative to address the opioid crisis. Highlights of Day 1 included welcoming the 4 new PRISM UG3 Demonstration Projects—AcuOA, FM TIPS, NOHARM, and OPTIMUM—and introducing the study teams to the NIH Collaboratory Program and Coordinating Center, hearing an overview of the HEAL Initiative goals and cooperative agreement, and learning about the aims of the new studies. Day 2 included face-to-face meetings between each PRISM Demonstration Project and the Collaboratory Core working groups to discuss anticipated challenges in design, implementation, analysis, and dissemination.

View or download the meeting materials on the website.



January 8, 2020: Registration Opens for 13th Annual Conference on Statistical Issues in Clinical Trials

Registration opened on January 1 for the 13th Annual University of Pennsylvania Conference on Statistical Issues in Clinical Trials. The theme of this year’s conference is “Cluster Randomized Clinical Trials: Challenges and Opportunities.”

The conference will be held on April 29 at the Smilow Center for Translational Research on the campus of the University of Pennsylvania Perelman School of Medicine in Philadelphia. Cosponsors include the American Statistical Association, the Society for Clinical Trials, and the National Institute of Statistical Sciences.

During the methods portion of the program, NIH Collaboratory investigator David Murray will present “Overview: Innovations in the Design and Analysis of Group- or Cluster-Randomized Trials.” The program also includes presentations on the uses of network- and individual-level information in design and analysis, the complexity introduced by noncompliance, current issues in stepped-wedge designs, and various applications of statistical techniques in cluster randomized studies.

Registration is required for this daylong event.

November 8, 2019: Lumbar Imaging with Reporting of Epidemiology: Initial Results and Some Lessons Learned (Jeffrey Jarvik, MD, MPH, Patrick Heagerty, PhD)


Jeffrey (Jerry) G. Jarvik MD MPH
Professor, Radiology, Neurological Surgery and Health Services
Adjunct Professor, Pharmacy and Orthopedics & Sports Medicine
University of Washington

Patrick Heagerty, PhD
Professor and Chair
Department of Biostatistics
University of Washington


Lumbar Imaging with Reporting of Epidemiology: Initial Results and Some Lessons Learned


Embedded pragmatic clinical trials; Radiology imaging; LIRE; Stepped-wedge; Cluster randomization; Epidemiology; Back pain

Key Points

  • The LIRE Demonstration Project evaluated whether prevalence benchmark data inserted into lumbar spine imaging reports would reduce overall spine-related healthcare utilization for patients referred from primary care.
  • The inserted intervention text urges caution when interpreting the presence of certain findings that are common in normal, pain-free volunteers.
  • While the study team found no decrease in spine-related healthcare utilization for the overall cohort, there was a small but potentially important effect on reducing opioid prescriptions.

Discussion Themes

A characteristic of stepped-wedge study design is that it yields two comparisons: between-group comparisons (clinic A vs clinic B) and within-group comparisons. But temporal trends can have an impact and must be adjusted for in the analysis.

For what type of intervention would a stepped-wedge design be suitable?

The hope is for a wider dissemination about interventions where radiologic testing is done and incidental findings are common.

Read more about the LIRE Demonstration Project.

#pctGR, #PragmaticTrials, @Collaboratory1

November 1, 2019: NIH Collaboratory: Looking Back, Looking Forward (Adrian Hernandez, MD, MHS, Lesley Curtis, PhD, Kevin Weinfurt, PhD)


Adrian F. Hernandez, MD, MHS
Professor of Medicine
Vice Dean for Clinical Research
Duke University School of Medicine

Lesley H. Curtis, PhD
Chair and Professor
Department of Population Health Sciences
Duke University School of Medicine
Interim Executive Director, Duke Clinical Research Institute

Kevin Weinfurt, PhD
Professor and Vice Chair of Research
Department of Population Health Sciences
Duke University School of Medicine


NIH Collaboratory: Looking Back, Looking Forward


Embedded pragmatic clinical trials; ePCTs; NIH Collaboratory; Health care systems research; Demonstration Projects; Living Textbook; HEAL Initiative; Coordinating Center; Research dissemination; Learning health systems; Real-world evidence

Key Points

Discussion Themes

How can we harmonize the different ideas about what it is to be “pragmatic” for NIH study sections, IRBs, and DSMB reviews? For example, if your DSMB isn’t knowledgeable about PCTs, you could end up with a very explanatory trial.

A willingness to share imperfections is an important part of learning and helps the clinical trial ecosystem evolve.

An important future topic would be how the NIH Collaboratory and PCORnet fit together.

Read more about the NIH Collaboratory Program and the Living Textbook of Pragmatic Clinical Trials.

#pctGR, #PragmaticTrials, @Collaboratory1, @texhern, @lmhcurtis, @KevinWeinfurt

October 30, 2019: Baseline Covariate Imbalance Influences Treatment Effect Bias in Cluster Randomized Trials

In a study supported by the NIH Collaboratory, researchers found that imbalance in individual-level baseline covariates influences bias in the observed treatment effect in cluster randomized trials. Using race as an example, the study highlights the importance of reducing covariate imbalance in the design stage of cluster randomized trials and of using statistical analysis techniques to minimize the resulting bias.

The innovative study, published in Contemporary Clinical Trials, used computer simulation models validated by real-data simulations from a large clinical trial to examine the influence of baseline covariate imbalance on treatment effect bias. They found that bias was proportional to the degree of baseline covariate imbalance and the covariate effect size. In the simulations, trials with larger numbers of clusters had less covariate imbalance. Statistical models that adjusted for important baseline confounders were more effective than unadjusted models in minimizing bias.

The authors recommend several design approaches and statistical analysis techniques for both reducing covariate imbalance and minimizing bias. Using the results of available prior data can help researchers identify important baseline confounders when designing cluster randomized trials.

This work was supported within the NIH Collaboratory by the NIH Common Fund through a cooperative agreement from the Office of Strategic Coordination within the Office of the NIH Director, and by a research supplement from the NIH Common Fund to promote diversity in health-related research.

July 15, 2019: PRIM-ER Gains Approval to Proceed to Implementation Phase: An Interview With Dr. Corita Grudzen

Primary Palliative Care for Emergency Medicine (PRIM-ER)Dr Corita Grudzen, an NIH Collaboratory Demonstration Project, received approval this month to enter its implementation phase. PRIM-ER is a pragmatic, cluster randomized trial of a multidisciplinary primary palliative care education and decision support intervention in a diverse sample of emergency departments that differ in their specialty geriatric and palliative care capacity, geographic region, payer mix, and patient demographic characteristics.

We spoke with the principal investigator of PRIM-ER, Dr. Corita Grudzen, at the NIH Collaboratory Steering Committee meeting in May about the completion of the study’s planning phase.

What is the status of your study?

PRIM-ER is currently transitioning from a 1-year UG3 planning phase to the UH3 implementation phase, where the intervention will be tested at full scale across 33 sites in a stepped-wedge trial. Our program manager is conducting training sessions in preparation for implementing the intervention at study sites.

Were there surprises during the planning phase of the study?

There weren’t surprises so much as confirmation of what we already knew, which was that enthusiasm can go a long way. Having a detail-oriented, responsive site principal investigator is also key. When you have a well-organized team on the ground, the distance doesn’t matter as much, and all the things you think are going to get in the way don’t get in the way. Things go smoothly when you have someone who is incredibly enthusiastic, loves the content, understands the content, and is infectious about sharing the training and information with others. I think that can overcome a lot of potential barriers.

What is an example of a challenge that you were able to overcome with the help of the NIH Collaboratory Core Working Groups?

Working with the Cores was reassuring, in that they showed us we weren’t alone. We were all struggling with the same issues, and it was okay not to be perfect in the way we were attacking all the problems. We were all planning pragmatic trials, and it was okay if we showed our warts and all.

What other challenges have you faced?

Having a good team at the primary organization is really important. It’s important to have a good administrative team—a program manager, project director—who can hold down the fort, especially with pragmatic trials, when you’re traveling to a lot of sites.

What advice do you have for investigators conducting their first embedded pragmatic clinical trial?

Give yourself a break. There are a ton of imperfections in conducting embedded pragmatic trials. It’s all about the people. Pick great site principal investigators. That’s more important than anything else about the institutions. You want to have enough eligible patients that you’re going to have an impact or whatever else is involved in picking your sites. Enthusiasm and organizational savvy go a long way. Be patient and flexible and open to new iterations of what you’re doing. It feels scary at first, but I think it will serve you to be open to change.

PRIM-ER is supported within the NIH Collaboratory by cooperative agreements from NCCIH and the National Institute on Aging and receives logistical and technical support from the NIH Collaboratory Coordinating Center. Read more about PRIM-ER and the NIH Collaboratory Demonstration Projects.

June 20, 2019: EMBED Investigators Discuss Progress and Transition to Implementation Phase

At the May 2019 meeting of the NIH Collaboratory Steering Committee, we talked with Drs. Ted Melnick and Gail D’Onofrio of EMBED, an NIH Collaboratory Demonstration Project, to hear about progress and challenges during the UG3 planning phase. The goal of EMBED is to test whether implementation of a user-centered clinical decision support system increases adoption of initiation of buprenorphine/naloxone into the routine emergency care of patients with opioid use disorder. In the UG3 phase, the study team put in place the infrastructure of a pragmatic, multicenter, parallel, group-randomized health IT intervention. EMBED recently transitioned to the UH3 implementation phase and plans to launch the intervention at 20 sites across 5 healthcare systems in August 2019.

“With EMBED, we’re trying to take evidence-based research and implement it to improve practice. EMBED is both a research and patient care project.”

Were there any surprises during the study’s planning phase?

The first surprise came at last year’s Steering Committee meeting, when we met with the Biostatistics and Study Design Core. They encouraged us to change our original study design from stepped-wedge to group-randomized, which we did. We think this advice led to a stronger study. The main reason for this is the group-randomized design’s ability to better account for temporal changes. Since our intervention is being conducted in the middle of an opioid crisis, there are potentially other concurrent interventions that could make it difficult to determine the effect of our intervention. The group-randomized design should give us better insight into whether our intervention is driving behavior change in treating patients with opioid use disorder.

What is an example of a challenge that you were able to overcome with the help of a Core Working Group?

In addition to design advice from the Biostatistics Core, we received expert guidance from the Ethics and Regulatory Core, who helped us prepare for the central IRB process. The Core’s input was essential to how we developed our protocol’s waiver of informed consent, data handling, and protection of patient privacy. We were able to demonstrate to the IRB that our approach was logical and informed. We think this helped the IRB “get it” and allowed us to more efficiently address patient privacy issues in a vulnerable population across multiple healthcare systems.

What other key challenges have you faced?

One challenge was on the IT side with electronic health record (EHR) integration, which required more customization than we initially planned. How we work with EHR vendors is evolving, and we’ve found good partners so that we can integrate across different systems. This has strengthened our intervention so that it is perceived as more universal than one designed only for a specific EHR system.

Another challenge is the general under-resourcing of healthcare delivery systems for pragmatic research. We found that, regardless of budget, getting approval from system leadership for an IT change is often not enough—what is needed is figuring out who is going to make the change, how much time is involved, and whether the team has the bandwidth to complete the task. You cannot underestimate the degree of difficulty a change poses to a health system that is still struggling to get the clinical side of things right.

The way a study is framed to leadership is important—understand what’s motivating them to participate and move a project forward. With EMBED, we’re trying to take evidence-based research and implement it to improve practice. EMBED is both a research and patient care project. We need to impress upon leadership that we can improve patient outcomes and we’ll pay for it, but we need their help and support in navigating the process through the institution.

What words of advice do you have for investigators conducting their first embedded PCT?

  • Study teams should think about potential barriers from the beginning and find solutions quickly.
  • Make sure that health system leadership discusses your project with those on the ground.
  • Enlist the experts your study needs for each site. In our case, we needed both an IT expert for the operational side and a clinical expert, or we couldn’t have moved the project forward.
  • Recognize that there are trade-offs in pragmatic design and remember that you’re working with health systems in which your intervention will need to be replicated.
  • Make your intervention sustainable and easily usable by the clinician, without the need for research or other additional staff.

EMBED is supported within the NIH Collaboratory by a cooperative agreement from the National Institute on Drug Abuse and receives logistical and technical support from the NIH Collaboratory Coordinating Center. Read more about EMBED in the Living Textbook, and learn more about the NIH Collaboratory Demonstration Projects.

June 13, 2019: Experience With Pragmatic Clinical Trials Gains Momentum

At the NIH Collaboratory Steering Committee Meeting in May 2019, participants shared their perspectives on the evolving landscape of embedded pragmatic clinical trials (ePCTs). Three initiatives were presented: the Patient-Centered Outcomes Research Institute (PCORI), the NIH-DoD-VA Pain Management Collaboratory, and the HEAL (Helping to End Addiction Long-term) Initiative. Although many challenges remain, the conduct of ePCTs is gaining momentum, and the synergy between the initiatives, along with the fellowship they engender, will continue to help pave the way for more embedded pragmatic research in the future.

Dr. Ann Trontell, Associate Director of Clinical Effectiveness and Decision Science at PCORI, shared PCORI’s experience with pragmatic clinical studies. Since 2014, PCORI has awarded $494 million dollars for 43 pragmatic studies that range in size from 425 to 100,000 participants (median, approximately 1700). The studies include 2 observational, 27 individually randomized, and 14 cluster randomized trials in a wide range of therapeutic areas.

Dr. Trontell urged those developing proposals for pragmatic trials to make them fit for purpose, as opposed to emphasizing pragmatism, a theme echoed in the Developing a Compelling Grant Application chapter of the Living Textbook.





Dr. Robert Kerns, a director of the NIH-DoD-VA Pain Management Collaboratory, shared progress with pragmatic trials designed to evaluate whether evidence-based nonpharmacological approaches are effective for pain management among US military personnel and veterans.

Modeled after the NIH Collaboratory, the Pain Management Collaboratory is supporting 11 projects through a 2-year planning phase and a 2- to 4-year implementation phase. Subject matter experts at the Pain Management Collaboratory Coordinating Center (PMC3) support the projects by sharing tools, best practices, and resources.


Dr. Wendy Weber, Program Officer for the NIH Collaboratory Coordinating Center, introduced the HEAL initiative, which is designed to enhance pain management and improve prevention and treatment strategies for opioid misuse and addiction. The goal of the initiative is to provide scientific solutions to the opioid crisis. It includes a set of large-scale pragmatic trials that will receive logistical and technical support from the NIH Collaboratory Coordinating Center.


While experience with ePCTs is growing, many distinct challenges remain. As the conduct of ePCTs gains momentum, there is a rich opportunity to use collective experiences to refine best practices to real-world evidence generation and help solve urgent public health problems.

May 31, 2019: Adapting Clinical Trial Design to Meet the Needs of Learning Health Systems (Harriette Van Spall, MD, MPH)


Harriette G.C. Van Spall, MD, MPH, FRCPC
Associate Professor of Medicine
Department of Medicine, Division of Cardiology
Department of Health Research Methods, Evidence, and Impact
McMaster University
Population Health Research Institute


Adapting Clinical Trial Design to Meet the Needs of Learning Health Systems


Learning health system; Pragmatic clinical trial; Patient-Centered Care Transitions in Heart Failure (PACT-HF); Heart failure; Stepped-wedge cluster trial

Key Points

  • Characteristics of a learning health system include:
    • Possessing a culture of knowledge and quality improvement
    • Encouraging research innovation by embedding research into clinical practice and generating knowledge at the point of care
    • Harnessing data from electronic health records and claims/administrative databases
    • Fostering trust between research and clinical teams
    • Engaging patients, clinicians, and key stakeholders
  • The Patient-Centered Care Transitions in Heart Failure (PACT-HF) trial evaluated the effectiveness of a group of transitional care services in patients hospitalized for HF within a publicly funded healthcare system.
  • Challenges of a learning health system include integrating care, intervention, and communications across silos; streamlining workflow; preventing “contamination” of usual care; and the limited interoperability of EHRs and slow updates to claims/administrative datasets.

Discussion Themes

Efficacy in explanatory randomized clinical trials (RCTs) does not equate to effectiveness in real-world settings.

Decisions about implementation of an intervention are not made “live”; you must wait until the study has ended, all the data are available for analysis, and analysis is complete before you can inform decision-maker partners about the risks and benefits of the intervention.

Read more about the PACT-HF study and results in JAMA Network (Van Spall et al. 2019)


#pctGR, @Collaboratory1