January 19, 2018: New Research Methods Resources Website on Group- or Cluster-Randomized Studies

The National Institutes of Health (NIH) Office of Extramural Research has released new clinical trial requirements for grant applications and contract proposals due on or after January 25, 2018. In anticipation of these new requirements, the NIH modified the Application Guide and the Review Criteria to address methodological problems common to many clinical trials. As group- or cluster-randomization designs are increasingly common in both basic and applied research, the new Application Guide includes links to the new Research Methods Resources website, which provides resources for investigators considering these group- or cluster-randomized designs, including lists of NIH webinars, key references, and statements to help investigators prepare sound applications and avoid methodological pitfalls.

December 7, 2017: Dr. Greg Simon Explains Individual, Cluster, and Stepped-Wedge Randomization in a New Prop Video

In a new video in the Living Textbook, Dr. Greg Simon describes the differences between individual, cluster, and stepped-wedge randomization using props, including marbles, Play-Doh, and glassware.

“In the end, it’s all about randomly assigning who gets which treatment, or who gets which treatment when, so that we’re able to make some un-biased judgement about which treatment is really better.” —Greg Simon, MD

November 20, 2017: NIH Collaboratory Core Working Group Interviews: Reflections from the Regulatory/Ethics Core

We recently asked Drs. Jeremy Sugarman and Kevin Weinfurt, Co-chairs of the Regulatory/Ethics Core, to reflect on the first 5 years of the Core as well as on the challenges ahead. The regulatory and ethical landscape for pragmatic clinical trials was not well defined when the Core began 5 years ago, and the Core helped to map and navigate the emerging landscape to enable the implementation of Demonstration Projects in ways that satisfied ethical and regulatory criteria.

“The Core’s work has led to the creation of a substantial body of scholarship contributing to the ongoing policy and ethics debates about pragmatic clinical trials.” – Drs. Sugarman and Weinfurt

Download the interview (PDF).

October 18, 2017: NIH Collaboratory Core Working Group Interviews: Reflections from the Biostatistics and Study Design Core

We recently asked Dr. Liz DeLong, Chair of the Biostatistics and Study Design Core, to reflect on the first 5 years of the Core’s work and the challenges ahead. She says the biggest impact of the Core has been working with the individual Demonstration Projects to provide a sounding board to discuss statistical challenges. Further, Core members have contributed to new knowledge through manuscripts that address key methodological issues related to pragmatic clinical trials. She’s hoping the Core will continue to push the boundaries of statistical methods in the coming years.

“The statisticians on the individual trials have not only developed excellent statistical methods for their own studies, but also contributed substantively to the Core.” Dr. Liz DeLong

Download the interview (PDF).

October 16, 2017: NIH Collaboratory Core Working Group Interviews: Reflections from the Stakeholder Engagement Core

At the NIH Collaboratory Steering Committee meeting in May 2017, we asked Ellen Tambor, a member of the Stakeholder Engagement Core, to reflect on the first 5 years of the Core’s work and the challenges ahead. She says it’s key for stakeholder engagement to take place throughout the entire healthcare system, from leadership to the frontline providers and staff. And, because of the nature of pragmatic trials conducted in clinical settings, engagement is essential from the early stages through trial completion. Tambor also suggests asking two questions to ensure the right people are involved throughout pragmatic research: Who is going to use the evidence that results from the study? Who will help ensure that the study is implemented as seamlessly as possible?

“Pragmatic trials take stakeholder engagement to a new level of importance in terms of both the scope of engagement and the array of potential stakeholders.” Ellen Tambor

Download the interview (PDF).

NIH & FDA seek feedback on new clinical trial protocol template

As part of their ongoing effort to improve the speed and efficiency of conducting clinical trials, the NIH-FDA Joint Leadership Council has created a draft clinical trial protocol template. The template contains instructional and sample text intended to assist NIH-funded investigators in writing protocols for phase 2 or 3 clinical trials that require Investigational New Drug (IND) or Investigational Device Exemption (IDE) applications. Feedback is sought from investigators, investigator-sponsors, institutional review board members, and other stakeholders involved in protocol development and review.

Our goal is to provide an organized way for creative investigators to describe their plans so that others can understand them. – Dr. Pamela McInnes, NIH

Details on the rationale and development of the protocol template are on these blog posts:

Notice Number: NOT-OD-16-043. Responses accepted through April 17, 2016.

You can access the template document as well as a template shell, comment form, and other resources at NIH’s Clinical Research Policy website.

STOP CRC Trial: Analytic Challenges and Pragmatic Solutions

Investigators from the STOP CRC pragmatic trial, an NIH Collaboratory Demonstration Project, have recently published an article in the journal eGEMs describing solutions to issues that arose in the trial’s implementation phase. STOP CRC tests a program to improve colorectal cancer screening rates in a collaborative network of Federally Qualified Health Centers by mailing fecal immunochemical testing (FIT) kits to screen-eligible patients at clinics in the intervention arm. Clinics in the control arm provided opportunistic colorectal-cancer screening to patients at clinic visits in Year 1 and implemented the intervention in Year 2. In this cluster-randomized trial, clinics are the unit of analysis, rather than individual patients, with the primary outcome being the proportion of screen-eligible patients at each clinic who complete a FIT.

The team dealt with various challenges that threatened the validity of their primary analysis, one of which related to potential contamination of the primary outcome due to the timing of the intervention rollout: for control participants, the Year 2 intervention actively overlapped with the Year 1 control measurements. The other challenge was due to a lack of synchronization between the measurement and accrual windows. To deal with these issues, the team had to slightly modify the study design in addition to developing a few sensitivity analyses to better estimate the true impact of the intervention.

“While the nature of the challenges we encountered are not unique to pragmatic trials, we believe they are likely to be more common in such trials due to both the types of designs commonly used in such studies and the challenges of implementing system-based interventions within freestanding health clinics.” (Vollmer et al. eGEMs 2015)

The Publish EDM Forum Community publishes eGEMs (generating evidence & methods to improve patient outcomes) and provides free and open access to this methods case study. Readers can access the article here.

Applying PRECIS Ratings to Collaboratory Pragmatic Trials

A new article published in the journal Trials provides a look at how the  Pragmatic–Explanatory Continuum Indicator Summary, or PRECIS, rating system can be applied to clinical trials designs in order to examine where a given study sits on the spectrum of explanatory versus pragmatic clinical trials.

The PRECIS-2 criteria are used to rate study designs as more or less “pragmatic” according to multiple domains that include participant eligibility, recruitment methods, setting, organization, analysis methods, primary outcomes, and more. In this context, “pragmatic” refers to trials that are designed to study a therapy or intervention in a “real world” setting similar or identical to the one in which the therapy will actually be used. Pragmatic trials stand in contrast to explanatory trials, which are typically designed to demonstrate the safety and efficacy of an intervention under highly controlled conditions and in carefully selected groups of participants, but which may also be difficult to generalize to larger or more varied populations.

Schematic of PRECIS-2 Wheel used to evaluate where a given trial design resides upon the explanatory-pragmatic spectrum.
PRECIS-2 Wheel.  Kirsty Loudon et al. BMJ 2015;350:bmj.h2147. Copyright 2015 by British Medical Journal Publishing Group. Used by permission.

Clinical trials are almost never wholly “explanatory” or wholly “pragmatic.” Instead, many studies exist somewhere on a spectrum between these two categories. However, understanding how these different attributes apply to trials can help researchers design studies that are optimally fit for purpose, whether that purpose is to describe a biological mechanism (as in an explanatory trial) or to show how effective an intervention is when used across a broad population of patients (as in a pragmatic trial).

In their article in Trials, authors Karin Johnson, Gila Neta, and colleagues  applied PRECIS-2 criteria to 5 pragmatic clinical trials (PCTs) being conducted through the NIH Collaboratory. Each trial was found to rate as “highly pragmatic” across the multiple PRECIS-2 domains, highlighting the tool’s potential usefulness in guiding decisions about study design, but also revealing a number of challenges in applying it and interpreting the results.

Study authors Johnson and Neta will be discussing their findings during the NIH Collaboratory’s Grand Rounds on Friday, January 22, 2016 (an archived version of the presentation will be available the following week).

Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, Rundell S, Smith DH, Liu B, Taplin S, Stoney CM, Farrell MM, Glasgow RE. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17(1):32. doi: 10.1186/s13063-016-1158-y. PMID: 26772801. PMCID: PMC4715340.
You can read more about the NIH Collaboratory PCTs featured as part of this project at the following links:

ABATE (Active Bathing to Eliminate Infection)

LIRE (A pragmatic trial of Lumbar Image Reporting with Epidemiology)

PPACT (Collaborative Care for Chronic Pain in Primary Care)

STOP-CRC (Strategies & Opportunities to Stop Colon Cancer in Priority Populations)

TIME (Time to Reduce Mortality in End-Stage Renal Disease)
Additional Resources

An introductory slide set on PCTs (by study author Karin Johnson) is available from the Living Textbook:

Introduction to Pragmatic Clinical Trials

The University of Colorado Denver - Anschutz Medical Campus publishes an electronic textbook on pragmatic trials:

Pragmatic Trials: A workshop Handbook




New Lessons Learned Document Draws on Experiences of Demonstration Projects

The NIH Collaboratory’s Health Care Systems Interactions Core has published a document entitled Lessons Learned from the NIH Health Care Systems Research Collaboratory Demonstration Projects. The Principal Investigators of each of the Demonstration Projects shared their trial-specific experience with the Core to develop the document, which presents problems and solutions for initiation and implementation of pragmatic clinical trials (PCTs). Lessons learned are divided into the following categories: build partnerships, define clinically important questions, assess feasibility, involve stakeholders in study design, consider institutional review board and regulatory issues, and assess potential issues with biostatistics and the analytic plan.

Other tools available from the Health Care Systems Interactions Core include a guidance document entitled Considerations for Training Front-Line Staff and Clinicians on Pragmatic Clinical Trial Procedures and an introduction to PCTs slide set.

Findings from STOP CRC on Pragmatic Trial Recruitment

Gloria Coronado, PhD, and Beverly Green, MD, MPH, Principal Investigators, STOP CRC Trial
Gloria Coronado, PhD, and Beverly Green, MD, MPH, Principal Investigators, STOP CRC Trial

Drs. Beverly Green and Gloria Coronado and colleagues have published an article in Clinical Trials describing the challenges of recruiting participants into large, multisite pragmatic clinical trials—particularly at the health system level. STOP CRC is one of the NIH Collaboratory’s pragmatic clinical trial Demonstration Projects, which are intended to provide a framework of implementation methods and best practices to enable participation of varied health care systems in clinical research.

STOP CRC is testing a culturally tailored, health care system–based program to improve colorectal cancer screening rates in a community-based collaborative network of federally qualified health centers. The authors observed that recruiting sites to participate in pragmatic trials is time-intensive and involves both preparing materials and organizing face-to-face meetings with staff and clinic leaders. Yet little is known about the characteristics of nonparticipating sites and clinic-level factors that may influence willingness to participate in a pragmatic trial.

“Our findings underscore the importance of assessing and reporting recruitment success at the organizational and/or clinic level in order to know the external validity of the findings and may inform future efforts to select and recruit health systems to participate in pragmatic research.” (Coronado, et al. Clin Trials 2015)