October 9, 2025: New Living Textbook Chapter Explores Decentralized Elements of Pragmatic Clinical Trials

The NIH Pragmatic Trials Collaboratory this week published a new chapter of its Living Textbook of Pragmatic Clinical Trials. The chapter, Decentralized Pragmatic Clinical Trials, covers activities of a pragmatic trial that can occur remotely—at a location separate from an investigator’s location—such as participant engagement, recruitment, consent, study interventions and procedures, collection of patient-reported outcomes, and follow-up.

The chapter describes special considerations for decentralized trials, such as community health considerations and the vigilance needed to assure data quality, particularly as it relates to adherence with the study intervention, outcome ascertainment, and event monitoring.

The new chapter includes the following sections:

  1. What Is a Decentralized Trial?
  2. What Decentralized Elements Are Used in Pragmatic Trials?
  3. Community Health Considerations for Decentralized Approaches
  4. Quality Assurance

Most of NIH Collaboratory Trials have decentralized elements, as described in detail in Section 2.

October 3, 2024: JAMA Commentary Highlights Gap Between Research Results and Implementation Decisions

Headshots of Richard Platt, Hayden Bosworth, and Greg SimonIn a JAMA Viewpoint published online this week, leaders from the NIH Pragmatic Trials Collaboratory discuss the discordance between the results of pragmatic clinical trials and the implementation of those results in healthcare settings, even in settings that championed the work.

Coauthors Richard Platt, Hayden Bosworth, and Gregory Simon posit that, to provide evidence that healthcare systems leaders will actually use, changes are necessary:

  • Trials need to be faster (2 to 3 years)
  • Trials should consider outcomes that healthcare system leaders care about (such as costs and subgroup analyses)
  • The evidence required for change should be at the level that healthcare system leaders use (such as not necessarily rejecting the null hypothesis at P < .05 but also considering results of Bayesian methods and interim analyses)

Read the full article.

“We believe it is possible to make pragmatic clinical trials of policies and procedures more useful to delivery systems by accommodating their priorities, introducing flexibility in the level of evidence trials require, shortening the duration of planning and implementation, addressing system leaders’ interest in multiple outcomes and subgroup analyses, encouraging modification of protocols during the implementation phase, and by providing timely interim analyses that can guide decisions about continuing or modifying an intervention,” the authors wrote.

This work was based on an NIH Pragmatic Trials Collaboratory workshop held in 2023, Getting the Right Evidence to Decision-Makers Faster. The workshop explored the critical cycle of evidence generation to decision by healthcare system leaders to implement the findings of pragmatic clinical trials conducted within healthcare systems.

Platt is a cochair of the NIH Collaboratory’s Distributed Research Network and the Harvard Pilgrim Health Care Institute Distinguished Professor of Population Medicine at the Harvard Pilgrim Health Care Institute and Harvard Medical School. Bosworth is a cochair of the Implementation Science Core and a professor in population health sciences at Duke University. Simon is the chair of the Health Care Systems Interactions Core and a senior investigator at the Kaiser Permanente Washington Health Research Institute.

View the full workshop materials.

August 8, 2024: FDA Commissioner and NIH Director Share Thoughts on the Future of Pragmatic Clinical Trials

Speaking at the NIH Pragmatic Trials Collaboratory’s 2024 Annual Steering Committee Meeting in May, Commissioner of Food and Drugs Robert Califf and NIH Director Monica Bertagnolli discussed the intersection of pragmatic research and national health priorities.

Califf and Bertagnolli spoke about the role the NIH Collaboratory can play in promoting scientifically rigorous, pragmatic clinical research that addresses the urgent need for better and faster implementation of research findings into patient care. They also addressed shortcomings of the electronic health record (EHR) and the challenges of building a learning health system.

Dr. Monica Bertagnolli and Dr. Robert Califf speak at the NIH Pragmatic Trials Collaboratory's 2024 Annual Steering Committee Meeting

Wendy Weber of the National Center for Complementary and Integrative Health moderated the discussion, which focused on 3 challenges in clinical research and their relation to the NIH Collaboratory:

  • Medical providers and patients lack scientific evidence to support medical decisions
  • A learning health system can help to develop this evidence, but challenges in developing this system remain
  • How can the NIH Pragmatic Trials Collaboratory aid in these challenges?

“The most fundamental flaw we have in general right now is that we don't tell people that their biggest risk is getting standard healthcare in the healthcare system,” Califf said.  “Without evidence, clinicians don't actually know what to do. We act like we know, but we don’t,” he added.

Medical providers and patients alike must often make underinformed decisions. The lack of scientific evidence to support these decisions poses a challenge, as it demands clinicians use their individual experience and judgment when making a recommendation. Here, the deficiencies of the EHR become most obvious, as it fails to meet the needs of the population by not providing sufficient data for providers to refer to. Bertagnolli emphasized this, stating the EHR needed to become a learning environment, facilitating both the collection and dissemination of knowledge even for busy doctors.

“We need to reach every person,” Bertagnolli challenged the researchers in the audience. “We need to get in every corner of the United States with our data, with our research. We need to make it possible for busy doctors and clinics to be in a learning environment,” she said.

“What's the one unifying bit of technology we have that is not delivering what we need? It's the electronic health record,” Bertagnolli added.

Developing a health system that fosters collaboration and constant learning is paramount to obtaining generalizable medical evidence. Califf emphasized this need, calling for a singular “voice” that can represent the population’s prevention and health issues, collectively demonstrating what works.

However, several barriers hinder the development of such a system. The healthcare industry often finds that delivery practices that optimize profit margins are not optimal for patient health outcomes, which creates an environment of ethical uncertainty. Furthermore, the United States spends more on healthcare than do comparable countries with higher life expectancies, leading to unnecessary expenditures and unwillingness to adopt new initiatives.

“We're paying for a lot of things that don't help. So how are we going to figure out what to stop doing?” Bertanolli asked. “The only way is data. And the only people who can bring that data are our care providers to collect those data and learn from them. It is imperative to get a learning health system in progress,” she said.

The NIH Pragmatic Trials Collaboratory has long played a role in establishing cost-effective, large-scale research that treats care providers as research partners to deliver generalizable, actionable data. The program’s efforts in promoting collection of data from clinical care environments ensure that research remains grounded in real-world patient outcomes. Moreover, the NIH Collaboratory's emphasis on gathering data from underrepresented populations enhances the relevance of research findings, addressing disparities in healthcare delivery and outcomes.

Along with data collection, Bertagnolli encouraged the NIH Collaboratory to be an advocate for this new health system and encourage organizations to participate in the development of this system. The NIH Collaboratory brings years of specialized clinical knowledge and input, ensuring that the system is informed by evidence-based practices and responsive to the needs of both care providers and patients.

“It's a matter of convincing people to get over the hump, embrace a learning health system, and participate—and specifically, I'd say get the wheel turning as fast as you can, answer as many questions as you can, and talk about it,” added Califf.

View the full materials from the 2024 Annual Steering Committee Meeting.

November 28, 2023: Workshop Summary Now Available From ‘Getting the Right Evidence to Decision-Makers Faster’

The workshop summary is now available from the NIH Pragmatic Trials Collaboratory’s recent workshop, “Getting the Right Evidence to Decision-Makers Faster.” The 2-day workshop explored the critical cycle of evidence generation by researchers to decision-making by healthcare system leaders to implement the findings of pragmatic clinical trials conducted within healthcare systems.

The workshop included 4 panels:

  • Panel 1: How Have Health Systems Made Decisions Based on Evidence Collected in PCTs?

    • Panelists: Devon Check, Vincent Mor, Lynn DeBar, Kathryn Glassberg, Douglas Zatzick, Eileen Bulger, Susan Huang, Kenneth Sands, Edward Septimus; Moderator: Gregory Simon
  • Panel 2: How Do We Generate the Right Evidence to Support Decision-Makers?

    • Panelists: Kenneth Sands, Eileen Bulger, Edward Septimus, Amy Kilbourne, Rosa Gonzalez-Guarda, Patrick Heagerty; Moderator: Hayden Bosworth
  • Panel 3: Learning Faster

    • Panelists: Gloria Coronado, Natalia Morone, Corita Grudzen, Kevin Chan, Pearl O’Rourke, Cheryl Boyce, Andrea Cook; Moderator: Kevin Weinfurt
  • Panel 4: Potential Structures and Incentives for Faster Learning

    • Panelists: David Chambers, Wynne Norton, Tisha Wiley, Kenneth Sands, Edward Septimus, Gloria Coronado, Natalia Morone, Corita Grudzen; Moderator: Richard Platt

Access the complete workshop materials, including slides and videocast recordings, as well as the keynote presentation by Andrew Bindman, executive vice president and chief medical officer for Kaiser Permanente.

August 31, 2023: FDA’s Robert Califf and Coauthors Assert More Is Needed to Achieve Learning Health Care

A graphic that includes the cover image from the August 2023 issue of the American Journal of Bioethics. The text in the graphic reads as follows: "American Journal of Bioethics Special issue on pragmatic trials, featuring target articles from the NIH Pragmatic Trials Collaboratory."In an article published this month in the American Journal of Bioethics, FDA Commissioner Robert Califf and coauthors suggest that—despite the potential of embedded pragmatic research to generate information to improve clinical practice and public health policy—it is still relatively uncommon in US healthcare.

“Simply stated, what we are currently doing does not work, and in the face of declining health status we lack answers to critical questions about what we should be doing in health care and public health practice.”

The authors state 3 major obstacles:

  • Inadequate data systems: Electronic health records are not designed for research use, and are driven by billing codes and reimbursement structures.
  • Data sharing malaise: We have failed to develop a convincing paradigm for sharing individual-level data from routine healthcare delivery
  • Current oversight: Research oversight is still not designed to facilitate embedded pragmatic clinical trials or research using real-world evidence.

The authors suggest that achieving a learning health system will require

  • More collaboration between health systems and businesses involved in healthcare
  • More innovative structures for data sharing across institutions
  • Incentives for building the sophisticated infrastructure necessary to enable this work
  • Considerations from the bioethics community about how best to foster this research while respecting all those who participate

This article was part of a special issue of the American Journal of Bioethics on pragmatic clinical trials. Members of the Ethics and Regulatory Core contributed the target articles to this issue regarding investigator obligations and the clinician’s duty to participate in embedded research.

 

June 7, 2022: Reflecting on 10 Years of the Health Care Systems Interactions Core

The NIH Pragmatic Trials Collaboratory Heath Care Systems Interactions Core supports and facilitates productive collaboration between researchers, clinicians, and health system leaders to conduct effective, relevant embedded pragmatic clinical research.

Health Care Systems Interactions Core Co-Chairs Dr. Eric B. Larson and Dr. Gregory Simon discussed the Core’s progress over the last 10 years in an interview at the NIH Pragmatic Trials Collaboratory Steering Committee meeting in April.

Over the last 10 years, the Health Care Systems Interactions (HCS) Core has allowed researchers to learn about working with healthcare delivery systems. Knowledge that is now common, was unknown when the Core was started, such as how dynamic healthcare delivery systems are and how the capabilities of and changes to the electronic health record can impact pragmatic clinical trials.

Through this type of discovery, the HCS Core has helped researchers become more sensitive to and aware of the priorities of healthcare delivery systems, resulting in better collaboration.

“The researchers’ priorities are usually not the same as the priorities of the people we are working with, whether they are patients, providers, or delivery systems. You have to know what other people’s purpose and drivers are and find a way to adapt,” said Larson. “We have learned and taken pretty seriously this idea of a learning health system with bidirectional engagement from research and from elements of the delivery system.”

The work of the HCS Core and NIH Pragmatic Trials Collaboratory has created a safe haven where researchers can share experiences and advance the field with common learning.

Simon sees the HCS Core as having internal and external missions. The internal mission is to support NIH Collaboratory Trials and be a community where researchers can come together, share their trials and tribulations, and experts in the Core can help these projects be successful, he described.

The external mission is focused on generalizable knowledge and advocacy. The HCS Core has shared knowledge with the research community and funders through publications and meetings and is advocating for research that includes the healthcare delivery system perspective.

“The discussion we are having is not just how do I work with healthcare systems to do my research, but how do I engage with healthcare systems about what research we should be doing, what are the right questions we should be asking for the studies that will be happening 5 years from now not the studies that are already underway,” Simon said.

Another lesson the HCS Core has learned is the importance of being flexible and adjusting. This lesson has been particularly relevant during the COVID-19 pandemic.

“COVID-19 is an extreme case of health system overwhelm, but I think we need to recognize that if we are going to serve people that have been traditionally not been well-served by the healthcare system, we will often be dealing with health systems that are chronically overwhelmed,” said Simon. “How do we do research in those settings? There are some really interesting challenges to think about.”

The HCS Core is focused on continued engagement between researchers and healthcare delivery systems that results in implementable new knowledge.

“My belief is that if we have the upstream involvement and are engaged in research projects that matter to the delivery system from the patients all the way up to the executives, we have a much better chance that when a result is valuable it becomes implementable and spreads to benefit everybody,” Larson said.

View the full interview.

See the complete materials from the 2022 Steering Committee meeting.

July 1, 2021: NIH Collaboratory Leadership Asks, ‘Is Learning Worth the Trouble?’

Cover of the New England Journal of MedicineIn an article published today in the New England Journal of Medicine, Drs. Richard Platt, Adrian Hernandez, and Greg Simon of the NIH Collaboratory discuss barriers to healthcare system participation in embedded research and strategies for improvement.

“We advocate creating a robust national [embedded pragmatic clinical trial] capability to generate evidence to guide decisions by patients, clinicians, health systems, and regulators and respond to urgent national health crises, like COVID-19 or the opioid crises,” the authors wrote.

The article recommends a 4-pronged strategy that researchers and funders should consider to increase healthcare system participation in pragmatic clinical trials:

  • Reimburse for the additional costs of trial participation.
  • In some highly engaged systems, support permanent, reusable infrastructure.
  • Offload research-specific tasks to minimize burden on sites (such as IRB oversight, obtaining informed consent, and mailing medications to participants).
  • Assign and promote reputational benefit for these activities.

In another perspective piece by Simon, Platt, and Hernandez published in the April 2020 issue of the journal, the authors explored why randomized A vs B comparisons remain uncommon in clinical trials.

June 24, 2021: New Article Offers Guidance on Accounting for Quality Improvement Activities During Embedded Pragmatic Trials

Cover of the journal HealthcareIn a new article in Healthcare’s special issue on embedded research, Leah Tuzzio and colleagues use experiences from the NIH Collaboratory to describe how quality improvement activities may pose challenges for embedded pragmatic clinical trials (ePCTs), especially if there are overlapping goals and timelines.

For ePCTs to be rigorous, study teams must monitor, adapt, and respond to QI during the design and the trial implementation. Both ePCTs and QI happen within the same context and aim to improve patient care, and they are inherently interconnected. — Tuzzio et al, Healthcare, 2021

ePCTs tend to be larger and more broadly generalizable than quality improvement initiatives and may generate high-quality evidence for care and clinical practice guidelines. Quality improvement initiatives may address the same high-impact health questions, but if they co-occur with ePCTs, they may dilute or confound the ability to detect change. As a result, study teams may need to monitor, adapt, and respond to quality improvement initiatives during the design and conduct of the trial. The authors suggest that routine collaboration with healthcare system stakeholders can help align research and quality improvement to support high-quality, patient-centered care.

Publication of the special issue was supported by AcademyHealth.

For more, see the Living Textbook Chapters on Advice from Health Care System Leadership,  Building Partnerships to Ensure a Successful Trial, and Monitoring Intervention Fidelity and Adaptations.

May 14, 2021: Keynote Session: Diverse Representation Among Clinical Trial Participants: Why It Is Important and How Can We Improve (Clyde W. Yancy, MD, MSc)

Speaker

Clyde W. Yancy, MD, MSc
Vice Dean, Diversity and Inclusion
Professor of Medicine
Chief, Cardiology
Feinberg School of Medicine
Northwestern University

Guest Moderator:
Kanecia Zimmerman, MD
Associate Professor of Pediatrics
Duke University School of Medicine

Topic

Diverse Representation Among Clinical Trial Participants: Why It Is Important and How Can We Improve

Keywords

Clinical trials; Health outcomes; Racial disparities; Diverse participant recruitment; Disease burden; Cardiovascular disease

Key Points

  • The reason we do clinical research is to improve the health of the community. But trials do not represent the full spectrum of the community with respect to elderly patients, minorities, and women. In excluding overlooked populations, trials do not explore—or generate appropriate evidence about—the totality of a health intervention’s benefit.
  • It is urgent that we advance the overall “research IQ” of the populace, thus overcoming a legacy of mistrust of the research enterprise and reducing barriers to participation in clinical trials.
  • Among the steps to advance diversity in clinical trials:
    • Consider economic incentives or penalties by FDA or payers
    • Revisit the design of trials, selection of investigators and sites, and geographic balance
    • Recruit and train more diverse coordinator and investigator research teams
    • Engage with peer investigators outside the U.S. to target more race/ethnicity diversity and gender balance in clinical trial recruitment
    • Incorporate novel digital health technologies to expand the pool of potential research participants

Discussion Themes

In cardiovascular health, having diverse representation in clinical trials is clinically necessary to address ongoing disparities. It’s essential that trialists study the condition in populations that have borne an outsized burden of disease.

To diversify and expand the populations we study, we must think differently and be intentional from the outset. When we start to get truly diverse representation in clinical trials—when we actually study the person who has the condition—there will be robust enthusiasm and a greater sense of purpose throughout the clinical trial ecosystem.

Adaptive trial designs could be used to see if recruitment is on target and then make real-time adjustments to catch missing populations.

In thinking about accountability, what is the role of journals and ClinicalTrials.gov on reporting of race/ethnicity of both participants and investigators?

Read more about how to enhance diversity in clinical trials in recent FDA guidance and in cardiovascular trials in particular in Ortega et al., Circulation, 2019.

Tags

#pctGR, @Collaboratory1

April 16, 2021: Minnesota EHR Consortium COVID-19 Project: A Statewide Collaboration to Inform Vaccine Equity (Paul E. Drawz, MD, MHS, MS; Tyler Winkelman, MD, MSc)

Speakers

Paul E. Drawz, MD, MHS, MS
Associate Professor
Division of Renal Disease and Hypertension
University of Minnesota

Tyler N.A. Winkelman, MD, MSc
Co-Director, Health, Homelessness, and Criminal Justice Lab
Associate Director, Virtual Data Warehouse
Hennepin Healthcare Research Institute

Topic

Minnesota EHR Consortium COVID-19 Project: A Statewide Collaboration to Inform Vaccine Equity

Keywords

COVID-19; Electronic health records (EHRs); Data analysis; Research consortium; Healthcare systems; Population health; Distributed data network; Vaccine equity

Key Points

  • The EHR Consortium’s COVID-19 vaccine project aims to inform policy and practice through data-driven collaboration among members of Minnesota’s health care community.
  • The collaborative network can monitor population-level health metrics and analyze changes over time using aggregations of data to inform public health policy. Sources of data include EHRs, census data, state-wide electronic immunization records, and population data.
  • The COVID-19 vaccine dashboard is updated weekly and provides data at the ZIP level by age categories and race/ethnicity.
  • Minnesotans who have received a COVID-19 vaccine (any source) and had a visit at a consortium site in the last 10 years (~90 percent of the state population) are reflected in the dashboard.

Discussion Themes

How were you able to convene this consortium during a pandemic year?

Was your hashing algorithm home-grown or did you have an outside partner?

In the future, this infrastructure will be expanded to incorporate smaller health systems and additional content expertise around comorbidities, disease prevalence, and identification of disparities in near real-time.

Read more about the MN EHR Consortium at Hennepin Healthcare and the University of Minnesota Clinical & Translational Science Institute.

Tags

#pctGR, @Collaboratory1