data quality

October 9, 2025: New Living Textbook Chapter Explores Decentralized Elements of Pragmatic Clinical Trials

Posted on by Karen Staman

The NIH Pragmatic Trials Collaboratory this week published a new chapter of its Living Textbook of Pragmatic Clinical Trials. The chapter, Decentralized Pragmatic Clinical Trials, covers activities of a pragmatic trial that can occur remotely—at a location separate from an investigator’s location—such as participant engagement, recruitment, consent, study interventions and procedures, collection of patient-reported outcomes, and follow-up.

The chapter describes special considerations for decentralized trials, such as community health considerations and the vigilance needed to assure data quality, particularly as it relates to adherence with the study intervention, outcome ascertainment, and event monitoring.

The new chapter includes the following sections:

  1. What Is a Decentralized Trial?
  2. What Decentralized Elements Are Used in Pragmatic Trials?
  3. Community Health Considerations for Decentralized Approaches
  4. Quality Assurance

Most of NIH Collaboratory Trials have decentralized elements, as described in detail in Section 2.

October 28, 2024: New Living Textbook Contribution Explains Differences Between Medicare Data Sources

Posted on by Karen Staman

Living Textbook iconResearch-identifiable Medicare data can come from traditional fee-for-service Medicare claims or from Medicare Advantage claims. A new contribution to the Living Textbook of Pragmatic Clinical Trials published this month, Use of Medicare Data in PCTs, describes the important differences between these data.

At the healthcare system level, differences in incentives for documenting diagnoses can affect the reliability and relevance of data used for pragmatic clinical trials. The populations served by fee-for-service Medicare plans and Medicare Advantage plans are also disparate, as Medicare Advantage plans include a higher proportion of patients who require chronic disease management. There are also variations in enrollment rates across states and counties that reflect characteristics of the counties themselves (urban vs rural) and the firms that offer Medicare Advantage plans across regions.

Read the new contribution.

For more on using Medicare data in pragmatic trials, see the following Living Textbook sections:

July 3, 2023: Report Shares Strategies for Addressing Lack of Generalizability of EHR Data

Posted on by Damon Seils

JAMIA cover imagePragmatic research is vulnerable to differences in data capture and access to care for different subsets of the population, which, if left unaddressed, can worsen health gaps.

In a new article published online ahead of print in JAMIA, the NIH Pragmatic Trials Collaboratory's Trial teams reflect on the challenges encountered by their trials and share the specific strategies they used to increase the generalizability of research results.

“Poor generalizability can occur because detailed information about specific populations is missing, and critically, is missing not at random,” the authors cautioned.

The NIH Collaboratory Trials are implementing approaches designed to ensure inclusion and retention of all populations, and to enable the complete collection of data.

“By improving data capture, access to care, and patient technology support, ePCTs hold the potential to yield insights and estimates pertinent to the entire population, not just a subset of the population,” they wrote.

This work was a collaboration between the Community Health Improvement Core, the EHR Core, and the Patient-Centered Outcomes Core of the NIH Pragmatic Trials Collaboratory.

August 31, 2022: New Section of Living Textbook Addresses Evaluating Fitness for Use of Real-World Data Sources

Posted on by Karen Staman

A new section of the NIH Pragmatic Trials Collaboratory’s Living Textbook of Pragmatic Clinical Trials discusses challenges associated with Evaluating Fitness for Use of real-world data for trials. The section uses a case study from the Harmony Outcomes EHR Ancillary Study (eHARMONY) to describe lessons learned and to provide recommendations for studies considering incorporating real-word data as a data source.

Among the lessons learned were:

  • Standalone clinical research sites had very little extractable EHR data about patients.
  • Most lab results and medications were either not extractable or not mapped to a useful terminology.
  • Many sites did not have the ability to transform their data into a common format and had to send rudimentary data extracts to the ancillary study coordinating center. Sites participating in other research networks, such as PCORnet, had no difficulty with this task.

For more, read the chapter or watch the Grand Rounds presentation Leveraging RWD in a Multinational Trial: Results from the other eHARMONY 

July 14, 2022: Grand Rounds Podcast Now Available, Featuring Dr. John Concato on FDA Draft Guidance on Real-World Evidence

Posted on by terren.green@duke.edu

Headshot of Dr. John ConcatoIn the latest episode of the NIH Collaboratory Grand Rounds podcast, Dr. John Concato and Dr. Richard Platt continue their discussion about the FDA draft guidance on real-world evidence.

The full June 24 Grand Rounds webinar with Dr. Concato is also available.

Podcast July 8, 2022: FDA Draft Guidance on Real-World Evidence (John Concato, MD, MS, MPH)

Posted on by terren.green@duke.edu

This podcast continues the discussion with Dr. John Concato as he discusses the FDA draft guidance on real-word evidence. Click on the recording below to listen to the podcast.

Want to hear more? View the full Grand Rounds presentation.

For alerts about new episodes, subscribe free on Apple Podcasts or SoundCloud. Read the transcript.

June 24, 2022: FDA Draft Guidance on Real-World Evidence (John Concato, MD, MS, MPH)

Posted on by terren.green@duke.edu

Speaker

John Concato, MD, MS, MPH
Associate Director for Real-World Evidence Analytics
Office of Medical Policy (OMP)
Center for Drug Evaluation and Research (CDER)
Food and Drug Administration (FDA)

 

 

Keywords

Big data; Real-word evidence; Real-world data; 21st Century Cures Act; FDA Draft Guidance

 

Key Points

  • Big Data, a term first used in the 1990s, leverages modern technology to increase the quantity, forms, speed, and capability to manipulate large-scale data. Real-world data (RWD) is a term with specific regulatory implications referring to health care data routinely collected from a variety of sources. Real-world evidence (RWE) is clinical evidence derived from analysis of RWD regardless of study design.
  • Terminology is important in research work, and we should strive to be as precise as possible with the terminology we use.
  • With the 21st Century Cures Act of 2016, the FDA established a program to evaluate the potential use of real-world evidence to support new indications for drugs and satisfy post-approval study requirements.
  • In 2021, the FDA issued 4 draft guidance documents for Real-world data and Real-world evidence intended to guide the selection and management of data sources to appropriately address the study question and support decision-making for drug and biological products.

Discussion Themes

– Could real-world data sources be certified and preclude the need for submission of source data on a study specific basis? From the FDA point-of-view, while reliability can be more readily evaluated and would tend to be more stable, the relevance to a particular study could not be determined as easily.

– While there can be a reflex that says we can never be sure about major confounding, it should not be the miasma of the 21st century. A thoughtful approach that considers the characteristics that matter is the best approach.

 

Read Dr. Concato’s publication Randomized, observational, interventional, and real-world—What’s in a name? and the FDA Draft Guidance for RWD/RWE.

Tags

#pctGR, @Collaboratory1

October 5, 2021: New Article Identifies Challenges and Prerequisites for Using Electronic Health Record Systems for Pragmatic Research

Posted on by Karen Staman

JAMIA Cover

In a new NIH Collaboratory study, 20 NIH Collaboratory Trials responded to a survey about challenges encountered when using the electronic health record (EHR) for pragmatic clinical research. The goal of the study was to elucidate challenges and develop solutions—or prerequisites for pragmatic research—to enable healthcare system leaders, policy makers, and EHR designers to improve the national capacity for generating real-world evidence.

The article was published in the Journal of American Medical Informatics Association (JAMIA).

The challenges identified by the projects fell into 6 broad themes, including inadequate collection of patient-centered data, lack of functionality for structured data collection, lack of standardization, lack of resources to support customization, difficulties aggregating data from multiple sites, and difficult and inefficient access to EHR data.

Researchers from the NIH Collaboratory’s EHR Core and colleagues from the Patient-Centered Outcomes and the Health Care Systems Interactions Core Working Groups discussed the issues and iterated possible solutions. The authors developed the following prerequisites for the conduct of pragmatic research:

  • Integrate collection of patient-centered data into EHR systems
  • Facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows
  • Support creation of high-quality research data by using standards
  • Ensure adequate IT staff to support embedded research
  • Create aggregate, multidata type resources for multisite trials
  • Create reusable and automated queries

The authors argue for the ability to tailor EHR systems to enable the collection of patient-centered outcomes and the extraction of high-quality, standardized data. Although the primary uses of the data are for clinical care and billing, high-quality data from the EHR also have the potential to improve clinical care and population health by providing reliable evidence and to support pragmatic research and learning within and across healthcare systems.

Read the full article.

This work was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. This work was also supported by the NIH through the NIH HEAL Initiative under award number U24AT010961.

 

February 26, 2021: Calibrating Real-World Evidence Against RCT Evidence: Early Learnings from RCT-DUPLICATE (Sebastian Schneeweiss, MD, ScD)

Posted on by Gina Uhlenbrauck

Speaker

Sebastian Schneeweiss, MD, ScD
Chief, Division of Pharmacoepidemiology and Pharmacoeconomics
Department of Medicine, Brigham and Women’s Hospital
Professor of Medicine, Harvard Medical School
Professor in Epidemiology, Harvard T.H. Chan School of Public Health

Topic

Calibrating Real-World Evidence Against RCT Evidence: Early Learnings from RCT-DUPLICATE

Keywords

Real-world evidence (RWE); Randomized controlled trials (RCTs); Epidemiology; Emulation; Fit-for-purpose data

Key Points

  • While RCTs are an accepted research study design to establish the efficacy of medical products, RWE studies can complement the evidence generated by RCTs, as well as expand the line of inquiry around population, endpoints, treatment patterns, and comparators.
  • The RCT-DUPLICATE study aimed to understand and improve the validity of RWE studies for regulatory decision making. One objective was to identify factors that predictably increase the validity of such studies.
  • In RCT-DUPLICATE, RWE studies were designed to emulate 20 target RCTs. The regulatory-standard RCTs for replication underwent feasibility checks and quality assessments.
  • With data that are fit-for-purpose and proper design and analysis, nonrandomized RWE studies usually come to the same conclusion as the RCT about a drug’s treatment effect.
  • In any emulation, despite best efforts, there will remain differences in population, measurement, and drug use. Data fit-for-purpose and study design choices are the most important considerations for emulation success.

Discussion Themes

Initial findings of RCT-DUPLICATE identify circumstances when RWE may offer causal insights in situations where RCT data are either not available or cannot be quickly or feasibly generated.

Can this approach be used to predict results for a new entity?

It will be useful to establish a repository of case studies to increase the predictability of future RWE studies; increase the use of common methodological approaches to emulate target trials; and point out areas that are currently difficult to address with RWE.

Read more about RCT-DUPLICATE in Circulation.

Tags

#pctGR, @Collaboratory1

September 2, 2020: Chapter on Assessing Fitness for Use of Real-World Data Sources Added to the Living Textbook

Posted on by Karen Staman

The NIH Collaboratory published a new chapter of its Living Textbook of Pragmatic Clinical Trials. The chapter, “Assessing Fitness-for-Use of Real World Data Sources,” describes several approaches for determining whether real-world data are fit for their intended purpose in pragmatic clinical trials.

“Real-world data” are collected for clinical care, insurance claims, administrative purposes, registries, or are generated directly by the patient. Because these data are collected for a purpose other than a specific research project, an investigator must understand the characteristics and limitations of the data to determine whether they can be used in a pragmatic trial.

The new chapter includes the following sections:

The new chapter updates a previous resource based on work by experts in the NIH Collaboratory’s Electronic Health Records Core Working Group.