data quality

July 3, 2023: Report Shares Strategies for Addressing Bias and Lack of Generalizability of EHR Data

Posted on by Damon Seils

JAMIA cover imagePragmatic research is vulnerable to biases due to differences in data capture and access to care for different subsets of the population, which, if left unaddressed, can propagate inequities in health and the healthcare system.

In a new article published online ahead of print in JAMIA, the NIH Pragmatic Trials Collaboratory’s Trial teams reflect on the health equity challenges encountered by their trials and share the specific strategies they used to mitigate sources of potential bias and increase the generalizability of research results.

“Biased results and poor generalizability can occur because detailed information about specific populations is missing, and critically, is missing not at random: these data are disproportionately missing in diverse and underserved populations,” the authors cautioned.

The NIH Collaboratory Trials are implementing approaches designed to detect and mitigate sources of bias, to ensure inclusion and retention of underrepresented populations, and to enable the complete collection of data that can help identify and support measurement of health inequities.

“By improving data capture, access to care, and patient technology support, ePCTs hold the potential to yield insights and estimates pertinent to the entire population, not just a subset of the population,” they wrote.

This work was a collaboration between the Health Equity Core, the EHR Core, and the Patient-Centered Outcomes Core of the NIH Pragmatic Trials Collaboratory.

August 31, 2022: New Section of Living Textbook Addresses Evaluating Fitness for Use of Real-World Data Sources

Posted on by Karen Staman

A new section of the NIH Pragmatic Trials Collaboratory’s Living Textbook of Pragmatic Clinical Trials discusses challenges associated with Evaluating Fitness for Use of real-world data for trials. The section uses a case study from the Harmony Outcomes EHR Ancillary Study (eHARMONY) to describe lessons learned and to provide recommendations for studies considering incorporating real-word data as a data source.

Among the lessons learned were:

  • Standalone clinical research sites had very little extractable EHR data about patients.
  • Most lab results and medications were either not extractable or not mapped to a useful terminology.
  • Many sites did not have the ability to transform their data into a common format and had to send rudimentary data extracts to the ancillary study coordinating center. Sites participating in other research networks, such as PCORnet, had no difficulty with this task.

For more, read the chapter or watch the Grand Rounds presentation Leveraging RWD in a Multinational Trial: Results from the other eHARMONY 

July 14, 2022: Grand Rounds Podcast Now Available, Featuring Dr. John Concato on FDA Draft Guidance on Real-World Evidence

Posted on by terren.green@duke.edu

Headshot of Dr. John ConcatoIn the latest episode of the NIH Collaboratory Grand Rounds podcast, Dr. John Concato and Dr. Richard Platt continue their discussion about the FDA draft guidance on real-world evidence.

The full June 24 Grand Rounds webinar with Dr. Concato is also available.

Podcast July 8, 2022: FDA Draft Guidance on Real-World Evidence (John Concato, MD, MS, MPH)

Posted on by terren.green@duke.edu

This podcast continues the discussion with Dr. John Concato as he discusses the FDA draft guidance on real-word evidence. Click on the recording below to listen to the podcast.

Want to hear more? View the full Grand Rounds presentation.

For alerts about new episodes, subscribe free on Apple Podcasts or SoundCloud. Read the transcript.

June 24, 2022: FDA Draft Guidance on Real-World Evidence (John Concato, MD, MS, MPH)

Posted on by terren.green@duke.edu

Speaker

John Concato, MD, MS, MPH
Associate Director for Real-World Evidence Analytics
Office of Medical Policy (OMP)
Center for Drug Evaluation and Research (CDER)
Food and Drug Administration (FDA)

 

 

Keywords

Big data; Real-word evidence; Real-world data; 21st Century Cures Act; FDA Draft Guidance

 

Key Points

  • Big Data, a term first used in the 1990s, leverages modern technology to increase the quantity, forms, speed, and capability to manipulate large-scale data. Real-world data (RWD) is a term with specific regulatory implications referring to health care data routinely collected from a variety of sources. Real-world evidence (RWE) is clinical evidence derived from analysis of RWD regardless of study design.
  • Terminology is important in research work, and we should strive to be as precise as possible with the terminology we use.
  • With the 21st Century Cures Act of 2016, the FDA established a program to evaluate the potential use of real-world evidence to support new indications for drugs and satisfy post-approval study requirements.
  • In 2021, the FDA issued 4 draft guidance documents for Real-world data and Real-world evidence intended to guide the selection and management of data sources to appropriately address the study question and support decision-making for drug and biological products.

Discussion Themes

– Could real-world data sources be certified and preclude the need for submission of source data on a study specific basis? From the FDA point-of-view, while reliability can be more readily evaluated and would tend to be more stable, the relevance to a particular study could not be determined as easily.

– While there can be a reflex that says we can never be sure about major confounding, it should not be the miasma of the 21st century. A thoughtful approach that considers the characteristics that matter is the best approach.

 

Read Dr. Concato’s publication Randomized, observational, interventional, and real-world—What’s in a name? and the FDA Draft Guidance for RWD/RWE.

Tags

#pctGR, @Collaboratory1

October 5, 2021: New Article Identifies Challenges and Prerequisites for Using Electronic Health Record Systems for Pragmatic Research

Posted on by Karen Staman

JAMIA Cover

In a new NIH Collaboratory study, 20 NIH Collaboratory Trials responded to a survey about challenges encountered when using the electronic health record (EHR) for pragmatic clinical research. The goal of the study was to elucidate challenges and develop solutions—or prerequisites for pragmatic research—to enable healthcare system leaders, policy makers, and EHR designers to improve the national capacity for generating real-world evidence.

The article was published in the Journal of American Medical Informatics Association (JAMIA).

The challenges identified by the projects fell into 6 broad themes, including inadequate collection of patient-centered data, lack of functionality for structured data collection, lack of standardization, lack of resources to support customization, difficulties aggregating data from multiple sites, and difficult and inefficient access to EHR data.

Researchers from the NIH Collaboratory’s EHR Core and colleagues from the Patient-Centered Outcomes and the Health Care Systems Interactions Core Working Groups discussed the issues and iterated possible solutions. The authors developed the following prerequisites for the conduct of pragmatic research:

  • Integrate collection of patient-centered data into EHR systems
  • Facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows
  • Support creation of high-quality research data by using standards
  • Ensure adequate IT staff to support embedded research
  • Create aggregate, multidata type resources for multisite trials
  • Create reusable and automated queries

The authors argue for the ability to tailor EHR systems to enable the collection of patient-centered outcomes and the extraction of high-quality, standardized data. Although the primary uses of the data are for clinical care and billing, high-quality data from the EHR also have the potential to improve clinical care and population health by providing reliable evidence and to support pragmatic research and learning within and across healthcare systems.

Read the full article.

This work was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. This work was also supported by the NIH through the NIH HEAL Initiative under award number U24AT010961.

 

February 26, 2021: Calibrating Real-World Evidence Against RCT Evidence: Early Learnings from RCT-DUPLICATE (Sebastian Schneeweiss, MD, ScD)

Posted on by Gina Uhlenbrauck

Speaker

Sebastian Schneeweiss, MD, ScD
Chief, Division of Pharmacoepidemiology and Pharmacoeconomics
Department of Medicine, Brigham and Women’s Hospital
Professor of Medicine, Harvard Medical School
Professor in Epidemiology, Harvard T.H. Chan School of Public Health

Topic

Calibrating Real-World Evidence Against RCT Evidence: Early Learnings from RCT-DUPLICATE

Keywords

Real-world evidence (RWE); Randomized controlled trials (RCTs); Epidemiology; Emulation; Fit-for-purpose data

Key Points

  • While RCTs are an accepted research study design to establish the efficacy of medical products, RWE studies can complement the evidence generated by RCTs, as well as expand the line of inquiry around population, endpoints, treatment patterns, and comparators.
  • The RCT-DUPLICATE study aimed to understand and improve the validity of RWE studies for regulatory decision making. One objective was to identify factors that predictably increase the validity of such studies.
  • In RCT-DUPLICATE, RWE studies were designed to emulate 20 target RCTs. The regulatory-standard RCTs for replication underwent feasibility checks and quality assessments.
  • With data that are fit-for-purpose and proper design and analysis, nonrandomized RWE studies usually come to the same conclusion as the RCT about a drug’s treatment effect.
  • In any emulation, despite best efforts, there will remain differences in population, measurement, and drug use. Data fit-for-purpose and study design choices are the most important considerations for emulation success.

Discussion Themes

Initial findings of RCT-DUPLICATE identify circumstances when RWE may offer causal insights in situations where RCT data are either not available or cannot be quickly or feasibly generated.

Can this approach be used to predict results for a new entity?

It will be useful to establish a repository of case studies to increase the predictability of future RWE studies; increase the use of common methodological approaches to emulate target trials; and point out areas that are currently difficult to address with RWE.

Read more about RCT-DUPLICATE in Circulation.

Tags

#pctGR, @Collaboratory1

September 2, 2020: Chapter on Assessing Fitness for Use of Real-World Data Sources Added to the Living Textbook

Posted on by Karen Staman

The NIH Collaboratory published a new chapter of its Living Textbook of Pragmatic Clinical Trials. The chapter, “Assessing Fitness-for-Use of Real World Data Sources,” describes several approaches for determining whether real-world data are fit for their intended purpose in pragmatic clinical trials.

“Real-world data” are collected for clinical care, insurance claims, administrative purposes, registries, or are generated directly by the patient. Because these data are collected for a purpose other than a specific research project, an investigator must understand the characteristics and limitations of the data to determine whether they can be used in a pragmatic trial.

The new chapter includes the following sections:

The new chapter updates a previous resource based on work by experts in the NIH Collaboratory’s Electronic Health Records Core Working Group.

August 28, 2020: Designing, Conducting, Monitoring, and Analyzing Data from Pragmatic Clinical Trials: Proceedings from a Multi-Stakeholder Think Tank Meeting (Trevor Lentz, PT, PhD, MHA; Lesley Curtis, PhD; Frank Rockhold, PhD)

Posted on by Gina Uhlenbrauck

Speakers

Trevor Lentz, PT, PhD, MHA
Assistant Professor in Orthopaedic Surgery
Duke Clinical Research Institute

Lesley Curtis, PhD
Chair and Professor, Department of Population Health Sciences
Duke University School of Medicine

Frank Rockhold, PhD, ScM, FASA, FSCT
Professor of Biostatistics and Bioinformatics
Duke Clinical Research Institute

Topic

Designing, Conducting, Monitoring, and Analyzing Data from Pragmatic Clinical Trials: Proceedings from a Multi-Stakeholder Think Tank Meeting

Keywords

Pragmatic clinical trials; Think tank; Risk-based monitoring; Data quality; Real-world data; Electronic health records

Key Points

  • Pragmatism in study design is not a binary concept: some trial elements are purely explanatory (to establish efficacy in ideal settings) and some elements are purely practical (to establish effectiveness in the real world). The study design must serve the research question.
  • Findings from the think tank discussions on best practices and actionable steps included:
    • Ask precise research questions, and select the appropriate degree of pragmatism.
    • Optimize data quality through study design.
    • Focus on primary endpoints in data capture to maximize likelihood of success.
    • Innovate on mechanisms for data capture.
    • Promote adherence to the study protocol.
    • Evolve trial operations staff to focus on data science and informatics.
    • Share learning experiences openly and widely.

Discussion Themes

There is a misconception that PCTs, because they pursue pragmatism, are less rigorous and conducted without proper oversight or adherence to a protocol. Quality by design and good clinical practice principles apply equally to PCTs.

Risk-based monitoring is a potentially dynamic system that could improve study safety and quality, and make better use of study resources.

There is great interest from regulators, sponsors, and the academic research community to move PCT methods forward. To achieve this, we need to see more examples of successful PCTs in a context of regulatory decision-making.

Read the proceedings from the think tank meeting published in Therapeutic Innovation & Regulatory Science.

Tags

#pctGR, @Collaboratory1

August 27, 2020: Chapter on Acquiring Real-World Data Added to the Living Textbook

Posted on by Karen Staman

The NIH Collaboratory this week published a new chapter of its Living Textbook of Pragmatic Clinical Trials. The chapter, “Acquiring Real-World Data,” outlines strategies for obtaining real-world data for use in research.

“Real-world data” include data relating to the health status of a patient or the delivery of healthcare services. Common sources include electronic health records (EHRs), administrative claims, patient-reported outcomes, patient-generated health data, medical product and device registries, and databases relating to environmental factors or social determinants of health. Real-world data can support a number of activities in pragmatic clinical trials, such as patient identification and recruitment, monitoring of outcomes, and ascertainment of endpoints.

The new chapter includes the following sections:

The new chapter updates a previous resource, one of the most popular on the Living Textbook, based on work by experts in the NIH Collaboratory’s Electronic Health Records Core Working Group.