April 17, 2025: Researchers Illustrate Potential for Observational Studies of Real-World Data to Emulate Randomized Trials

Headshots of Dr. Xiaojuan Li and Dr. Sonal SinghResearchers applied the “target trial emulation framework” to highlight important design considerations for observational studies that use real-world data to emulate randomized clinical trials.

The work, which was supported by the NIH Pragmatic Trials Collaboratory’s Distributed Research Network and by a grant from the National Institute on Aging, was published this week in Alzheimer’s & Dementia: Translational Research & Clinical Interventions.

The US Food and Drug Administration has approved anti-amyloid beta monoclonal antibodies for the treatment of patients with early Alzheimer disease. However, the findings of the randomized trials that supported these approvals may have limited generalizability to clinical practice due to the trials’ strict eligibility criteria, limited treatment and follow-up periods, and close monitoring. Thus, little is known about the safety of anti-amyloid therapies in real-world settings.

Existing real-world data, such as information available in electronic health records and administrative claims databases, can support studies of safety and utilization outcomes. The target trial emulation framework can guide the design of such studies while minimizing the forms of bias commonly encountered in observational research.

Using the anti-amyloid therapy lecanemab as an example, the researchers described the key design and analytical considerations for observational studies intended to emulate randomized trials.

Read the full report.

Authors of the article include Xiaojuan Li, Bahareh Rasouli, Jennifer Lyons, Noelle Cocoros, and Richard Platt of Harvard Medical School and the Harvard Pilgrim Health Care Institute; and Sonal Singh, Ivan Abi-Elias, and Jerry Gurwitz of UMass Chan Medical School.

Learn more about the NIH Collaboratory’s Distributed Research Network.

March 20, 2025: iPATH Team Explores Integration of Artificial Intelligence Into Analysis of Qualitative Data

Headshot of Dr. Sara Singer
Dr. Sara Singer, principal investigator for iPATH

Researchers from iPATH, an NIH Collaboratory Trial, described key considerations for integrating artificial intelligence tools into analyses of qualitative data.

The report was posted this month on the AcademyHealth Blog.

The iPATH trial, led by principal investigator Sara Singer at Stanford University, will test the implementation of a practice transformation strategy for type 2 diabetes in federally qualified health centers in California, Massachusetts, Ohio, and Puerto Rico. In the first phase of the project, the study team is refining the strategy by conducting case studies with 12 health centers to identify organizational conditions and processes that promote or impede the effectiveness of diabetes care.

Interviews for the 12 case studies generated 170 hours of qualitative data plus related materials. The study team explored how rapidly evolving artificial intelligence tools, such as large language models, might enhance researchers’ handling of large qualitative datasets, including labor-intensive and time-consuming processes of transcription, coding, and analysis.

Read the full report.

iPATH is supported by a grant award from the National Institute on Minority Health and Health Disparities. Learn more about iPATH.

March 4, 2025: PRIM-ER Team Develops Innovative Statistical Techniques for Stepped-Wedge Trials

Cover image of Statistics in MedicineResearchers with PRIM-ER, an NIH Collaboratory Trial, published 2 innovative statistical techniques for evaluating intervention effects in stepped-wedge, cluster randomized trials. The new models, which use Bayesian methods, outperformed traditional analytic methods and other Bayesian approaches in simulations and real-world applications.

The article was published online in Statistics in Medicine.

In cluster randomized trials with stepped-wedge designs, the clusters are randomized into several groups, and all groups start the trial in the control condition. Groups of clusters cross over to the intervention condition on a staggered timeline, and all groups receive the intervention before the end of the trial.

Stepped-wedge designs can be advantageous when simultaneous rollout of the intervention to all clusters is infeasible, or when withholding the intervention from any cluster would be unethical, or when there is a risk of contamination between intervention subjects and control subjects. However, stepped-wedge designs can also introduce confounding by time, as the intervention is rolled out to clusters in waves. Temporal trends during the study can influence the study’s outcomes.

(Learn more about stepped-wedge designs in the Living Textbook.)

The PRIM-ER researchers tested 2 new Bayesian hierarchical penalized spline models to improve the estimation of intervention effects in stepped-wedge trials. The first model focuses on immediate intervention effects and accounts for large numbers of clusters and time periods. The second model extends the first by accounting for time-varying intervention effects. The researchers applied both models to data from PRIM-ER.

Read the full report.

PRIM-ER tested a multidisciplinary primary palliative care intervention in a diverse mix of emergency departments in the United States to improve the delivery of goal-directed emergency care of older adults. The study was supported by the National Institute on Aging. Learn more about PRIM-ER.

September 12, 2024: NIH Collaboratory Biostatisticians Evaluate Analytic Models for Individually Randomized Group Treatment Trials

Headshot of Dr. Jonathan Moyer
Dr. Jonathan Moyer

To avoid inflation in the rate of type 1 error, or false positives, in individually randomized group treatment (IRGT) trials, researchers should choose an analytic model that accounts for the correlations in outcome measures that arise when study participants receive an intervention from the same source, according to a report from the NIH Pragmatic Trials Collaboratory’s Biostatistics and Study Design Core.

The report was published online ahead of print in Statistics in Medicine.

Many IRGT trials randomly assign individuals to study arms but deliver the study intervention through shared “agents,” such as clinicians, therapists, or trainers. After randomization, interactions between participants who share the same agent can lead to correlations in study outcomes. The delivery agents may be nested in or crossed with study arm, and participants may interact with a single agent or multiple agents. There has been no systematic effort to identify the appropriate analytic models for these complex study designs.

To address this knowledge gap, members of the NIH Collaboratory’s Biostatistics and Study Design Core conducted a simulation study to examine the performance of a variety of analytic models for IRGT trials in which complex clustering arises from participants interacting with multiple agents or single agents in both nested and crossed designs. They found substantial inflation in the type I error rate in studies with nested designs when the analytic model did not account for participants interacting with multiple agents.

Read the full article.

This article is the latest in a series of reports completed this year by members of the Biostatistics and Study Design Core to explore analytic approaches to clinical trials with complex clustering and other novel design features:

Lead author Jonathan Moyer, a statistician in the NIH Office of Disease Prevention, led a discussion of complex clustering in pragmatic trials in a session of the NIH Collaboratory’s weekly Rethinking Clinical Trials webinar series: “The Perils and Pitfalls of Complex Clustering in Pragmatic Trials.”

Learn more about the NIH Collaboratory’s Biostatistics and Study Design Core.

January 3, 2024: Special Biostatistics Series Concludes With Missing Data in Cluster Randomized Trials

In this Friday's PCT Grand Rounds, Rui Wang of Harvard Medical School will offer the final session in our special series, Advances in the Design and Analysis of Pragmatic Clinical Trials, with "Methods for Handling Missing Data in Cluster Randomized Trials." The session will be held on Friday, January 5, at 1:00 pm eastern.

Wang is an associate professor of population medicine and the director of the Division of Biostatistics in the Department of Population Medicine at Harvard Medical School and the Harvard Pilgrim Health Care Institute. She is also an associate professor in the Department of Biostatistics at the Harvard T.H. Chan School of Public Health. She is a longtime member of the NIH Pragmatic Trials Collaboratory's Biostatistics and Study Design Core Working Group.

This session's moderator, Fan Li, is an assistant professor of biostatistics at the Yale School of Public Health.

Join the online meeting.

This special Grand Rounds series includes moderated webinar discussions that bring together biostatisticians, clinical trials methodologists, and investigators to discuss challenges and share lessons learned in the design, implementation, and analysis of pragmatic trials. Download the series flyer and see the full schedule below, including archived webinar recordings and slides from previous sessions.

All sessions are free and open to the public. No registration is required.

November 29, 2023: Special Biostatistics Series Continues With Guidelines for Stepped-Wedge Trials

In this Friday’s PCT Grand Rounds, Jim Hughes of the University of Washington will continue our special series, Advances in the Design and Analysis of Pragmatic Clinical Trials, with his presentation, “Guidelines for Design and Analysis of Stepped-Wedge Trials.” The session will be held on Friday, December 1, at 1:00 pm eastern.

Hughes is a professor emeritus of biostatistics at the University of Washington. This session’s moderator, Patrick Heagerty, is a professor of biostatistics at the University of Washington and a cochair of the NIH Pragmatic Trials Collaboratory’s Biostatistics and Study Design Core.

Join the online meeting.

This special Grand Rounds series will include additional moderated webinar discussions that bring together biostatisticians, clinical trials methodologists, and investigators to discuss challenges and share lessons learned in the design, implementation, and analysis of pragmatic trials. Download the series flyer and see the full schedule below.

All sessions are free and open to the public; no registration is required.

March 8, 2023: Biostatistics Core Sponsors This Week’s PCT Grand Rounds on Estimands in Cluster Randomized Trials

Headshot of Brennan KahanIn this Friday’s PCT Grand Rounds, Brennan Kahan of University College London will present “Estimands in Cluster-Randomized Trials: Choosing Analyses That Answer the Right Question.” This session is sponsored by the NIH Pragmatic Trials Collaboratory’s Biostatistics and Study Design Core Working Group.

The Grand Rounds session will be held on Friday, March 10, 2023, at 1:00 pm eastern.

Kahan is a senior research fellow in the Institute of Clinical Trials and Methodology at University College London.

Join the online meeting.

March 30, 2022: Two Weights Make a Wrong: New Article From the Biostatistics and Study Design Core

Contemporary Clinical TrialslsIn a new article from the NIH Pragmatic Trials Collaboratory Biostatistics and Study Design Core, the authors share analytic considerations for cluster randomized trials with hierarchical nesting of participants within clusters. The authors illustrate the problem using theoretical derivations, a simulation study, and data from the STOP CRC NIH Collaboratory Trial as an example.

“We conclude that an analysis using both an exchangeable working correlation matrix and weighting by inverse cluster size, which may be considered the natural analytic approach, can lead to incorrect results. That is, two weights make a wrong. The bias is minimal when there is homogeneity of treatment effects according to cluster size but unacceptable when there is heterogeneity of treatment effects according to cluster size. In addition, we show that only an analysis with an independence working correlation matrix and weighting by inverse cluster size always provides valid results for the UATE [unit average treatment effect] estimand.”

Read the full article.

December 16, 2021: NIH Collaboratory Publishes COVID-19 Checklist for Statistical Analysis Plans in Pragmatic Trials

Thumbnail image of the COVID-19 checklistA new tool from the NIH Collaboratory assists investigators in identifying impacts of the COVID-19 public health emergency on ongoing pragmatic clinical trials. The Statistical Analysis Plan Checklist for Addressing COVID-19 Impacts summarizes impacts on trial conduct that study teams should document, measure, analyze, and report.

The new checklist was developed by the NIH Collaboratory’s Biostatistics and Study Design Core Working Group. Since the beginning of the COVID-19 pandemic, many of the NIH Collaboratory Trials have had to postpone recruitment, alter methods of participant engagement, and modify tools for research assessment and intervention delivery.

The leaders of the Biostatistics Core, Dr. Patrick Heagerty and Dr. Liz Turner, spoke in a recent interview about the impacts of the pandemic on the NIH Collaboratory Trials. Early next year, the Coordinating Center will report the results of a survey of the study teams about their experiences with these impacts.

Download the Statistical Analysis Plan Checklist for Addressing COVID-19 Impacts.

April 16, 2021: Minnesota EHR Consortium COVID-19 Project: A Statewide Collaboration to Inform Vaccine Equity (Paul E. Drawz, MD, MHS, MS; Tyler Winkelman, MD, MSc)

Speakers

Paul E. Drawz, MD, MHS, MS
Associate Professor
Division of Renal Disease and Hypertension
University of Minnesota

Tyler N.A. Winkelman, MD, MSc
Co-Director, Health, Homelessness, and Criminal Justice Lab
Associate Director, Virtual Data Warehouse
Hennepin Healthcare Research Institute

Topic

Minnesota EHR Consortium COVID-19 Project: A Statewide Collaboration to Inform Vaccine Equity

Keywords

COVID-19; Electronic health records (EHRs); Data analysis; Research consortium; Healthcare systems; Population health; Distributed data network; Vaccine equity

Key Points

  • The EHR Consortium’s COVID-19 vaccine project aims to inform policy and practice through data-driven collaboration among members of Minnesota’s health care community.
  • The collaborative network can monitor population-level health metrics and analyze changes over time using aggregations of data to inform public health policy. Sources of data include EHRs, census data, state-wide electronic immunization records, and population data.
  • The COVID-19 vaccine dashboard is updated weekly and provides data at the ZIP level by age categories and race/ethnicity.
  • Minnesotans who have received a COVID-19 vaccine (any source) and had a visit at a consortium site in the last 10 years (~90 percent of the state population) are reflected in the dashboard.

Discussion Themes

How were you able to convene this consortium during a pandemic year?

Was your hashing algorithm home-grown or did you have an outside partner?

In the future, this infrastructure will be expanded to incorporate smaller health systems and additional content expertise around comorbidities, disease prevalence, and identification of disparities in near real-time.

Read more about the MN EHR Consortium at Hennepin Healthcare and the University of Minnesota Clinical & Translational Science Institute.

Tags

#pctGR, @Collaboratory1