February 28, 2018: New Meeting Summary Examines How to Integrate Patient‐Reported Health Data for Pragmatic Research

A recently released summary from the ADAPTABLE Roundtable Meeting explores ways to better understand the sets of circumstances and considerations that could guide when and how to gather and integrate patient-reported health data with other data sources in pragmatic trials.

For outcomes that represent subjective experiences, such as pain, symptoms, and physical functioning, the patient is the unique and privileged source of information. Other patient-reported health data may not have a clear source of truth, such as co-morbidities and hospitalizations. In such cases, patient-reported health data may supplement, contradict, or agree with EHR and claims data. For example, medication data reported by patients might be a more accurate reflection of what patients are actually taking than medication data in the EHR, especially for over-the-counter medications.

Patient-reported health data come from various sources and can be feasibly collected in the conduct of a pragmatic clinical trial, but the optimal approaches for capturing and analyzing these data are unclear. Questions include how to integrate this information with other data collected as part of a study, including data from the EHR.

To better understand patient-reported health data and how to use them in pragmatic trials, 18 experts from 8 institutions convened at the roundtable meeting, coming from a wide variety of backgrounds including biostatistics, epidemiology, oncology, nursing, psychiatry, health policy, and regulation. Representatives from the NIH Collaboratory included Drs. Lesley Curtis and Rachel Richesson from the EHR Core and Dr. Kevin Weinfurt from the Patient-Reported Outcomes Core.

In addition to the meeting summary, two white papers are forthcoming. For more information about using patient-reported data in pragmatic trials, see the Living Textbook Chapter on Endpoints and Outcomes.

This effort was funded by Office of the Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services through a supplement provided to the NIH Collaboratory Coordinating Center.

February 22, 2017: NIH Collaboratory Launches First Embedded Pragmatic Clinical Trial Training Workshop

On February 20-21, 2018, a group of 27 clinical investigators met in Durham to learn from the NIH Collaboratory about the design and conduct of embedded pragmatic clinical trials (ePCTs). Through the experiences of the NIH Collaboratory, much has been learned about how to launch and implement successful ePCTs. The workshop’s goal was to help investigators advance their ePCT research ideas and build a larger community of researchers capable of conducting high-quality ePCTs. Participants from across the country were selected based on a rigorous application process.

“ePCTS hold the promise of an efficient and powerful way to generate evidence. We’ve learned so much through the Collaboratory, but the knowledge is not helpful if we’re not passing it along to other clinical investigators so they can contribute even more evidence to the knowledge base,” said Kevin Weinfurt, PhD, a co-principal investigator of the NIH Collaboratory’s Coordinating Center who helped develop and organize the workshop along with a planning committee.

Representatives from the Coordinating Center, Demonstration Projects, and Core Groups were on hand to provide information, guidance, and lessons learned from their experience with the NIH Collaboratory ePCTs. The topics included an introduction to ePCTs, engaging stakeholders and aligning with healthcare system partners, designing with implementation in mind, design and analytic considerations, regulatory and ethical challenges, measuring outcomes, dissemination, and ePCT team composition. Representatives from the NIH were also in attendance and presented sessions about the importance of pilot and feasibility testing and developing a compelling application for funding. During the workshop, participants completed hands-on exercises and worked through trial design challenges with the experts and their colleagues.

Wendy Weber, ND, PhD, MPH, Acting Deputy Director of the National Center for Complementary and Integrative Health (NCCIH), presenting on pilot and feasibility testing in ePCTs.

“We’re trying to change the world, and to change the world you need generalizable data and interventions that are scalable. The chasm for implementation of clinical evidence is huge, and PCTs have the capacity to take great interventions and get them to the patients so that they can have impact and improve care,” said Wendy Weber, ND, PhD, MPH, acting deputy director of the National Center for Complementary and Integrative Health (NCCIH), and one of the workshop presenters. “While local improvements can be accomplished through quality improvement activities, if you have outcomes that matter, randomizing interventions and studying them on a large scale can provide the evidence that will make a real difference in the care of people across the United States.”

Another goal of the workshop was to pilot the educational materials and collect feedback on their quality and appropriateness from both attendees and subject matter experts. The Collaboratory Coordinating Center will use the feedback to refine the content for future workshops and educational materials. The NIH Collaboratory’s Living Textbook provided a foundation for much of the workshop, and a host of new information was presented that can be used in future materials and updates to the textbook.

The training was funded as an NIH Roadmap Initiative 3U54AT007748-05S2.


February 15, 2018: Timely Tips from Demonstration Project Principal Investigators

The NIH Collaboratory will soon welcome a new group of Demonstration Projects and guide them through the piloting and implementation phases of their embedded pragmatic clinical trials (ePCTs). We asked three seasoned principal investigators—Drs. Laura Dember, MD (TiME), Lynn DeBar, PhD (PPACT), and Jerry Jarvik, MD, MPH (LIRE)­—to share tips and advice for investigators who are new to ePCTs.

Download the PDF.

“Integrate really well with the key stakeholders from your sites.”
– Lynn DeBar, PhD

“Engage with senior mentors and people who have gone down this road before.” – Jerry Jarvik, MD, MPH

“Recognize that the perspectives of key stakeholders may be different than anticipated by investigators.” – Laura Dember, MD


December 12, 2017: NIH Collaboratory Core Working Group Interviews: Reflections from the Patient-Reported Outcomes Core

We recently asked Dr. Kevin Weinfurt, Chair of the Patient-Reported Outcomes (PRO) Core, to reflect on the first 5 years of the Core’s work and the challenges ahead. He says the biggest impact of the Core has been working with national initiatives to improve inclusion of PROs in the electronic health record (EHR). Further, Core members have contributed to new knowledge through white papers and chapters in the Living Textbook. In the coming years, he’s hoping the Core will be able to identify the value proposition of PROs.

“Because there are costs associated with collecting PROs, we need to determine when PROS are essential, supporting, or not at all informative for the clinical questions. This gets at the value proposition: When are they of value and to whom?” —Dr. Kevin Weinfurt.

Read more from Dr. Weinfurt in the full interview (pdf).

December 7, 2017: Dr. Greg Simon Explains Individual, Cluster, and Stepped-Wedge Randomization in a New Prop Video

In a new video in the Living Textbook, Dr. Greg Simon describes the differences between individual, cluster, and stepped-wedge randomization using props, including marbles, Play-Doh, and glassware.

“In the end, it’s all about randomly assigning who gets which treatment, or who gets which treatment when, so that we’re able to make some un-biased judgement about which treatment is really better.” —Greg Simon, MD

November 17, 2017: New Video in Living Textbook Explores Data Sharing and Embedded Research

As part of an article published in Annals of Internal Medicine, Dr. Greg Simon created a short video in which he describes concerns related to data sharing and embedded research, as well as potential solutions for those concerns. We recently added this video to the Living Textbook chapter on Data Sharing and Embedded Research. In the chapter, the authors expand on the ideas presented in the Annals article and fame them using lessons learned from the Collaboratory’s Demonstration Projects. Data collected as part of research embedded in a health system comes from a fundamentally different context than stand-alone explanatory trials. When they are taken out of context or used for comparisons, they have the potential to do harm—something that can potentially discourage health systems from volunteering to participate in embedded research. The authors suggest that data sharing plans for embedded research be developed in partnership with health system leaders in ways that maximize the amount of data that can be shared while protecting patient privacy and healthcare system interests.

“Ultimately, it’s a practical question: if we want healthcare providers and healthcare systems to participate in research, we shouldn’t expect them to bear extra risk. In an ideal world, all information about the quality of health care and healthcare outcomes across the country would be completely open to everyone, but we don’t live in that world now. So if we are asking healthcare providers and healthcare systems to open up and be more transparent by participating in research, we certainly would not want to punish those who volunteer.” — Simon et al. in video for Ann Intern Med


Simon G, Coronado G, DeBar L, et al. Data Sharing and Embedded Research: Introduction. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. Available at: http://www.rethinkingclinicaltrials.org/data-share-top/data-sharing-and-embedded-research-introduction/. Updated November 13, 2017.

November 3, 2017: Dr. Miguel Vazquez Shares Lessons From the Improving Chronic Disease Management with Pieces (ICD-Pieces) Trial

In this interview, Dr. Miguel Vazquez gives an update on the first years of the Improving Chronic Disease Management with Pieces (ICD-Pieces) trial. Dr. Vazquez discussed the status of his trial, challenges and surprises, and advice he has for new investigators.

“Try to really learn from others who have done this—even if you are the first one doing your specific trial with your specific questions. It was helpful for us to learn from the other Collaboratory projects; they had already faced some problems, and we were able to anticipate and develop solutions proactively.” – Dr. Miguel Vazquez

Read more from Dr. Vazquez in the full interview (PDF).

October 27, 2017: Dr. Doug Zatzick Shares Lessons From the Trauma Survivors Outcomes and Support (TSOS) Trial

In this interview, Dr. Doug Zatzick gives an update on the first years of the Trauma Survivors Outcomes and Support (TSOS) trial. Dr. Zatzick discussed the status of his trial, challenges and surprises, and advice he has for new investigators.

Dr. Zatzick’s advice: “Embed implementation teams within embedded trials. The bottom line is, go to the sites, do training at the sites and with the team, and take field notes in real time. ”

Read more from Dr. Zatzick in the full interview.

October 20, 2017: Dr. Vincent Mor Shares Lessons From the Pragmatic Trial of Video Education in Nursing Homes (PROVEN) Trial

In this interview, Dr. Vincent Mor gives an update on the first years of the Pragmatic Trial of Video Education in Nursing Homes (PROVEN) project. Dr. Mor discussed the status of his trial, challenges and surprises, and advice he has for new investigators.

Dr. Mor’s advice: “The health care system must agree that the outcome your intervention is seeking to achieve is consistent with their mission. Your outcome goal should be something they care about.”

Read more from Dr. Mor in the full interview.

Recent Collaboratory Publications on Research Ethics

The American Journal of Bioethics has recently published three articles authored by members of the Regulatory/Ethics core group describing various questions related to research on medical practices:

  • Is shared decision making an appropriate analytic frame for research on medical practices (Sugarman 2015) discusses the role of shared decision making (SDM) in research on medical practices. The author cautions that “while SDM is in many ways similar to informed consent, there are some important differences, especially in the research setting.” This publication is freely accessible through PubMed Central.
  • Patients’ views concerning research on medical practices: implications for consent (Weinfurt et al. 2015) describes the results of focus group sessions that elicited a range of patients’ views and opinions about different types of research on usual medical practices. The authors state that “our data suggest that effective policy and guidance will involve balancing different patients’ interests and potentially different sets of interests for different types of research studies on usual medical practices.”
  • Ethics of research in usual care settings: data on point (Sugarman 2016) introduces a special five-article supplement in the American Journal of Bioethics, stating that the “growing empirical ethics literature regarding research in usual care settings provides data to inform conceptual and policy debates regarding this research and suggests areas that require further study.”

These publications were supported by a bioethics supplement awarded to the Regulatory/Ethics Core group by the NIH’s Office of the Director.