An updated template from the NIH Pragmatic Trials Collaboratory provides guidance for the transparent reporting of the primary results of pragmatic clinical trials.
The template includes elements from the Consolidated Standards of Reporting Trials (CONSORT) statement and its extensions. It also addresses secondary use of electronic health record data, involvement of research partners and healthcare systems in the conduct of pragmatic trials, and special ethical and regulatory considerations.
The updated template is organized by the recommended reporting elements presented in the CONSORT checklist and draws on recent experiences and lessons learned from the NIH Collaboratory Trials. Appendices include links to CONSORT and its relevant extensions, the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS-2) tools and resources, and examples of figures to include in pragmatic trial reports.
Harlan M. Krumholz, MD, SM Harold H. Hines, Jr. Professor of Medicine and Public Health Yale University
Joseph S. Ross, MD, MHS Professor of Medicine and Public Health Yale University
Topic
Is It Time to Embrace Preprints? A Conversation About the First 18 Months of medRxiv
Keywords
Preprints; Preprint server; medRxiv; Open science; Health science research; Research transparency; Preliminary research reports
Key Points
A preprint is a research manuscript yet to be certified by peer review and accepted for publication by a journal. A preprint server, like medRxiv, is an online platform dedicated to the distribution of preprints.
Server submission requirements for authors help to mitigate concerns about preprints. These include clear posting criteria (ie, original research articles only), an established screening process, and a caution to users of preprints, including researchers, journalists, and the public, that states: “Preprints are preliminary reports of work that have not been peer-reviewed. They should not be relied on to guide clinical practice or health-related behaviors and should not be reported in news media as established information.”
Not allowed are commentaries, editorials, opinion pieces or essays, letters to editors, narrative reviews, medical-legal research, and case reports.
Discussion Themes
The concept of “living data” or “living analyses” has grown out of the pandemic crisis and could stay on as a feature of scientific communication.
How do you think the public conversation around a preprint may positively or negatively impact the peer review process itself?
How are academic institutions acknowledging preprints in the sense of “evidence of productivity” (as for academic promotion)?
Preprints can serve as a teaching opportunity not only for reminding scientists to be discerning readers of reported science, but also for reminding the media.
NIH Collaboratory investigators Drs. Greg Simon, Rachel Richesson, and Adrian Hernandez published an opinion piece in Healthcare arguing that clinical trials investigators should align their dissemination processes with industry-sponsored trials to favor speed, and that years-long delays in dissemination reduce the relevance of clinical research.
“Delays reduce the ability for researchers to apply trial findings to new research questions, impede clinicians from having the most up-to-date information, and perhaps most importantly, are a disservice to patients who could benefit from the information.”
The authors use experiences with pragmatic trials supported by the NIH Collaboratory to explore faster dissemination of results, and suggest the following solutions:
Real-time access to outcome data
Continuous data curation and cleaning
Immediate data analysis
Rapid reporting of trial results
Much change is needed to reach these goals. The authors suggest that by modeling processes after industry-sponsored trials, researchers may be able to improve the speed and quality of results reporting.
“Cultural incentives are aligned in industry sponsored trials to favor speed: readiness for generalizing topline results is considered valuable to shareholders, and the culture encourages a system where data are liquid, available, and continuously cleaned and curated, such that topline results can be reported within a timespan of two weeks rather than two years.”
As part of the NIH Collaboratory’s commitment to dissemination and sharing, all NIH Collaboratory Trials are expected to share data and resources, and topline results are reported in our weekly Grand Rounds Webinars.
Real-World Evidence for Drug Effectiveness Evaluation: Addressing the Credibility Gap
Keywords
Real-world evidence; Non-interventional studies; Health economics; ISPOR; Transparency; Reproducibility
Key Points
ISPOR is an international, multistakeholder nonprofit dedicated to advancing health economics and outcomes research excellence to improve decision making for health globally.
Key characteristics of credible and useful real-world evidence include:
Careful data collection or curation
Appropriate analytic methods
Good procedural practices for transparent study process
Replicability and reproducibility
Informed interpretation
Fit-for-purpose application
For transparency, it is recommended that researchers declare their study to be an exploratory (hypothesis evaluation) study and post the study protocol and analysis plan on a public study registration site prior to conducting the study analysis.
Guiding principles of open science include appropriate access to research information; proper oversight with minimum barriers to data access; maintaining utility of data; an expectation that results of shared data will similarly be shared; and acknowledgment of those who contribute original data.
Despite efforts at supporting open science, no academic institution has an open science policy yet.
Discussion Themes
Open science remains an important goal to build trust and expand knowledge.
Data sharing is not a traditional measure of academic success. What incentives would need to change in order to support open science?
Harlan M. Krumholz, MD, SM
Harold H. Hines, Jr. Professor of Medicine and Public Health
Yale University
Joseph S. Ross, MD, MHS
Associate Professor of Medicine and Public Health
Yale University
Topic
medRxiv: A Paradigm Shift in Disseminating Clinical and Public Health Research
Keywords
Open science; Clinical research dissemination; Preprints; medRxiv preprint server
Key Points
medRxiv (med archive) is a server for health science preprints. It is a free service to the research community, managed in partnership with BMJ and Yale.
Benefits of preprints in medicine include early sharing of new information; enabling less “publishable” studies to be more readily available; and facilitating replication and reproducibility studies.
medRxiv submissions require:
Following ICMJE guidance, including author names, contact info, affiliation
Funding and competing interests statements
Statement of IRB or ethics committee approval
Study registration (ClinicalTrials.gov or other ICMJE approved registry for trials, PROSPERO for reviews) or link to protocol
Data sharing availability statement
EQUATOR Network reporting guidelines checklists
The medRxiv preprint server urges caution in using and reporting preprints, and includes language explaining that preprints are preliminary reports of work that have not been peer-reviewed, should not be relied on to guide clinical practice or health-related behaviors, and should not be reported in news media as established information.
Discussion Themes
Preprint servers do not replace, but rather complement, peer review.
Preprint has the potential for being a vehicle for high-quality but “negative” results. If we teach students that a negative result is also a good result, providing an avenue for us to walk-the-talk more easily via open communication seems largely positive despite the limitations.
A recent study published in BMC Medicine found that many randomized controlled trials (RCTs) self-labeled as “pragmatic” were actually explanatory in nature, in that they assessed investigational medicines compared with placebo to test efficacy before licensing. Of the RCTs studied, one-third were pre-licensing, single-center, or placebo-controlled trials and thus not appropriately described as pragmatic.
Appropriately describing the design and characteristics of a pragmatic trial helps readers understand the trial’s relevance for real-world practice. The authors explain that RCTs suitably termed pragmatic compare the effectiveness of 2 available medicines or interventions prescribed in routine clinical care. The purpose of such pragmatic RCTs is to provide real-world evidence for which interventions should be recommended or prioritized.
The authors recommend that investigators use a standard tool, such as the CONSORT Pragmatic Trials extension or the PRECIS-2 tool, to prospectively evaluate the pragmatic characteristics of their RCTs. Use of these tools can also assist funders, ethics committees, and journal editors in determining whether an RCT has been accurately labeled as pragmatic.
The BMC Medicine article cites NIH Collaboratory publications by Ali et al. and Johnson et al., as well as the Living Textbook, in its discussion of pragmatic RCTs and the tools available to assess their relevance for real-world practice.
“Submissions of RCTs to funders, research ethics committees, and peer-reviewed journals should include a PRECIS-2 tool assessment done by the trial investigators. Clarity and accuracy on the extent to which an RCT is pragmatic will help [to] understand how much it is relevant to real-world practice.” (Dal-Ré et al. 2018)
An updated template from the NIH Pragmatic Trials Collaboratory provides guidance for the transparent reporting of the primary results of pragmatic clinical trials.