Guiding principles of open science include appropriate access to research information; proper oversight with minimum barriers to data access; maintaining utility of data; an expectation that results of shared data will similarly be shared; and acknowledgment of those who contribute original data.
Despite efforts at supporting open science, no academic institution has an open science policy yet.
Open science remains an important goal to build trust and expand knowledge.
Data sharing is not a traditional measure of academic success. What incentives would need to change in order to support open science?
Harlan M. Krumholz, MD, SM
Harold H. Hines, Jr. Professor of Medicine and Public Health
Joseph S. Ross, MD, MHS
Associate Professor of Medicine and Public Health
medRxiv: A Paradigm Shift in Disseminating Clinical and Public Health Research
Open science; Clinical research dissemination; Preprints; medRxiv preprint server
medRxiv (med archive) is a server for health science preprints. It is a free service to the research community, managed in partnership with BMJ and Yale.
Benefits of preprints in medicine include early sharing of new information; enabling less “publishable” studies to be more readily available; and facilitating replication and reproducibility studies.
medRxiv submissions require:
Following ICMJE guidance, including author names, contact info, affiliation
Funding and competing interests statements
Statement of IRB or ethics committee approval
Study registration (ClinicalTrials.gov or other ICMJE approved registry for trials, PROSPERO for reviews) or link to protocol
Data sharing availability statement
EQUATOR Network reporting guidelines checklists
The medRxiv preprint server urges caution in using and reporting preprints, and includes language explaining that preprints are preliminary reports of work that have not been peer-reviewed, should not be relied on to guide clinical practice or health-related behaviors, and should not be reported in news media as established information.
Preprint servers do not replace, but rather complement, peer review.
Preprint has the potential for being a vehicle for high-quality but “negative” results. If we teach students that a negative result is also a good result, providing an avenue for us to walk-the-talk more easily via open communication seems largely positive despite the limitations.
A recent study published in BMC Medicine found that many randomized controlled trials (RCTs) self-labeled as “pragmatic” were actually explanatory in nature, in that they assessed investigational medicines compared with placebo to test efficacy before licensing. Of the RCTs studied, one-third were pre-licensing, single-center, or placebo-controlled trials and thus not appropriately described as pragmatic.
Appropriately describing the design and characteristics of a pragmatic trial helps readers understand the trial’s relevance for real-world practice. The authors explain that RCTs suitably termed pragmatic compare the effectiveness of 2 available medicines or interventions prescribed in routine clinical care. The purpose of such pragmatic RCTs is to provide real-world evidence for which interventions should be recommended or prioritized.
The authors recommend that investigators use a standard tool, such as the CONSORT Pragmatic Trials extension or the PRECIS-2 tool, to prospectively evaluate the pragmatic characteristics of their RCTs. Use of these tools can also assist funders, ethics committees, and journal editors in determining whether an RCT has been accurately labeled as pragmatic.
The BMC Medicine article cites NIH Collaboratory publications by Ali et al. and Johnson et al., as well as the Living Textbook, in its discussion of pragmatic RCTs and the tools available to assess their relevance for real-world practice.
“Submissions of RCTs to funders, research ethics committees, and peer-reviewed journals should include a PRECIS-2 tool assessment done by the trial investigators. Clarity and accuracy on the extent to which an RCT is pragmatic will help [to] understand how much it is relevant to real-world practice.” (Dal-Ré et al. 2018)
On January 20, 2016, the International Committee of Medical Journal Editors (ICMJE) published an editorial in 14 major medical journals in which they propose that clinical researchers must agree to share the deidentified data set used to generate results (including tables, figures, and appendices or supplementary material) as a condition of publication in one of their member journals no later that six months after publication. By changing the requirements for manuscripts they will consider for publication, they aim to ensure reproducibility (independent confirmation of results), foster data sharing, and enhance transparency. To meet the new requirements, authors will need to include a plan for data sharing as a component of clinical trial registration that includes where the data will be stored and a mechanism for sharing the data.
Evolving Standards for Data Reporting and Sharing
As early as 2003, the National Institutes of Health published a data sharing policy for research funded through the agency, stipulating that “Data should be made as widely and freely available as possible while safeguarding the privacy of participants, and protecting confidential and proprietary data.” Under this policy, federally funded studies receiving over $500,000 per year were required to have a data sharing plan that describes how data will be shared, that shared data be available in a usable form for some extended period of time, and that the least restrictive method for sharing of research data is used.
In 2007, Congress enacted the Food and Drug Administration Amendments Act. Section 801 of the Act requires study sponsors to report certain kinds of clinical trial data within a specified interval to the ClinicalTrials.gov registry, where it is made available to the public. Importantly, this requirement applied to any study classified as an “applicable clinical trial” (typically, an interventional clinical trial), regardless of whether it was conducted with NIH or other federal funding or supported by industry or academic funding. However, recent academic and journalistic investigations have demonstrated that overall compliance with FDAAA requirements is relatively poor.
In 2015, the Institute of Medicine (now the National Academy of Medicine) published a report that advocates for responsible sharing of clinical trial data to strengthen the evidence base, allow for replication of findings, and enable additional analyses. In addition, these efforts are being complemented by ongoing initiatives aimed at widening access to clinical trial data and improving results reporting, including the Yale University Open Data Access project (YODA), the joint Duke Clinical Research Institute/Bristol-Myers Squibb Supporting Open Access to clinical trials data for Researchers initiative (SOAR), and the international AllTrials project.
Responses to the Draft ICMJE Policy
The ICMJE recommendations are appearing in the midst of a growing focus on issues relating to the integrity of clinical research, including reproducibility of results, transparent and timely reporting of trial results, and facilitating widespread data sharing, and the release of the draft policy is amplifying ongoing national and international conversations taking place on social media and in prominent journals. Although many researchers and patient advocates have hailed the policy as timely and needed, others have expressed concerns, including questions about implementation and possible unforeseen consequences.
The ICMJE is welcoming feedback from the public regarding the draft policy at www.icmje.org and will continue to collect comments through April 18, 2016.
A new article published in the journal Trials provides a look at how the Pragmatic–Explanatory Continuum Indicator Summary, or PRECIS, rating system can be applied to clinical trials designs in order to examine where a given study sits on the spectrum of explanatory versus pragmatic clinical trials.
The PRECIS-2 criteria are used to rate study designs as more or less “pragmatic” according to multiple domains that include participant eligibility, recruitment methods, setting, organization, analysis methods, primary outcomes, and more. In this context, “pragmatic” refers to trials that are designed to study a therapy or intervention in a “real world” setting similar or identical to the one in which the therapy will actually be used. Pragmatic trials stand in contrast to explanatory trials, which are typically designed to demonstrate the safety and efficacy of an intervention under highly controlled conditions and in carefully selected groups of participants, but which may also be difficult to generalize to larger or more varied populations.
Clinical trials are almost never wholly “explanatory” or wholly “pragmatic.” Instead, many studies exist somewhere on a spectrum between these two categories. However, understanding how these different attributes apply to trials can help researchers design studies that are optimally fit for purpose, whether that purpose is to describe a biological mechanism (as in an explanatory trial) or to show how effective an intervention is when used across a broad population of patients (as in a pragmatic trial).
In their article in Trials, authors Karin Johnson, Gila Neta, and colleagues applied PRECIS-2 criteria to 5 pragmatic clinical trials (PCTs) being conducted through the NIH Collaboratory. Each trial was found to rate as “highly pragmatic” across the multiple PRECIS-2 domains, highlighting the tool’s potential usefulness in guiding decisions about study design, but also revealing a number of challenges in applying it and interpreting the results.
Study authors Johnson and Neta will be discussing their findings during the NIH Collaboratory’s Grand Rounds on Friday, January 22, 2016 (an archived version of the presentation will be available the following week).
Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, Rundell S, Smith DH, Liu B, Taplin S, Stoney CM, Farrell MM, Glasgow RE. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17(1):32. doi: 10.1186/s13063-016-1158-y. PMID: 26772801. PMCID: PMC4715340.
You can read more about the NIH Collaboratory PCTs featured as part of this project at the following links:ABATE (Active Bathing to Eliminate Infection)
LIRE (A pragmatic trial of Lumbar Image Reporting with Epidemiology)
PPACT (Collaborative Care for Chronic Pain in Primary Care)
STOP-CRC (Strategies & Opportunities to Stop Colon Cancer in Priority Populations)
TIME (Time to Reduce Mortality in End-Stage Renal Disease)
As part of a project that examined the degree to which sponsors of clinical research are complying with federal requirements for the reporting of clinical trial results, the Clinical Trials Transformation Initiative (CTTI) and the authors of the study are making the primary dataset used in the analysis available to the public. The full analysis dataset, study variables, and data definitions are available as Excel worksheets from the CTTI website and on the Living Textbook’s Tools for Research page.
A new analysis of data from the ClinicalTrials.gov website shows that despite federal laws requiring the public reporting of results from clinical trials, most research sponsors fail to do so in a timely fashion—or, in many cases, at all. The study, published in the March 12, 2015 issue of the New England Journal of Medicine, was conducted by researchers at Duke University and supported by the NIH Collaboratory and the Clinical Trials Transformation Initiative (CTTI). The study’s authors examined trial results as reported to ClinicalTrials.gov and evaluated the degree to which research sponsors were complying with a federal law that requires public reporting of findings from clinical trials of medical products regulated by the U.S. Food and Drug Administration (FDA).
“We thought it would be a great idea to see how compliant investigators are with results reporting, as mandated by law,” said lead author Dr. Monique Anderson, a cardiologist and assistant professor of medicine at Duke University.
Using a publicly available database developed and maintained at Duke by CTTI, the authors were able to home in on trials registered with ClinicalTrials.gov that were highly likely to have been conducted within a 5-year study window and to be subject to the Food and Drug Administration Amendments Act (FDAAA). This federal law, which was enacted in 2007, includes provisions that obligate sponsors of non-phase 1 clinical trials testing medical products to report study results to ClinicalTrials.gov within 12 months of the trial’s end. It also describes allowable exceptions for failing to meet that timeline.
However, when the authors analyzed the data, they found that relatively few studies overall—just 13 percent—had reported results within the 12-month period prescribed by FDAAA, and less than 40 percent had reported results at any time between the enactment of FDAAA and the 5-year benchmark.
“We were really surprised at how untimely the reporting was—and that more than 66 percent hadn’t reported at all over the 5 years [of the study interval],” said Dr. Anderson, noting that although prior studies have explored the issue of results reporting, they have until now been confined to examinations of reporting rates at 1 year.
Another unexpected result was the finding that industry-sponsored studies were significantly more likely to have reported timely results than were trials sponsored by the National Institutes of Health (NIH) or by other academic or government funding sources. The authors noted that despite a seemingly widespread lack of compliance with both legal and ethical imperatives for reporting trial results, there has so far been no penalty for failing to meet reporting obligations, even though FDAAA spells out punishments that include fines of up to $10,000 per day and, in the case of NIH-sponsored trials, loss of future funding.
“Academia needs to be educated on FDAAA, because enforcement will happen at some point. There’s maybe a sense that ‘this law is for industry,’ but it applies to everyone,” said Anderson, who points out that this study is being published just as the U.S. Department of Health and Human Services and the NIH are in the process of crafting new rules that deal specifically with ensuring compliance with federal reporting laws.
According to Anderson, increased awareness of the law, coupled with stepped-up enforcement and infrastructure designed to inform researchers about their reporting obligations, have the potential to improve compliance with both the letter and the spirit of the regulations. “I think reporting rates will skyrocket after the rulemaking,” she says.
In the end, Anderson notes, reporting clinical trials results in order to contribute to scientific and medical knowledge is as much an ethical obligation for researchers as a legal one: “It’s something we really promise to every patient when they enroll on a trial.”
In June 2014, the NIH held a joint workshop with the Nature Publishing Group and Science on the issue of reproducibility and rigor of research findings. The workshop’s goal was to strengthen approaches to support biomedical research that is reproducible, robust, and transparent. An editorialappears in the November 5, 2014, online edition of Nature.
Workshop participants included journal editors representing more than 30 basic/preclinical science journals in which NIH-funded investigators have most often published. Attendees reached consensus on a set of principles and guidelinesto facilitate the interpretation and repetition of experiments as they have been conducted in published studies. Principles endorsed by the group cover five areas, recommended to be delineated in each journal’s Information for Authors section or other public place:
Rigorous statistical analysis: Outline the journal’s policy for statistical analysis and have a method of checking the statistical accuracy of submissions
Transparency in reporting: Provide a checklist of reporting standards (replicates, statistics, randomization, blinding, sample-size estimation, inclusion/exclusion criteria) and require authors to state where this information is located in the manuscript
Data and material sharing: Stipulate that all datasets on which the conclusions of the paper rely must be made available upon request, where appropriate, during manuscript review and upon publication
Consideration of refutations: Include the journal’s policy for considering refutations of the paper, subject to its usual standards of quality
Best practices guidelines: Establish methods for dealing with image-based data and biological material (antibodies, cell lines, animals)
The existence of these guidelines does not preclude the need for replication or independent verification of research results, but should make it easier to perform such replication. Journals endorsing the proposed principles and guidelines are listed here.