ARCHIVED PAGE
Archived on November 26, 2025. Go to the latest version.
Dissemination Approaches For Different Stakeholders
Section 3
Case Study: Journal Reviews of NIH Collaboratory Trials
Pragmatic clinical trials test whether evidence-based interventions work in real-world settings (Simon et al 2020). Unlike traditional clinical trials, the study interventions in pragmatic trials are typically integrated into routine clinical processes and workflows and often rely on existing sources of electronic data collected at the point of care (Staman et al 2023). When large-scale pragmatic trials are embedded in healthcare systems, they have the potential to directly inform clinical practice, guidelines, and health policy decisions (Palazzo et al 2022).
The NIH Pragmatic Trials Collaboratory supports several large-scale, multicenter pragmatic clinical trials embedded in healthcare systems (see the NIH Collaboratory Trials). Several of these trials have been completed, and their primary results have been published in the peer-reviewed biomedical literature (Coronado et al 2018; DeBar et al 2022; Dember et al 2019; Huang et al 2019; Jarvik et al 2020; Melnick et al 2022; Mitchell et al 2020; Simon et al 2022; Vazquez et al 2024; Zatzick et al 2021).
In the early years of the NIH Pragmatic Trials Collaboratory, which began in 2012, there was a question about how journals' peer reviewers would react to manuscripts reporting on pragmatic trials, given that such designs were still relatively new. It was unknown whether reviewers would have sufficient familiarity with or understanding of innovative or complex pragmatic trials methods. Therefore, we sought to address this question by exploring reviews of manuscripts reporting the primary outcomes of NIH Collaboratory Trials.
We invited the principal investigators of the first 8 completed NIH Collaboratory Trials (Coronado et al 2018; DeBar et al 2022; Dember et al 2019; Huang et al 2019; Jarvik et al 2020; Mitchell et al 2020; Simon et al 2022; Zatzick et al 2021), after their primary results were published, to confidentially share with the NIH Collaboratory Coordinating Center the comments they received on their manuscripts during the peer review process. The manuscript reviews were received between 2018 and 2020, and the manuscripts were published between 2018 and 2022 (Table 1).
Table 1. Study Designs Described in the Reviewed Manuscriptsa
| a Some of the 8 manuscripts were reviewed by more than 1 journal. | |
| Journal | Study Designs Described in the Manuscripts |
| Annals of Internal Medicine | 2 parallel cluster randomized trials |
| JAMA | 1 randomized controlled trial |
| 1 parallel cluster randomized trial | |
| 2 stepped-wedge cluster randomized trials | |
| JAMA Internal Medicine | 2 parallel cluster randomized trials |
| JAMA Surgery | 1 stepped-wedge cluster randomized trial |
| Journal of the American Society of Nephrology | 1 parallel cluster randomized trial |
| The Lancet | 1 parallel cluster randomized trial |
| New England Journal of Medicine | 1 stepped-wedge cluster randomized trial |
The 3 of us independently reviewed the peer review comments received for the first 3 manuscripts and met to discuss the themes that emerged. We then independently reviewed the peer review comments for all 8 of the manuscripts to note common questions and themes.
We did not generally observe that reviewers were unfamiliar with pragmatic clinical trials or had difficulty understanding the design and analytic approaches of the studies. Rather, the questions that most commonly arose in the manuscript reviews were related to the implementation and content of the study interventions. Specifically, reviewers asked for more information about (1) implementation outcomes, (2) implementation strategies, and (3) intervention content. See Table 2 for representative comments.
Table 2. Representative Comments by Journal Editors and Reviewers
| a Portions of some comments are redacted for deidentification of the studies and the reviewing journals. | |
| Theme | Commenta |
| Include more information about implementation outcomes | “Is there additional information on patient adherence with the [intervention protocol]? …[It is] unclear the extent to which patients were able to adhere to [the intervention protocol], which could influence the results.”
"Major flaw is lack of implementation data—for this pragmatic trial, a few observations every quarter would not generate an estimation of adherence to the intervention…”
“[It is] important to distinguish between problems with efficacy and problems (merely) with adherence. If the [interventions] are not efficacious, we should abandon them (at least for this population and/or application), whereas if the [interventions] are efficacious but this effect was diluted by low adherence, then we should focus on methods to improve adherence. I believe the authors have the requisite data and trial design to perform a complier average treatment effect (CATE) analysis (an instrumental variable analysis in which randomization arm is the instrument).” |
| Include more information about implementation strategies | “Please give more detail about the steps taken to the implement the intervention. The intervention appears very top down. Was there any effort to engage and motivate the [staff] responsible for implementing the intervention?”
“In light of the difficulty in achieving adherence to the intervention, several questions arise with regard to the methods for implementing the intervention. For instance, some of the detail in the protocol about investigator efforts to ‘influence’ [clinic staff] and the guidance provided to the [clinics] should be pulled into the main manuscript to provide better clarity on what steps the investigators took to get [clinics] to implement the practice change.”
“It would be of clinical significance to further discuss approaches to improve uptake of the trial intervention. It appears that there was excellent collaboration/stakeholder engagement at the executive and research level of the [healthcare systems] but that engagement did not extend to the [clinic] level where the intervention was carried out. Were environmental assessments carried out before the trial? Was there a pilot feasibility test of the trial conducted? Were qualitative interviews conducted with key stakeholders including patients, [clinicians] and [clinic] managers? None of these were reported.” |
| Include more information about intervention content | “Please include more description in the paper about the content of the intervention, include [a table] about the elements of the intervention received, and more description about the content of the [training].… The paper does not include enough information to really understand the process.”
“The intervention is billed as ‘brief,’ but there is mention of [more intensive care] for various persons, so it is useful to know what the minimal intervention consists of in terms of contact time with patient, and up to what amount is considered maximum time with the patient. What is the minimum that everybody gets?” |
Implementation outcomes are the effects of the actions taken in a pragmatic trial to implement the practices or processes being studied. For example, the uptake (or adoption) of an intervention and the adherence (or fidelity) to an intervention are implementation outcomes. In several of the manuscript reviews, reviewers asked the authors to provide information about whether and how such outcomes were measured in the trial and, if they were measured, to report data on intervention implementation in the manuscript. The reviewers were interested in learning about the implementation outcomes themselves and about how outcomes like uptake and adherence may have been related to the primary study results.
Reviewers also asked questions about implementation strategies, or the steps the study teams took to implement their study interventions. For example, one reviewer requested more information about how the study team engaged with frontline staff in the participating clinics and what, if any, incentives were provided for referring patients to the intervention. Another asked whether the study team conducted pilot or feasibility studies of strategies for engaging healthcare system partners. In another case, a reviewer wanted to know more about the integration of the intervention into existing clinic workflows and about reported difficulties in rolling out the intervention. In asking about implementation strategies, reviewers were seeking to better understand how methods of engaging healthcare practitioners in implementing the interventions may have been related to implementation outcomes and the study results.
Some reviewers also wanted more information about intervention content. For example, in the case of a manuscript reporting the results of a complex intervention involving a team-based care model, a reviewer was interested in seeing a more detailed description of the intervention.
See also the Dissemination and Implementation chapter of the Living Textbook.
Embedding a pragmatic clinical trial in the existing workflow of a healthcare system presents complex challenges that can influence the uptake of and adherence to the intervention and the study team's ability to detect treatment effects (Staman et al 2023). In addition to publishing their primary study results, some NIH Collaboratory Trial teams have published separate implementation-focused papers that report implementation outcomes, barriers to implementation, and lessons learned. Our experience with journal peer reviews of the first several completed NIH Collaboratory Trials suggests that reviewers may want or expect some of this information to be included with the report of primary study outcomes to aid in the interpretation of results.
The small sample of manuscripts for which we explored peer reviews were published in journals with relatively high impact factors in their fields. The outcomes reported in these manuscripts were from pragmatic clinical trials that started between 2013 and 2016. It is unknown whether our observations are generalizable to the experiences of pragmatic trials investigators with other journals or those reporting the results of more recently initiated studies.
Full and transparent reporting of pragmatic clinical trials—including details about intervention implementation—are important for helping readers to understand how the studies were conducted and to place the results in context. The NIH Pragmatic Trials Collaboratory has developed a reporting template for pragmatic trials. Reporting guidelines, such as the pragmatic trials extension of the CONSORT statement (Zwarenstein et al 2008), are available to help investigators improve their reporting of pragmatic trials and assist peer reviewers and journal editors in understanding these studies and their importance.
SECTIONS
Resources
NIH Collaboratory Trials Publication Types Handout
A reference to help pragmatic trial teams understand potential opportunities for publication
Data and Resource Sharing
Data and resources shared by the NIH Collaboratory Trials, including protocols, consent documents, public use datasets, computable phenotypes, analytic code, and more
REFERENCES
Coronado GD, Petrik AF, Vollmer WM, et al. 2018. Effectiveness of a mailed colorectal cancer screening outreach program in community health clinics: the STOP CRC cluster randomized clinical trial. JAMA Intern Med. 178(9):1174-1181. doi: 10.1001/jamainternmed.2018.3629. PMID: 30083752.
DeBar L, Mayhew M, Benes L, et al. 2022. A primary care-based cognitive behavioral therapy intervention for long-term opioid users with chronic pain: a randomized pragmatic trial. Ann Intern Med. 175(1):46-55. doi: 10.7326/M21-1436. PMID: 34724405.
Dember LM, Lacson E Jr, Brunelli SM, et al. The TiME trial: a fully embedded, cluster-randomized, pragmatic trial of hemodialysis session duration. J Am Soc Nephrol. 30(5):890-903. doi: 10.1681/ASN.2018090945. PMID: 31000566.
Huang SS, Septimus E, Kleinman K, et al. 2019. Chlorhexidine versus routine bathing to prevent multidrug-resistant organisms and all-cause bloodstream infections in general medical and surgical units (ABATE Infection trial): a cluster-randomised trial. Lancet. 393(10177):1205-1215. doi: 10.1016/S0140-6736(18)32593-5. PMID: 30850112.
Jarvik JG, Meier EN, James KT, et al. 2020. The effect of including benchmark prevalence data of common imaging findings in spine image reports on health care utilization among adults undergoing spine imaging: a stepped-wedge randomized clinical trial. JAMA Netw Open. 3(9):e2015713. doi: 10.1001/jamanetworkopen.2020.15713. PMID: 32886121.
Melnick ER, Nath B, Dziura JD, et al. 2022. User centered clinical decision support to implement initiation of buprenorphine for opioid use disorder in the emergency department: EMBED pragmatic cluster randomized controlled trial. BMJ. 377:e069271. doi: 10.1136/bmj-2021-069271. PMID: 35760423.
Mitchell SL, Volandes AE, Gutman R, et al. 2020. Advance care planning video intervention among long-stay nursing home residents: a pragmatic cluster randomized clinical trial. JAMA Intern Med. 180(8):1070-1078. doi: 10.1001/jamainternmed.2020.2366. PMID: 32628258.
Palazzo L, Tuzzio L, Simon GE, Larson EB. 2022. A value proposition for pragmatic clinical trials. Am J Manag Care. 28(9):e312-e314. doi: 10.37765/ajmc.2022.89224. PMID: 36121362.
Simon GE, Platt R, Hernandez AF. 2020. Evidence from pragmatic trials during routine care - slouching toward a learning health system. N Engl J Med. 382(16):1488-1491. doi: 10.1056/NEJMp1915448. PMID: 32294344.
Simon GE, Shortreed SM, Rossom RC, et al. 2022. Effect of offering care management or online dialectical behavior therapy skills training vs usual care on self-harm among adult outpatients with suicidal ideation: a randomized clinical trial. JAMA. 327(7):630-638. doi: 10.1001/jama.2022.0423. PMID: 35166800.
Staman KL, Check DK, Zatzick D, et al. 2023. Intervention delivery for embedded pragmatic clinical trials: Development of a tool to measure complexity. Contemp Clin Trials. 126:107105. doi: 10.1016/j.cct.2023.107105. PMID: 36708968.
Vazquez MA, Oliver G, Amarasingham R, et al. 2024. Pragmatic trial of hospitalization rate in chronic kidney disease. N Engl J Med. 390(13):1196-1206. doi: 10.1056/NEJMoa2311708. PMID: 38598574.
Zatzick D, Jurkovich G, Heagerty P, et al. 2021. Stepped collaborative care targeting posttraumatic stress disorder symptoms and comorbidity for US trauma care systems: a randomized clinical trial. JAMA Surg. 156(5):430-474. doi: 10.1001/jamasurg.2021.0131. PMID: 33688908.
Zwarenstein M, Treweek S, Gagnier JJ, et al. 2008. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 337:a2390. doi: 10.1136/bmj.a2390. PMID: 19001484.