Why Are We Talking About Pragmatic Trials?

What is a Pragmatic Clinical Trial

Section 1

Why Are We Talking About Pragmatic Trials?


Kevin Weinfurt, PhD


Contributing Editors

Karen Staman, MS

Jonathan McCall, MS

Liz Wing, MA

Healthcare in the United States is increasingly complex and expensive, and there is a great need for more evidence to inform decisions that lead to improved, efficient, and affordable care (Alper and Grossmann 2015). Care providers, researchers, administrators, payers, regulators, and the public agree that the provision of medical care should be informed by high-quality scientific evidence regarding the risks and benefits of treatments. Yet, this high-quality evidence—generated by conducting randomized controlled trials (RCTs) and disseminated through clinical practice guidelines—is severely lacking across a multitude of specialties (Tricoci et al. 2009; Roos et al. 2011; Wright et al. 2011; Koh et al. 2013; Feuerstein et al. 2014; Neuman et al. 2014). In the absence of this evidence, clinicians must make educated guesses to determine treatment based on personal judgment and knowledge of the patient, rather than on the consensus of a group of clinical experts (Tricoci et al. 2009). Clinicians and patients simply do not have enough evidence to effectively inform clinical decisions. For example, in the field of cardiology, which arguably has one of the most robust evidence bases among specialties, the majority of treatment recommendations are founded upon lower-quality trials, observational studies, or expert opinion (Tricoci et al. 2009).

In addition, when we survey the US clinical trials enterprise from a broad perspective, the kind of trials needed to provide medical evidence to support treatment decisions are, for the most part, not being done. Analyses of trials contained in the ClinicalTrials.gov database have shown that the vast majority of clinical trials are too small to provide sufficient statistical power to definitively answer clinical questions, they fail to address critical treatment priorities, or they suffer from shortcomings in design and execution that limit their usefulness (Califf et al. 2012; Pasquali et al. 2012; Alexander et al. 2013; Goswami et al. 2013; Hirsch et al. 2013; Lakey et al. 2013; Todd et al. 2013; Witsell et al. 2013; Inrig et al. 2014; Subherwal et al. 2014). In addition, the data from many of these trials are not being reported in timely and transparent ways (Anderson et al. 2015). Adding to these complications, there has been a steady drumbeat of revelations indicating that many findings published in the peer-reviewed literature are fundamentally unreliable (Ioannidis 2005; Ioannidis 2016; Open Science Collaboration 2015; Le Noury et al. 2015).

For much of modern medical history, clinical research has been kept separate from the delivery of routine patient care, resulting in an environment in which research data are collected using standalone systems. These systems are designed to ensure that the information gathered during research activities is valid and complete. However, having separate systems for research and care comes at a significant cost. There is growing concern that the results obtained from clinical research may not apply to “real-world” situations (Ioannidis 2005), because the research is often done under artificial conditions with volunteers who may not reflect the patients who actually live with a given disease or condition. Further, standalone systems require enormous amounts of money and effort to sustain.

Learning Health Systems and Embedded Clinical Research

Many are now advocating a move to a learning health system in which tools such as computing power, connectivity, team-based care, and systems engineering techniques will produce a culture of continuous learning at lower cost (Institute of Medicine 2013). Ideally, clinical trials would be embedded within a system of healthcare delivery where evidence is rapidly and continually fed back into clinical care, and clinical care itself would inform the further development of medical evidence (Embi 2019; Simon 2020). At the same time, the widespread use of electronic health records and advances in information technology and informatics are creating opportunities to combine very large, complex sets of data (“big data”) in ways that until now were almost unimaginable. As systems for managing data continue to improve within US health systems, the availability of electronic data is likewise improving rapidly.

There is a need for “a different context to clinical research that could speed the discovery and implementation of evidence-based advancements to healthcare delivery. Pragmatic clinical trials (PCTs) are a promising type of trial conducted within real-world health care delivery systems” (Tuzzio and Larson 2019).

The emergence of PCTs embedded in healthcare systems, or ePCTs, represents one approach that could support a goal of a learning health system by informing real-world practice with digital health data collected at the point of care. ePCTs have the potential to inform policy and practice with high-quality evidence at reduced cost and increased efficiency compared with traditional clinical trials.

The next sections introduce characteristics of ePCTs, with examples drawn from the NIH Collaboratory Demonstration Projects, and point readers to rich resources available in this Living Textbook that describe best practices around how to design, conduct, and disseminate ePCTs.




back to top

Alexander KP, Kong DF, Starr AZ, et al. 2013. Portfolio of clinical research in adult cardiovascular disease as reflected in ClinicalTrials.gov. J Am Heart Assoc. 2:e000009. doi:10.1161/JAHA.113.000009. PMID: 24072529.

Alper J, Grossmann C. 2015. Integrating Research and Practice: Health System Leaders Working toward High-Value Care: Workshop Summary. Washington, DC: National Academies Press.

Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM. 2015. Compliance with Results Reporting at ClinicalTrials.gov. New Engl J Med. 372:1031–1039. doi:10.1056/NEJMsa1409364. PMID: 25760355.

Califf RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. 2012. Characteristics of clinical trials registered in ClinicalTrials.gov, 2007-2010. JAMA. 307:1838. doi:10.1001/jama.2012.3424. PMID: 22550198.

Embi PJ, Richesson R, Tenenbaum J, et al. 2019. Reimagining the research-practice relationship: policy recommendations for informatics-enabled evidence-generation across the US health system. JAMIA Open. 2:2-9. doi:10.1093/jamiaopen/ooy056.

Institute of Medicine. 2013. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds. Washington, DC: National Academies Press. http://www.ncbi.nlm.nih.gov/books/NBK207225/. Accessed January 28, 2015.

Feuerstein JD, Akbari M, Gifford AE, et al. 2014. Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc. 89:16–24. doi:10.1016/j.mayocp.2013.09.013. PMID: 24388018.

Goswami ND, Pfeiffer CD, Horton JR, Chiswell K, Tasneem A, Tsalik EL. 2013. The state of infectious diseases clinical trials: a systematic review of ClinicalTrials.gov. PLoS ONE. 8:e77086. doi:10.1371/journal.pone.0077086. PMID: 24146958.

Hirsch BR, Califf RM, Cheng SK, et al. 2013. Characteristics of oncology clinical trials: insights from a systematic analysis of ClinicalTrials.gov. JAMA Intern Med. 173:972. doi:10.1001/jamainternmed.2013.627. PMID: 23699837.

Inrig JK, Califf RM, Tasneem A, et al. 2014. The landscape of clinical trials in nephrology: a systematic review of Clinicaltrials.gov. Am J Kidney Dis. 63:771–780. doi:10.1053/j.ajkd.2013.10.043. PMID: 24315119.

Ioannidis JPA. 2005. Why most published research findings are false. PLoS Med. 2:e124. doi:10.1371/journal.pmed.0020124. PMID: 16060722.

Ioannidis JP. 2016. Why most clinical research is not useful. PLoS Med. 13:e1002049. doi:10.1371/journal.pmed.1002049. PMID: 27328301.

Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. 2013. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations. Hepatology. 58:2142–2152. doi:10.1002/hep.26578. PMID: 23775835.

Lakey WC, Barnard K, Batch BC, Chiswell K, Tasneem A, Green JB. 2013. Are current clinical trials in diabetes addressing important issues in diabetes care? Diabetologia. 56:1226–1235. doi:10.1007/s00125-013-2890-4. PMID: 23564296.

Le Noury J, Nardo JM, Healy D, et al. 2015. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. h4320. doi:10.1136/bmj.h4320. PMID:26376805.

Neuman MD, Goldstein JN, Cirullo MA, Schwartz JS. 2014. Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations. JAMA. 311:2092–2100. doi:10.1001/jama.2014.4949. PMID: 24867012.

Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science. 349:aac4716–aac4716. doi:10.1126/science.aac4716. PMID:26315443.

Pasquali SK, Lam WK, Chiswell K, Kemper AR, Li JS. 2012. Status of the pediatric clinical trials enterprise: an analysis of the US ClinicalTrials.gov registry. Pediatrics. 130:e1269–e1277. doi:10.1542/peds.2011-3565. PMID:23027172.

Roos M, Brodbeck J, Sarkozy A, Chierchia GB, De Asmundis C, Brugada P. 2011. A critical analysis of the scientific evidence behind international guidelines related to cardiac arrhythmias. Circ Arrhythm Electrophysiol. 4:202–210. doi:10.1161/CIRCEP.110.958181. PMID: 21372270.

Simon GE, Platt R, Hernandez AF. 2020. Evidence from pragmatic trials during routine care — slouching toward a learning health system. N Engl J Med. 382:1488-1491. doi:10.1056/NEJMp1915448.

Subherwal S, Patel MR, Chiswell K, et al. 2014. Clinical trials in peripheral vascular disease: pipeline and trial designs: an evaluation of the ClinicalTrials.gov database. Circulation. 130:1812–1819. doi:10.1161/CIRCULATIONAHA.114.011021. PMID: 25239436.

Todd JL, White KR, Chiswell K, Tasneem A, Palmer SM. 2013. Using ClinicalTrials.gov to understand the state of clinical research in pulmonary, critical care, and sleep medicine. Ann Am Thorac Soc. 10:411–417. doi:10.1513/AnnalsATS.201305-111OC. PMID: 23987571.

Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. 2009. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 301:831–841. doi:10.1001/jama.2009.205. PMID: 19244190.

Tuzzio L, Larson EB. 2019. The promise of pragmatic clinical trials embedded in learning health systems. EGEMS (Wash DC). 7:10. doi:10.5334/egems.285. PMID: 30972359.

Witsell DL, Schulz KA, Lee WT, Chiswell K. 2013. An analysis of registered clinical trials in otolaryngology from 2007 to 2010: ClinicalTrials.gov. Otolaryngol Head Neck Surg. 149:692–699. doi:10.1177/0194599813506545. PMID: 24107478PMCID: PMC4121965.

Wright JD, Pawar N, Gonzalez JSR, et al. 2011. Scientific evidence underlying the American College of Obstetricians and Gynecologists’ practice bulletins. Obstet Gynecol. 118:505–512. doi:10.1097/AOG.0b013e3182267f43. PMID: 21826038.

Version History

May 12, 2020: Added quote from Tuzzio 2019; added references to Embi 2019, Ioannidis 2016, Simon 2020, Tuzzio 2019; and made nonsubstantive changes as part of annual content update (changes made by L. Wing).

Published August 25, 2017


Weinfurt K. What is a Pragmatic Clinical Trial: Why Are We Talking About Pragmatic Trials?. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. Available at: https://rethinkingclinicaltrials.org/chapters/pragmatic-clinical-trial/what-is-a-pragmatic-clinical-trial/. Updated May 13, 2020. DOI: 10.28929/089.