The Institute of Medicine and the Learning Healthcare System Concept

[Back to top]

In 2007, the Washington-based Institute of Medicine (IOM), now the National Academy of Medicine (NAM), released a book-length report titled The Learning Healthcare System [1]. The first in a series of a dozen reports from the IOM’s Roundtable on Evidence-Based Medicine (now the Roundtable on Value & Science-Driven Health Care), the report defined and described a new conceptual approach for integrating the disparate spheres of clinical research and clinical medicine:

A learning healthcare system is [one that] is designed to generate and apply the best evidence for the collaborative healthcare choices of each patient and provider; to drive the process of discovery as a natural outgrowth of patient care; and to ensure innovation, quality, safety, and value in health care [1].

The Learning Healthcare System itself grew out of earlier Institute reports that had identified medical errors and avoidable shortcomings in treatment as substantial causes of harm in the United States [2] and proposed approaches for systemically improving healthcare nationwide [3]. In addition, it sought to address longstanding concerns that much of the decision-making surrounding clinical care in the United States—even when guided by clinical practice guidelines—was inadequately supported by high-quality evidence [4]. At the same time, the emergence of powerful and sophisticated information and communications systems, including electronic health records (EHRs) and enterprise data warehouses that collected and combined data from across entire health systems, pointed toward a future in which data routinely captured at the point of patient care could ultimately be leveraged across linked networks of hospitals and health systems to answer questions about treatment efficacy, safety, quality of care, and the comparative effectiveness of different therapies.

Attributes of a Learning Healthcare System

[Back to top]

Although in practice the learning healthcare system (LHS) concept involves multiple factors and complex interactions, the overarching construct is relatively straightforward. Put simply, the LHS creates a continuous cycle or feedback loop in which scientific evidence informs clinical practice while data gathered from clinical practice and administrative sources inform scientific investigation.

Key Attributes of a Learning Healthcare System [1]

• Adaptation to the pace of change
• Stronger synchrony of efforts
• Culture of shared responsibility
• New clinical research paradigm
• Clinical decision support systems
• Universal electronic health records
• Tools for database linkage, mining, and use
• Notion of clinical data as a public good
• Incentives aligned for practice-based evidence
• Public engagement
• Trusted scientific broker
• Leadership

Reproduced with permission from The Learning Healthcare System: Workshop Summary, 2007 by the National Academy of Sciences, Courtesy of the National Academies Press, Washington, DC.

Using Clinical Data to Drive Learning Healthcare

[Back to top]

A 2008 IOM report, Clinical Data as a Basic Staple of Health Learning, distilled a series of discussions focused on the use of data gathered as part of routine patient care and administrative contact (as opposed to formal clinical research) as a key element underpinning an LHS [5]. Although results from randomized controlled trials are well-established as the “gold standard” for medical evidence, there has long been concern that such trials, which are often performed outside the usual system of care and recruit highly selected populations for participation, may not truly represent the populations of patients with particular diseases or conditions to whom the results of the trials will be applied. In addition, traditional clinical trials are costly and time consuming.

For these reasons, the concept of drawing upon data gathered directly from the patient care environment has enormous potential for accelerating the rate at which useful knowledge is accumulated, including information about quality of care, efficiency, safety, and cost-effectiveness:

Health information is a key vehicle for changing the healthcare system, but how do we create the data or evidence? How do we actually get assembled, coherent, representative, timely, and valid health information that can inform decisions at a patient level or at a broad population level or even at a very large population level?...Information drawn from actual experience in care delivery must be able to shape the care delivery process. Specifically, the data that inform our policy or inform population care should not be separate from what is really going on in care [5].

However, use of clinical data for continuous learning poses numerous practical, technological, logistical, methodological, and ethical challenges. The report frames a discussion of these issues within the context of clinical data as a public good to be used for broad societal benefit.

Ethical and Regulatory Oversight of Learning Healthcare Systems

[Back to top]

These new approaches themselves entail a re-imagining of regulatory and ethical oversight models that currently govern the conduct of patient care and research involving human subjects. A special report from the Hastings Center published in early 2013 describes and explores in detail the ethical and regulatory implications raised by conducting research in learning healthcare systems, and provides recommendations for adapting to this new paradigm of treatment and research [6].

Patient and Public Engagement

[Back to top]

An equally important aspect of the LHS vision is the central role played by patients and the public. As the IOM report Patients Charting the Course observes:

…most health systems today are not centered on patients. Instead, volume drives service; supply influences demand; and clinician—not patient—preferences shape practice (Wennberg et al., 2007). The notion of patient-centeredness often still feels unfamiliar, even disruptive, for many of those unexposed to the advantages of such a culture (Berwick, 2009) [7].

In this new construct, patients are active and engaged participants in research and care, both of which are increasingly viewed as patient-centered activities. The IOM describes this as “patient-anchored care.”

Patient-Anchored Care in a Learning Health System

The IOM points out that patient engagement is critical to the success of the learning process in health systems increasingly characterized by uptake of genetic and “omics” technologies and a focus on personalized medicine:

Each patient experience offers the potential to deepen the knowledge base that drives care quality and outcomes—at the individual, practice, and societal levels.…These are issues in which patients have a strong stake, and they must have confidence in the system’s functionality for the generation of timely and reliable new insights [7].

EHRs, Information Technology & Informatics  Infrastructure

[Back to top]

In the years since the IOM’s initial report identified the leveraging of advances in information technology as a critical component of building an LHS, multiple additional meetings and their resulting reports have examined specific elements of this component in detail. In 2011, the IOM published a workshop summary titled Engineering a Learning Healthcare System: A Look at the Future, which examined the challenges of building the overarching systems and infrastructure needed to support the LHS from an engineering perspective [8]. A further report, Digital Infrastructure for the Learning Healthcare System, noted that while the use of EHRs and other sophisticated health informatics tools is rapidly accelerating, their uptake has been relatively piecemeal and shortcomings among various technological solutions could potentially hamper their success in the learning healthcare environment. The report discusses potential solutions to challenges involving privacy, security, interoperability, data standards, and governance, among other issues [9]. Additional workshop summaries including Learning What Works: Infrastructure Required for Comparative Effectiveness Research and Digital Data Improvement Priorities for Continuous Learning in Health and Health Care, further explore these issues and offer strategies for achieving the broad infrastructure needed to support an LHS [10,11]. The IOM discussions stress that collaboration will be needed among traditionally “siloed” medical institutions and private companies.

In recognition of the potential for the LHS model to improve the nation’s healthcare, The Health Information Technology for Economic and Clinical Health (HITECH) Act, a part of the American Recovery and Reinvestment Act of 2009, authorized approximately $20 billion in spending over 5 years to promote the adoption and use of EHRs that could ultimately be connected through a national health information network [12]. Under the HITECH Act, hospitals and physicians who make meaningful use of interoperable EHRs can qualify for incentive payments through Medicare and Medicaid.

Challenges to Building the Learning Healthcare System

[Back to top]

The creation of a true LHS poses significant challenges as existing systems and approaches are adapted or disrupted by evolving approaches to research and care. Some of these challenges are technical in nature, such as developing infrastructures and methodologies to accommodate ever-increasing streams of data from diverse sources. Other challenges will be more cultural in nature, as existing understandings of ethical and regulatory governance are forced to adapt to new exigencies.

EHRs, Data Standards, Interoperability & Computable Phenotypes

The use of EHRs in both patient care and clinical research is a key element of the LHS vision. However, if the transformative potential of EHRs is to be realized in clinical research, certain key conditions must first be met. EHRs must be fully interoperable with each other and governed by agreed-upon and uniform data standards and definitions. Shared terms, definitions, quality standards, and best practices must be accessible to all participants and made available through the work of standards organizations (such as the Clinical Data Interchange Standards Consortium and HL7). A particularly complex challenge hinges upon the development of computable electronic phenotypes that allow patients with conditions of interest to be identified directly via a query directed at repositories of EHR data [13,14].

Professional/Health System Interactions

One persistent obstacle to the transformation of existing systems into a linked LHS is the friction that may arise between researchers, healthcare providers, and health systems. Many health systems and individual hospitals are primarily configured to support the efficient provision of patient care, but are less accommodating to the presence of research activities within the same space. To enable an LHS in which care and research are part of single continuous cycle, health systems may be required to adjust incentive structures and expedite the removal of administrative burdens and barriers that impede research activities. Healthcare providers must be engaged across the spectrum of clinical care to participate actively in research activities. Policies and best practices for engaging researchers and supporting their interactions with health systems should be shared as widely as possible.

Ethics/Regulatory Challenges

The ongoing transformation in clinical research that has taken place with application of the LHS model has blurred the line between clinical practice and research, creating challenges within the U.S. regulatory system, which has traditionally sought to keep these activities separate to avoid a phenomenon known as “therapeutic misconception” [15,16], in which patients enrolled in clinical research may “…(fail) to appreciate the difference between research and treatment” [17]. Quality improvement (QI) studies aimed at improving processes or patient management within individual hospitals or health systems have often been exempted from institutional review board (IRB) oversight, the obligation to obtain informed consent, and Health Insurance Portability and Accountability Act (HIPAA) requirements because they are not considered to constitute research. Such QI studies are understood to address factors affecting care at a local level and thus do not meet a key criterion for being labeled as research: namely, the aim of producing “generalizable knowledge” (45 CFR §46.102).

Under current regulatory frameworks that make this distinction between QI and research, studies conducted within an LHS could avoid some regulatory strictures by being labeled as QI. However, there is increasing debate over which QI and LHS studies should be classified as research, and regulations surrounding this issue are complex and may be inconsistently applied [18,19].

Quality improvement is defined as “systematic, data-guided activities designed to bring about immediate improvements in health care delivery in particular settings” [20].

There have been attempts to better define which QI efforts should be categorized as research [18,20,21]. In 2011, the Advance Notice of Proposed Rulemaking (ANPRM) proposed changes to the Common Rule solicited input on whether to clarify the definition of research in regards to which QI or program evaluation studies are covered [22]. (For detailed discussion on issues related to the ANPRM, click here).

The Hastings Center published a special report in 2006 addressing the ethics of QI methods [18] and in 2011 dedicated a supplemental publication to the topic of ethical oversight of learning healthcare systems [6]. Some have argued for a new ethics framework to govern research in an LHS [23], with the justification that “current consent and oversight practices too often overprotect patients from research that has little effect on what matters to patients, whereas in other cases oversight practices underprotect patients from medical errors and inappropriate medical management because they make research to reduce these problems unduly burdensome to conduct” [24]. In contrast, others assert that the research/care distinction remains important [25], the right to informed consent must be maintained [26], and comparative effectiveness research can be conducted with some streamlining of current practices [27].

Additional barriers to conducting research in an LHS include inefficiencies in IRB review and the contracts process [28,29]. Despite regulators’ encouragement for the use of a centralized IRB review for multisite studies [30], implementation of this practice has been limited [22,28,29]. Local review of multisite trials by each participating institution can result in hundreds of IRB reviews and all the costs and delays associated with them [22,28,29,31], yet there is a lack of evidence to suggest that these additional reviews result in substantive ethical or scientific improvements to the studies [28]. The ANPRM proposed Common Rule changes included consideration for a mandate requiring centralized IRB review for sites within the United States participating in multisite studies [22]. Contracts covering confidentiality, intellectual property, indemnification, and publication terms further add to the burden of multisite trial startup and can be plagued by lengthy delays and disputes [28].

Statistical Methods

The growing wealth of electronic health data available for research through EHRs will not be meaningful unless it can help patients, physicians, administrators, and policy makers to make more informed decisions about health and healthcare. This type of research poses challenges in terms of the sheer volume of data to be analyzed, whether from vast networks of EHRs or cluster randomized trials involving multiple clinics, hospitals, or health systems. Together with the revolution in “big data,” there have been developments in mathematics to allow generation of actionable medical evidence from these large datasets [11].

The different types of studies that might be undertaken in an LHS require customized statistical approaches [32]. For example, analysis of observational studies must address confounding, defined as “when a potential cause-outcome relationship is distorted by a second cause whose effect is falsely attributed to the first cause.” Methods to address confounding include propensity score analysis, sensitivity analysis, and instrumental variable analysis [32–34]. In cluster randomized trials, statistical models adjust for the tendency of outcomes to cluster by site due to shared characteristics of the patients, such as having the same healthcare providers [32].

More robust statistical methods are needed for developing clinical decision support tools that can use EHR data at the individual patient level to assist healthcare providers in determining “the right treatment, for the right patient, at the right time” [10]. The future in learning health may bring integrated analytical tools and statistical models within EHR databases for real-time learning or “virtual studies.”

Patient-Reported Outcomes

Patient-reported outcomes (PROs) in an LHS can provide real-time, actionable information that affects clinical care and can also be used to answer and generate patient-centric research questions. Ideally, PRO data would be collected electronically and linked with the EHR. Potential methods for electronic data collection include devices such as computer tablets or smart phones, online submission forms, and interactive phone systems. More recently, wearable devices that can track and automatically report various kinds of biological data have enjoyed increasing popularity. However, safeguards are necessary to ensure data security, availability of an audit trail, data backup, and appropriate access to investigators and clinical staff [35].

The FDA guidance on the use of PRO measures to support claims of treatment benefit in product labeling states that the PRO measure should have demonstrated validity, reliability, and ability to detect change, and should have been defined as an endpoint in the protocol [35]. The Patient-Centered Outcomes Research Institute (PCORI), an organization established by the Patient Protection and Affordable Care Act of 2010 to promote research guided by patients, caregivers, and the broader healthcare community, has developed a methodological report on patient-centered research, which includes standards for incorporating PRO measures and covers issues such as how to handle missing data [36]. In its first 3 years, PCORI funded 30 projects aimed at accelerating the implementation of patient-centered outcomes research and methodologic research [37]. These efforts should continue to inform the design of patient-centric studies and inclusion of PROs.

The National Patient-Centered Clinical Research Network (PCORnet), launched in 2014, brings together 29 individual networks to conduct comparative effectiveness research on a nationwide scale and answer questions of greatest concern to patients and caregivers. A key part of this effort is the involvement of patients as clinical research partners participating in the governance of the network. PCORnet research relies on patient-generated information as well as EHR data from routine care and clinical settings.

Abernethy and colleagues demonstrated the utility of PRO data for cancer research in the setting of an LHS [38]. In this study, which was conducted at a single academic cancer clinic, patients with cancer were surveyed to assess the correlation between reported sexual problems and other PROs such as quality of life and distress. Electronic data collection was found to be feasible and secure and to have similar results as paper data collection. The initial results showing that sexual problems were common and associated with deficits in other PROs were then used to design an intervention in which a psychosocial care program was offered to affected patients as identified by questionnaire. Patients enrolling in the care program reported significantly improved psychosocial outcomes on follow-up. These results were used to inform changes in care at the local/clinic level as well as the design of further studies.

Early Examples

[Back to top]

AcademyHealth

AcademyHealth, a nonprofit healthcare policy and research network consortium, conducts multiple initiatives related to building learning healthcare systems, with a particular focus on the role of health information technology (HIT). Among these initiatives is the Health IT for Actionable Knowledge project, which is investigating and reporting on the experiences of multiple large health systems as they incorporate EHRs and other sources of electronic data into clinical care and research. An early report on this project details ways in which participants are leveraging these data to create learning healthcare systems. Other reports related to this initiative are also available from Academy Health, including ones focusing on data quality, legal and policy challenges, and agenda setting, as well as findings from a case study of HIT to support learning healthcare in underserved communities.

Agency for Healthcare Research and Quality

The U.S. Department of Health and Human Services’ Agency for Healthcare Research and Quality (AHRQ) is charged with producing evidence to make healthcare safer, higher quality, and more accessible, equitable, and affordable, and making sure the evidence is understood and used. One of its priorities is accelerating the implementation of patient-centered outcomes research. AHRQ’s portfolio includes the Accelerating Change and Transformation in Organizations and Networks II (ACTION II), which has more than 350 participating organizations and is estimated to reach approximately 50% of the US population, and the Primary Care Practice-Based Research Networks, consisting of primary care clinicians working to address community-based health care questions. The agency’s Effective Health Care Program conducts research through partnerships such as the Developing Evidence to Inform Decisions about Effectiveness (DECiDE) Network and Centers for Education and Research on Therapeutics; the program also includes several groups dedicated to the translation, dissemination, and implementation of clinical evidence into practice.

FDA Mini-Sentinel

The FDA-sponsored Mini-Sentinel (now the Sentinel Initiative) distributed data system project [39,40] embodies several key working aspects of an LHS, including the capability to mine multiple sources of data collected for other purposes (such as routine patient care or clinical research) and combine information from multiple hospitals and health systems to support active, real-time safety monitoring of therapeutics currently approved for marketing, including drugs, biological therapies, and medical devices. The Mini-Sentinel program has developed policies and procedures that permit network participants controlled access to a pool of data that have undergone quality assurance assessment and conform to specified standards and formats. Participants maintain control over their own data and transfer of protected health information is minimized [41].

Kaiser Permanente Washington

Kaiser Permanente Washington (formerly Group Health Cooperative) is a nonprofit integrated health system comprising 25 individual centers and serving nearly 700,000 persons across Washington State. Following the 2007 creation of a “Patient-Centered Medical Home” that served as a pilot NIH Collaboratory Trial, in 2008 the organization applied lessons learned from those initial efforts to re-engineer its entire healthcare system to enable rapid and continuous learning [42]. Additional pilot projects, including one being developed through the NIH Collaboratory, are yielding

…further insights about how research can be immersed in real-world settings to create shared opportunities that can benefit both practice and the scientific community’s knowledge base to achieve transformational learning [42].

Health Care Systems Research Network

The Health Care Systems Research Network (HCSRN) (formerly called HMORN) is a nationwide collaborative network of nearly 20 institutions and health systems that is primarily focused on population health and healthcare delivery. Similar to FDA Mini-Sentinel, HCSRN offers its members access to a distributed data system that allows controlled access to network datasets. The centerpiece of this project is HCSRN’s Virtual Data Warehouse, which provides a template for constructing multiple parallel databases at participating research sites according to a set of shared data and communication standards. Mini-Sentinel, the Cancer Research Network, the Group Health Research Institute's Mental Health Research Network, and the NIH Collaboratory are among HCSRN's many large consortium projects and health system-based research partnerships.

NIH Health Care Systems Research Collaboratory

The NIH Collaboratory is an NIH Common Fund–supported project devoted to building new national infrastructure for collaborative research and enabling the creation of a nationwide LHS. The Collaboratory’s centerpiece is a series of NIH Collaboratory Trials that feature innovative approaches to clinical research utilizing EHRs as data sources across multiple institutions and health systems. Findings from these projects will be “fed back” to create datasets, tools, and best practices that will be organized and disseminated through the Collaboratory’s Coordinating Center.

Veterans Health Administration

The Veterans Health Administration (VHA) is the nation’s largest integrated healthcare system, serving more than 8 million veterans annually at more than 1700 sites. An early adopter of EHRs as part of its VistA system, the VHA has a rich source of data to yield clinical insights and is using its network to evaluate practice and implement evidence-based changes to improve the quality of healthcare for veterans [1]. The VHA’s Quality Enhancement Research Initiative (QUERI), launched in 1998, supports research partnerships in 9 high-impact conditions among veterans, plus eHealth. The VHA also conducts point-of-care clinical trials that randomize patients to an intervention at their usual healthcare provider and require no study-related procedures outside of standard care [43]. The VHA also employs a central IRB to facilitate the conduct of multisite studies.

Bibliography

[Back to top]

1. Institute of Medicine. The Learning Healthcare System: Workshop Summary. Olsen L, Aisner D, McGinnis JM, eds. Washington, DC: National Academies Press; 2007. Available at: http://www.iom.edu/Reports/2007/The-Learning-Healthcare-System-Workshop-Summary.aspx. Accessed April 4, 2014. ↑

2. Institute of Medicine. To Err Is Human: Building a Safer Health System. Kohn LT, Corrigan JM, Donaldson MS, eds. Washington, DC: National Academies Press; 2000. Available at: http://www.iom.edu/Reports/1999/To-Err-is-Human-Building-A-Safer-Health-System.aspx. Accessed April 16, 2014. ↑

3. Committee on Quality of Health Care in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001. Available at: http://www.iom.edu/Reports/2001/Crossing-the-Quality-Chasm-A-New-Health-System-for-the-21st-Century.aspx. Accessed April 16, 2014. ↑

4. Tricoci P, Allen JM, Kramer JM, et al. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 2009;301:831–841. PMID: 19244190. doi: 10.1001/jama.2009.205. ↑

5. Institute of Medicine. Clinical Data as the Basic Staple of Health Learning: Creating and Protecting a Public Good: Workshop Summary. Grossman C, Goolsby AW, Olsen L, et al., eds. Washington, DC: National Academies Press; 2010. Available at: http://www.iom.edu/Reports/2011/Clinical-Data-as-the-Basic-Staple-for-Health-Learning.aspx. Accessed April 4, 2014. ↑

6. Hastings Cent Rep 2013;43:S2–S44. ↑

7. Institute of Medicine. Patients Charting the Course: Citizen Engagement in the Learning Health System: Workshop Summary. Olsen L, Saunders RS, McGinnis JM, eds. Washington, DC: National Academies Press; 2011. Available at: http://www.iom.edu/Reports/2011/Patients-Charting-the-Course-Citizen-Engagement-in-the-Learning-Health-System-Workshop-Summary.aspx. Accessed April 9, 2014. ↑

8. Institute of Medicine. Engineering a Learning Healthcare System: A Look at the Future: Workshop Summary. Grossman C, Goolsby AW, Olsen L, et al., eds. Washington, DC: National Academies Press; 2013. Available at: http://www.iom.edu/Reports/2011/Engineering-a-Learning-Healthcare-System.aspx. Accessed April 4, 2014. ↑

9. Institute of Medicine. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Grossmann C, Powers B, McGinnis JM, eds. Washington, DC: National Academies Press; 2011. Available at: http://www.iom.edu/Reports/2011/Digital-Infrastructure-for-a-Learning-Health-System.aspx. Accessed April 16, 2014. ↑

10. Institute of Medicine. Learning What Works: Infrastructure Required for Comparative Effectiveness Research: Workshop Summary. Olsen L, Grossman C, McGinnis JM, eds. Washington, DC: National Academies Press; 2011. Available at: http://www.iom.edu/Reports/2011/Learning-What-Works-Infrastructure-Required-for-Comparative-Effectiveness-Research.aspx. Accessed April 9, 2014. ↑

11. Institute of Medicine. Digital Data Improvement Priorities for Continuous Learning in Health and Health Care: Workshop Summary. Grossman C, Powers B, Sanders J, eds. Washington, DC: National Academies Press; 2013. Available at: http://www.iom.edu/Reports/2012/Digital-Data-Improvement-Priorities-for-Continuous-Learning-in-Health-and-Health-Care.aspx. Accessed April 9, 2014. ↑

12. Executive Office of the President, President’s Council of Advisors on Science and Technology. Report to the President: Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward. 2010. Available at: http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-health-it-report.pdf. Accessed April 9, 2014. ↑

13. Richesson RL, Hammond WE, Nahm M, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory. J Am Med Inform Assoc 2013;20:e226–e231. PMID: 23956018. doi: 10.1136/amiajnl-2013-001926. ↑

14. Richesson RL, Rusincovitch SA, Wixted D, et al. A comparison of phenotype definitions for diabetes mellitus. J Am Med Inform Assoc 2013;20:e319–e326. PMID: 24026307. doi: 10.1136/amiajnl-2013-001952. ↑

15. Largent EA, Joffe S, Miller FG. Can research and care be ethically integrated? Hastings Cent Rep 2011;41:37–46. PMID: 21845922. ↑

16. Kass NE, Faden RR, Goodman SN, et al. The research-treatment distinction: a problematic approach for determining which activities should have ethical oversight. Hastings Cent Rep 2013;43:S4–S15. PMID: 23315895. doi: 10.1002/hast.133. ↑

17. Henderson GE, Churchill LR, Davis AM, et al. Clinical trials and medical care: defining the therapeutic misconception. PLoS Med 2007;4:e324. PMID: 18044980. doi: 10.1371/journal.pmed.0040324. ↑

18. Baily MA, Bottrell M, Lynn J, et al. The ethics of using QI methods to improve health care quality and safety. Hastings Cent Rep 2006;36:S1–S40. PMID: 16898359. ↑

19. Baily MA. Harming through protection? N Engl J Med 2008;358:768–769. PMID: 18287599. doi: 10.1056/NEJMp0800372. ↑

20. Lynn J, Baily MA, Bottrell M, et al. The ethics of using quality improvement methods in health care. Ann Intern Med 2007;146:666–673. PMID: 17438310. doi: 10.7326/0003-4819-146-9-200705010-00155. ↑

21. Casarett D, Karlawish JH, Sugarman J. Determining when quality improvement initiatives should be considered research: proposed criteria and potential implications. JAMA 2000;283:2275–2280. PMID: 10807388. doi: 10.1001/jama.283.17.2275. ↑

22. Department of Health and Human Services. Human subjects research protections: enhancing protections for research subjects and reducing burden, delay, and ambiguity for investigators. Federal Register 2011;76:44512–44531. ↑

23. Faden RR, Kass NE, Goodman SN, et al. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep 2013;43:S16–S27. PMID: 23315888. doi: 10.1002/hast.134. ↑

24. Faden RR, Beauchamp TL, Kass NE. Informed consent, comparative effectiveness, and learning health care. N Engl J Med 2014;370:766–768. PMID: 24552325. doi: 10.1056/NEJMhle1313674. ↑

25. Largent EA, Miller FG, Joffe S. A prescription for ethical learning. Hastings Cent Rep 2013;43:S28–S29. PMID: 23315890. doi: 10.1002/hast.135. ↑

26. Menikoff J. The unbelievable rightness of being in clinical trials. Hastings Cent Rep 2013;43:S30–S31. PMID: 23315891. doi: 10.1002/hast.136. ↑

27. Kim SYH, Miller FG. Informed consent for pragmatic trials—the integrated consent model. N Engl J Med 2014;370:769–772. PMID: 24552326. doi: 10.1056/NEJMhle1312508. ↑

28. Eapen ZJ, Vavalle JP, Granger CB, et al. Rescuing clinical trials in the United States and beyond: a call for action. Am Heart J 2013;165:837–847. PMID: 23708153. doi: 10.1016/j.ahj.2013.02.003. ↑

29. Infectious Diseases Society of America. Grinding to a halt: the effects of the increasing regulatory burden on research and quality improvement efforts. Clin Infect Dis 2009;49:328–335. PMID: 19566438. doi: 10.1086/605454. ↑

30. U.S. Food and Drug Administration. Guidance for Industry: Using Centralized IRB Review Process in Multicenter Trials. 2006. Available at: http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm080606.pdf. Accessed April 16, 2014. ↑

31. Kramer JM, Smith PB, Califf RM. Impediments to clinical research in the United States. Clin Pharmacol Ther 2012;91:535–541. PMID: 22318614. doi: 10.1038/clpt.2011.341. ↑

32. Sox HC, Goodman SN. The methods of comparative effectiveness research. Annu Rev Public Health 2012;33:425–445. PMID: 22224891. doi: 10.1146/annurev-publhealth-031811-124610.↑

33. Hlatky MA, Winkelmayer WC, Setoguchi S. Epidemiologic and statistical methods for comparative effectiveness research. Heart Fail Clin 2013;9:29–36. PMID: 23168315. doi: 10.1016/j.hfc.2012.09.007. ↑

34. Brookhart MA, Rassen JA, Schneeweiss S. Instrumental Variable Methods in Comparative Safety and Effectiveness Research. Rockville, MD: Agency for Healthcare Research and Quality; 2010. Available at: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=439. Accessed November 21, 2017. ↑

35. U.S. Food and Drug Administration. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. 2009. Available at: http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm193282.pdf. Accessed April 9, 2014. ↑

36. PCORI Methodology Committee. The PCORI Methodology Report. Hickam D, Totten A, Rader K, et al., eds. 2013. Available at: http://www.pcori.org/assets/2013/11/PCORI-Methodology-Report.pdf. Accessed April 9, 2014. ↑

37. Selby JV, Lipstein SH. PCORI at 3 years—progress, lessons, and plans. N Engl J Med 2014;370:592–595. PMID: 24521104. doi: 10.1056/NEJMp1313061. ↑

38. Abernethy AP, Ahmad A, Zafar SY, et al. Electronic patient-reported data capture as a foundation of rapid learning cancer care. Med Care 2010;48:S32–S38. PMID: 20473201. doi: 10.1097/MLR.0b013e3181db53a4. ↑

39. Platt R, Wilson M, Chan KA, et al. The new Sentinel Network—improving the evidence of medical-product safety. N Engl J Med 2009;361:645–647. PMID: 19635947. doi: 10.1056/NEJMp0905338. ↑

40. Behrman RE, Benner JS, Brown JS, et al. Developing the Sentinel System—a national resource for evidence development. N Engl J Med 2011;364:498–499. PMID: 21226658. doi: 10.1056/NEJMp1014427. ↑

41. Platt R, Carnahan RM, Brown JS, et al. The U.S. Food and Drug Administration’s Mini-Sentinel program: status and direction. Pharmacoepidemiol Drug Saf 2012;21(suppl 1):1–8. PMID: 22262586. doi: 10.1002/pds.2343. ↑

42. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med 2012;157:207–210. PMID: 22868839. doi: 10.7326/0003-4819-157-3-201208070-00012. ↑

43. Fiore LD, Brophy M, Ferguson RE, et al. A point-of-care clinical trial comparing insulin administered using a sliding scale versus a weight-based regimen. Clin Trials 2011;8:183–195. PMID: 21478329. doi: 10.1177/1740774511398368. ↑

Topic chapter originally published on June 11, 2014.

• Questions or comments about Rethinking Clinical Trials can be submitted via email. Please add "Living Textbook" to the Subject line of the email.
• Follow the NIH Collaboratory on Twitter!