Enrollment or recruitment: Identifying the study population, often in terms of phenotypes or clinical profiles including current health status or medical history
Identifying a cohort of patients for screening, recruitment, or enrollment.
Identifying clusters (clinicians, clinics, etc.) for cluster-randomized or stepped-wedge trials
Assessing baseline prognostic characteristics of the research sample or cohort for a number of objective and subjective measures
In a cluster-randomized design, patient populations may be unbalanced due to facility differences in patient mix. Prognostic factors can be used to detect the balance (or imbalance) of specified independent variables. However, if the factors are used as part of the stratification scheme for cluster-randomization, then this type of balancing is not possible.
Example of an objective measure: definitive lab values for a specific disease, such as chronic kidney disease or HbA1C for diabetes control. (These are subject, of course, to the kinds of limitations noted above. In PCTs, investigators make do with whatever value is in the chart, typically without capturing any information about the clinical lab’s quality control values for these tests.)
Example of a subjective measure: clinical judgement about autism spectrum disorder or an observation by a clinician on skin discoloration or patient discomfort.
Implementing and monitoring the delivery of an intervention (using EHR system functions, such as alerts and computerized provider order entry (CPOE) interfaces)
Measuring the outcomes for both the intervention and control populations.
Longitudinal data linkage—following a cohort of patients over time may require aggregating data from multiple encounters, providers, and information systems. Linkage techniques to ensure that the correct data are being applied to each patient are critical for randomized PCTs, especially if the longitudinal data are being used to measure the effectiveness of the intervention.
In the following sections, we describe each of these uses for EHR data by PCT activity.
As with any type of research study, PCTs begin with a scientific question. A clearly articulated research question defines the phenomena of interest, the purpose for using EHR data, the possible sources of data to detect that phenomena, and, more specifically, the data requirements, definitions, quality, and data collection plan. In practice, however, there is often an iterative process between defining the data requirements for the EHR and understanding what is actually available at a given institution. The researcher must consider what information is needed to answer the question, and in turn, the available data may then influence the research question (i.e., ongoing cycles of this conversation: Health System: What data do you need for the trial? Researcher: That depends… What data is available?) This dialogue may lead investigators to refine their research question slightly to one that is likely more “answerable” based upon what data are collected or available, and is part of the process of Building Partnerships to Ensure a Successful Trial. Anecdotally, many researchers see the potential for discovery in clinical data warehouses derived from EHRs, and are excited to use the available data to generate questions and answers. While this approach is understandable and practical in theory, the complexity of EHR architectures, and their inherent bias and possible error, could produce inaccurate data and lead to misinterpretation of results and erroneous conclusions. Consequently, we assert that the scientific research question should be the fundamental driver for the study design and hence the foundation for any PCT.
Good clinical research practice and ethics dictate that clinical trials collect the necessary data (and ONLY the necessary data) to answer a specific research question (FDA Good Clinical Practice Guidance 2018). In most PCTs, it is vitally important to identify and measure co-variates between study arms or clusters. In these cases, there need to be high quality (accurate, complete) data from a number of variables so that researchers can assess the comparability between groups. The objective is to achieve balance between the groups along as many dimensions that are relevant, important, and feasible.
Because developing a PCT that will use data from the EHR can be extremely complicated, the NIH Collaboratory has developed a set of papers and chapters that provide a deeper dive into many of the issues, including developing data definitions and phenotyping, acquiring real-world data, and assessing fitness-for use.
A resource chapter describing mechanisms for identifying and evaluating phenotype definitions, with a particular focus on standardization efforts from the Collaboratory
This Living Textbook chapter describes several approaches that can be used to facilitate assessments to determine whether data are fit for their intended use or purpose prior to their use in research settings
U.S. Food and Drug Administration. Guidance Document: E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1). 2018. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e6r2-good-clinical-practice-integrated-addendum-ich-e6r1. Assessed December 3, 2025.
When designing pragmatic trials that include the use of EHR systems and data, it is important to understand and be continually aware at every step —from the conception of the study to the design, conduct, and analysis —that the data obtained from EHR and administrative systems of health care organizations by definition are potentially incomplete and may be influenced or biased by the conditions and incentives of clinical practice. Other limiting factors are the technical constraints of the given EHR system, the amount of time available in a clinical encounter, the focus of the encounter on other matters (such as acute care), and the ability of clinic staff and providers to adequately document all aspects of the encounter that might be important for future research or investigation.
As shown in figure below, the data from EHR and administrative systems can be used as a surrogate for some real event, but do not indicate the presence of a condition or clinical phenomenon with certainty. Different forces (e.g., organizational, sociopolitical, psychological, and technical) influence how clinical observations are made, documented, and interpreted. Each of the steps shown in the figure below is a possible source of information loss or error. Investigators who are designing pragmatic trials should be aware of all of these possible sources of error and bias for the data sources used, and also should identify proactive strategies to reduce the error and its impact on a trial.
Addressing Bias and Lack of Generalizability of EHR Data
Pragmatic research is vulnerable to biases due to differences in data capture and access to care for different subsets of the population, which, if left unaddressed, can create inequities in health and the healthcare system.
The NIH Pragmatic Trials Collaboratory’s Trial teams reflected on the health equity challenges encountered in their trials and shared tactics used to mitigate sources of potential bias and increase the generalizability of research results (Boyd et al. 2023).
“Biased results and poor generalizability can occur because detailed information about specific populations is missing, and critically, is missing not at random: these data are disproportionately missing in diverse and underserved populations." (Boyd et al. 2023)
The NIH Collaboratory Trials are implementing approaches designed to detect and mitigate sources of bias, to ensure inclusion and retention of underrepresented populations, and to enable the complete collection of data that can help identify and support measurement of health inequities.
“By improving data capture, access to care, and patient technology support, ePCTs hold the potential to yield insights and estimates pertinent to the entire population, not just a subset of the population.” (Boyd et al. 2023)
Because data recorded in an EHR provide only a partial and likely incomplete representation of the conditions and events they describe, these data may be appropriate for some uses yet inadequate for others. Therefore, an assessment of data quality from the EHR is essential.
The authors of a paper on the role of the EHR in addressing bias and health inequities delineated examples of challenges and solutions encountered whilst investigating the role of ePCTs in population health problems (Boyd et al. 2023).
Case Examples
Data on race and ethnicity and social determinants of health (SDOH) are inconsistently and inaccurately captured in EHR systems as part of clinical care.
Recommendation: Intentional and systematic collection of both demographic and SDOH variables may help researchers to describe and understand study outcomes more thoroughly. Additionally, providing more support at low-resource sites helps to counter the lack of adoption.
Differences in access to healthcare persist across settings and populations, which impact the presence and completeness of EHR data for diverse and underserved populations.
Recommendation: Utilize various recruitment strategies and intentionally engage with diverse and underserved populations as well as addressing inequities (such as insurance coverage, transportation, and technology barriers) while planning a trial.
Patients seeking care in low resource settings are less likely to use technology (e.g., email or patient portals) or to provide their own pre-visit data through electronic approaches.
Recommendation: Employ a variety of recruitment methods to better reach underserved populations, ensure that interventions are developed at the appropriate reading level and barriers towards understanding interventions are minimized.
Because data recorded in an EHR provide only a partial and likely incomplete representation of the conditions and events they describe, these data may be appropriate for some uses yet inadequate for others. Therefore, an assessment of data quality from the EHR is essential.
In July 20204, the FDA issued a guidance, Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. This guidance, issued as part of the FDA’s Real-World Evidence (RWE) program, applies to clinical studies that use real-world data (RWD) sources, such as information from routine clinical practice, to derive RWE. The guidance “is intended to provide sponsors and other interested parties with considerations when proposing to use electronic health records (EHRs) or medical claims data in clinical studies to support a regulatory decision for effectiveness or safety."
Watch the video module: Assessing Data Quality
Even research quality measurement instruments rarely offer the “truth” about patients’ health status; rather they offer measures that should be reliably reproducible. An understanding of the nature, limitations, and structure of EHR data is important to evaluate its fitness-for-use of these data for different possible uses. Consequently, any consideration for using EHR data in a PCT study should begin with several key questions:
What exactly is the phenomenon you are trying to identify or measure?
In what type of health care activity, event, documentation or data value could a “signal” of that phenomenon be detected? (e.g., Ordering of a test? Documenting a diagnosis? Prescribing a medication? Noting an abnormal laboratory test value? Referring a patient to another provider.)
What are the sources of error for each of those health care activities, events, documentation or data collection? (Who makes the clinical judgement recorded in the EHR? Who orders the clinical test entered into the EHR?)
How can an investigator assess and reduce that error?
Understanding and Controlling Variation
In traditional clinical trials, the deliberate and prospective collection of data is meant to limit and control any variation in observation and documentation. When using EHR data for research, the researcher loses much of this control. In this context, therefore, it is the obligation of the researcher to identify and understand the variability in the observation and documentation of the data, and control it to the greatest extent possible. Variation between how providers treat various patients and conditions is well known, as is that fact that treatment patterns can vary by geographic regions for a number of reasons (patient population, regulations, costs, training and incentives for providers). When using EHR data for research, new sources of potential variation occur—emerging from, for example, the EHR system used, the interfaces, and ordering or filtering of entry terms. This challenges the fundamental premise of research, which is to understand and control variation. Fortunately, a number of data quality assessment reporting tools (Zozus et al. 2014) and data quality methods (Weiskopf and Weng 2013) provide ways for researchers to compare metrics (between providers) to detect this variation between different data sources or study sites.
Variability of Data Documentation and Clinical Phenomena Across Providers and Sites
Because the structure and representation of clinical data is imposed at the facility according to their standards for clinical documentation and business needs, they are subject to variation across sites. Local and regional variation for different health care activities, events, documentation, treatment patterns, or data collection is to be expected. In addition to regional variation on health care delivery and documentation patterns, there are natural variations across populations from one region to another. This variation, in turn, can be seen in the clusters that are a common design feature in pragmatic trials. For example, one Collaboratory trial involves randomization of clinics located in Hawaii, the Pacific Northwest, and Georgia—and these populations would be expected to be different across a number of variables, such as age, race, and risk exposures. Consequently, differences between clinics could reflect pre-existing regional differences in patient populations, regional differences in practice patterns, or “true” differences attributable to study interventions. However, it is possible to use facility/provider fixed effects to reduce the effect of the provider on measurement IF the investigator believes that the demographic differences in the populations across providers are real and can be adjusted for. The real problem is IF the measurement error is systematic and biased in one provider vs. another because this makes it challenging to differentiate the effect of the intervention vs. the provider effect.
The quality and completeness of EHR data will vary by EHR, by institution and their workflows, by features of patients and their conditions, and over time. (For example, providers may struggle to assess and document all visit data for complex patients with many chronic conditions.) In multisite trials, different organizational and business approaches (e.g., reimbursement policies, EHR systems) may introduce new sources of variation in addition to provider preferences or treatment variation across providers. Thus, each study may encounter more or less utility from a given set of EHR data depending on the objectives and clinical topic of the study, the design of the trial, and whether the organization collected relevant data during the given study time period. In the design of PCTs, researchers need to identify these types of variation and evaluate the potential impact on the trial.
Bastarache et al. (2021) summarized that “Decisions about data included in the EHR are rarely, if ever, neutral...the nature of what is recorded is highly dependent on who cares for the patient, the intended use of the information, and why they are seeking care in the first place.”
In the sections that follow, we present the process of designing pragmatic trials that include the use of EHR data. The error and bias that are inherent to EHR data limit the use of these data for many research purposes, and the variation that is inherent across sites needs to be assessed and reported. Potential investigators must understand the nature, limitations, and structure of EHR data they plan to (or consider to) use for their research, and this must be done in the context of a specific research question. Therefore, we start with developing and refining the research question and then explore specific uses of EHR data based on the major activities of a trial.
This paper by Dr. Greg Simon et al. proposes specific questions regarding potential error or bias at different stages between the clinical phenomenon of interest and its representation in a research database.
This Living Textbook chapter describes several approaches that can be used to facilitate assessments to determine whether data are fit for their intended use or purpose prior to their use in research settings.
Bastarache, L, Brown, JS, Cimino, JJ, et al. 2022. Developing real-world evidence from real-world data: Transforming raw data into analytical datasets. Learn Health Sys. 6(1):e10293. doi:10.1002/lrh2.10293.
Boyd AD, Gonzalez-Guarda R, Lawrence K, et al. 2023. Potential bias and lack of generalizability in electronic health record data: reflections on health equity from the National Institutes of Health Pragmatic Trials Collaboratory. Journal of the American Medical Informatics Association. 30:1561–1566. doi:10.1093/jamia/ocad115.
Hripcsak G, Elhadad N, Chen Y-H, Zhou L, Morrison FP. 2009. Using Empiric Semantic Correlation to Interpret Temporal Assertions in Clinical Texts. J Am Med Inform Assoc. 16:220–227. doi:10.1197/jamia.M3007. PMID: 19074297.
Weiskopf NG, Weng C. 2013. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 20:144–151. doi:10.1136/amiajnl-2011-000681. PMID: 22733976.
Zozus MN, Hammond WE, Green BB, et al. 2014. Assessing Data Quality for Healthcare Systems Data Used in Clinical Research.
Version History
October 7, 2025: Updated text as part of annual review (changes made by K. Staman).
April 16, 2023: Added information about Boyd et al paper as part of annual review (changes made by K. Staman).
August 26, 2022: Updates made as part of annual review (changes made by K. Staman).
January 22, 2021: Added embedded video (change made by G. Uhlenbrauck).
July 3, 2020: Minor corrections to layout and formatting (changes made by D. Seils).
November 30, 2018: Updated text as part of annual update and added a resource (changes made by K. Staman).
Provides a practical overview of data monitoring in clinical trials, including the purpose, responsibilities, and operation of DMCs. Useful for those managing and conducting clinical trials and those serving on DMCs or regulatory and ethical committees.
Describes practical challenges of collecting and reporting safety data and of monitoring trial conduct while maintaining routine clinical care practice.
While not specific to ePCTs, the recommendations address the role of the DMC, DMC composition, communication among DMC members and other stakeholders, the DMC charter, and issues related to DMC training.
This page provides background and links to a series of 12 articles on the ethics and regulatory challenges in pragmatic clinical trials. Each article in the special issue of Clinical Trials describes an issue in detail (e.g., privacy, informed consent) and, where possible, attempts to provide guidance for future pragmatic clinical trials.
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
Determining a feasible and acceptable data monitoring plan can be a major barrier to getting an embedded (ePCT) through the planning phase (see SPOT investigator interview). ePCTs may use cluster-randomized designs, which raise special statistical issues. DMCs for cluster-randomized trials need one or more statistical members who are knowledgeable about these issues. Training for all DMC members on the particular issues of monitoring ePCTs is also expected to be helpful.
As described throughout this chapter, some typical monitoring practices used for traditional trials may need to be rethought when applied to ePCTs. The NIH Collaboratory plans to develop training resources that can be used to educate DMCs on the intricacies of monitoring ePCTs. Until such training is available, the investigator may need to expend extra effort holding open discussions with the sponsor and DMC during the planning phase to reach a satisfactory monitoring plan.
The NIH Collaboratory Ethics and Regulatory Core has developed a DMC charter template that can be used for ePCTs. A charter defines the primary responsibilities of a DMC, its membership, the purpose and timing of its meetings, and its procedures and statistical monitoring guidelines. A charter customized for ePCTs may include specific requirements, such as the need for at least one DMC member to have prior experience in conducting and interpreting data from ePCTs.
Data monitoring for ePCTs is evolving, as more ePCTs are conducted and more is learned about special issues that might need to be considered.
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
As for any trial, the DMC should include an experienced statistician and medical experts from the setting being monitored.
Given the focus on patient-centered outcomes in PCTs, the role of a patient representative [on DMCs] may be particularly important. — Ellenberg et al 2015
Patient representatives can offer a unique and valuable perspective on the emerging risk-benefit profile during the trial. Other expertise that may be advisable include a biomedical informatician.
Resources for Integrating Stakeholder Perspectives
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
The study team for the Trauma Survivors Outcomes and Support (TSOS) study, an NIH Collaboratory Trial, had a proactive site visit and regulatory review with their sponsor liaison to the DSMB from the National Institute of Mental Health. The site visit was treated as a knowledge exchange and was helpful to both the TSOS study team and the sponsor. The visit helped the TSOS team work out issues such as how to obtain consent from patients remotely, the process for review of serious adverse events, and reporting to and interactions with the trial's central IRB.
The TSOS principal investigator was concerned about having some level of oversight for adverse events that did not meet criteria for expedited submission to the IRB. The NIH Collaboratory Ethics and Regulatory Core suggested that the DSMB might be able to provide an appropriate safety check in these instances, either with DSMB review of these events in aggregate or by a subset of the DSMB (eg, chair) of individual adverse events when necessary.
The site visit required a lot of preparation and differed from traditional site visits in that there was more focus on information technology (IT) aspects of the study. The sponsor representative had experience with electronic data capture systems and was able to provide some helpful input. After the discussions, the TSOS team was tasked with finalizing tables for data sorting and cleaning for the DSMB. The principal investigator expressed a need to be careful with the level of data cleaning, because the PCT is not resourced for staff to spend too much time on these tasks.
Conducting a site visit and follow-up discussions showed how a proactive dialog on data monitoring between the sponsor and investigator can benefit both entities and help to reach a mutually acceptable monitoring plan.
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
In some ePCTs, serious adverse events (SAEs) such as death may be an expected outcome (eg, suicide prevention trials [Sisti et al 2018] or studies with very ill populations). As in a traditional RCT with such a population, monitoring of individual SAEs is not likely to be helpful; however, monitoring comparative rates of SAEs between treatment arms can help to ensure that the study intervention is not causing more SAEs.
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
The source of trial data for a PCT should be considered when determining the monitoring plan. Two main issues for PCTs are data quality and timeliness.
Data Quality
When using the EHR as the source of trial data, monitoring for data quality early in the trial can help ensure that valid results will be obtained (Ellenberg et al 2015). Data reported by clinicians in the everyday care setting may have more variability than DMCs are accustomed to seeing with traditional trials, which have highly controlled settings, strict reporting protocols, and separate systems for documentation (e.g., case report forms). In traditional trials, every effort is made to ensure consistency of measurement across sites. More heterogeneity may be observed in PCTs due to real-world variations in populations and approaches used by healthcare delivery organizations and their practicing clinicians.
Thus, DMCs may need to pay close attention to site-specific data to determine whether an emerging result may be attributable to one or two sites and is perhaps not widely generalizable. — Ellenberg et al 2015
As an element of data quality, completeness of data can also be a challenge in PCTs. Pragmatic trial protocols may allow flexibility in follow-up according to standard clinical practices at participating sites. The DMC and sponsor will need to agree on necessary data for follow-up in accordance with the risks of the trial intervention. Variations in follow-up practices across sites may also need to be accounted for in the randomization process to ensure that sites with more frequent follow-up practices are balanced across study arms.
Data Timeliness
Data obtained from sources such as claims data, state mortality data, or even the EHR will often have a delay and not be available real-time. In some trials, sites may perform data analysis locally due to privacy concerns, then submit results for centralized analysis and aggregation. This process, along with necessary data quality checks, can result in additional time before data are available for review (Ellenberg et al 2015). Some determination will need to be made regarding the amount of delay that is acceptable to ensure adequate participant protection. DMCs should be involved in these discussions.
Case Example
In a traditional study of a mental health intervention, a suicide attempt or suicide death would be considered a serious adverse event, calling for immediate reporting to the DSMB and a DSMB determination regarding relatedness of the event to study treatment. However, in the SPOT suicide prevention trial, information on deaths is obtained from state mortality data, which are delayed up to 16 months. Death was also an expected occurrence in this population at high risk for suicide. In this situation, it would not be sensible to stop the trial for investigation of a death 16 months after the death occurred. These special circumstances required negotiation with the DSMB to work out a more practical monitoring plan.
Contributing Editors Gina Uhlenbrauck Liz Wing, MA
It is sometimes argued that PCTs, as opposed to traditional trials, should do very little to address adherence within the trial, because adherence to an intervention can be considered a representation of how well the intervention would be implemented in the clinical setting. However, without information about adherence, it will be impossible to interpret trial results and determine whether a change (or lack of change) in outcomes was in fact due to the trial intervention.
Due to potential changes in clinical care that occur in healthcare delivery settings, monitoring for use of non-protocol interventions and the difference in adherence rates between the control and intervention arms can be an important part of the monitoring plan for ePCTs. The table below describes an example from the NIH Collaboratory Trials.
Example of Adherence Monitoring in PCTs
Trial
Adherence monitoring
ICD-Pieces
The trial monitors adherence to the intervention by reporting use of some key components of the intervention, including use of angiotensin converting enzyme inhibitors/angiotensin receptor blockers, use of statins, and checking hemoglobin A1c.
