When designing pragmatic trials that include the use of EHR systems and data, it is important to understand and be continually aware at every step —from the conception of the study to the design, conduct, and analysis —that the data obtained from EHR and administrative systems of health care organizations by definition are potentially incomplete and may be influenced or biased by the conditions and incentives of clinical practice. Other limiting factors are the technical constraints of the given EHR system, the amount of time available in a clinical encounter, the focus of the encounter on other matters (such as acute care), and the ability of clinic staff and providers to adequately document all aspects of the encounter that might be important for future research or investigation.

As shown in figure below, the data from EHR and administrative systems can be used as a surrogate for some real event, but do not indicate the presence of a condition or clinical phenomenon with certainty. Different forces (e.g., organizational, sociopolitical, psychological, and technical) influence how clinical observations are made, documented, and interpreted. Each of the steps shown in the figure below is a possible source of information loss or error. Investigators who are designing pragmatic trials should be aware of all of these possible sources of error and bias for the data sources used, and also should identify proactive strategies to reduce the error and its impact on a trial.

Addressing Bias and Lack of Generalizability of EHR Data

Pragmatic research is vulnerable to biases due to differences in data capture and access to care for different subsets of the population, which, if left unaddressed, can create inequities in health and the healthcare system.

The NIH Pragmatic Trials Collaboratory’s Trial teams reflected on the health equity challenges encountered in their trials and shared tactics used to mitigate sources of potential bias and increase the generalizability of research results (Boyd et al. 2023).

“Biased results and poor generalizability can occur because detailed information about specific populations is missing, and critically, is missing not at random: these data are disproportionately missing in diverse and underserved populations." (Boyd et al. 2023)

The NIH Collaboratory Trials are implementing approaches designed to detect and mitigate sources of bias, to ensure inclusion and retention of underrepresented populations, and to enable the complete collection of data that can help identify and support measurement of health inequities.

“By improving data capture, access to care, and patient technology support, ePCTs hold the potential to yield insights and estimates pertinent to the entire population, not just a subset of the population.” (Boyd et al. 2023)

Because data recorded in an EHR provide only a partial and likely incomplete representation of the conditions and events they describe, these data may be appropriate for some uses yet inadequate for others. Therefore, an assessment of data quality from the EHR is essential.

The authors of a paper on the role of the EHR in addressing bias and health inequities delineated examples of challenges and solutions encountered whilst investigating the role of ePCTs in population health problems (Boyd et al. 2023).

Case Examples

• Data on race and ethnicity and social determinants of health (SDOH) are inconsistently and inaccurately captured in EHR systems as part of clinical care.
  • Recommendation: Intentional and systematic collection of both demographic and SDOH variables may help researchers to describe and understand study outcomes more thoroughly. Additionally, providing more support at low-resource sites helps to counter the lack of adoption.
• Differences in access to healthcare persist across settings and populations, which impact the presence and completeness of EHR data for diverse and underserved populations.
  • Recommendation: Utilize various recruitment strategies and intentionally engage with diverse and underserved populations as well as addressing inequities (such as insurance coverage, transportation, and technology barriers) while planning a trial.
• Patients seeking care in low resource settings are less likely to use technology (e.g., email or patient portals) or to provide their own pre-visit data through electronic approaches.
  • Recommendation: Employ a variety of recruitment methods to better reach underserved populations, ensure that interventions are developed at the appropriate reading level and barriers towards understanding interventions are minimized.

Because data recorded in an EHR provide only a partial and likely incomplete representation of the conditions and events they describe, these data may be appropriate for some uses yet inadequate for others. Therefore, an assessment of data quality from the EHR is essential.

In July 20204, the FDA issued a guidance, Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. This guidance, issued as part of the FDA’s Real-World Evidence (RWE) program, applies to clinical studies that use real-world data (RWD) sources, such as information from routine clinical practice, to derive RWE. The guidance “is intended to provide sponsors and other interested parties with considerations when proposing to use electronic health records (EHRs) or medical claims data in clinical studies to support a regulatory decision for effectiveness or safety."

Watch the video module: Assessing Data Quality

Even research quality measurement instruments rarely offer the “truth” about patients’ health status; rather they offer measures that should be reliably reproducible. An understanding of the nature, limitations, and structure of EHR data is important to evaluate its fitness-for-use of these data for different possible uses. Consequently, any consideration for using EHR data in a PCT study should begin with several key questions:

• What exactly is the phenomenon you are trying to identify or measure?
• In what type of health care activity, event, documentation or data value could a “signal” of that phenomenon be detected? (e.g., Ordering of a test? Documenting a diagnosis? Prescribing a medication? Noting an abnormal laboratory test value? Referring a patient to another provider.)
• What are the sources of error for each of those health care activities, events, documentation or data collection? (Who makes the clinical judgement recorded in the EHR? Who orders the clinical test entered into the EHR?)
• How can an investigator assess and reduce that error?

Understanding and Controlling Variation

In traditional clinical trials, the deliberate and prospective collection of data is meant to limit and control any variation in observation and documentation. When using EHR data for research, the researcher loses much of this control. In this context, therefore, it is the obligation of the researcher to identify and understand the variability in the observation and documentation of the data, and control it to the greatest extent possible. Variation between how providers treat various patients and conditions is well known, as is that fact that treatment patterns can vary by geographic regions for a number of reasons (patient population, regulations, costs, training and incentives for providers). When using EHR data for research, new sources of potential variation occur—emerging from, for example, the EHR system used, the interfaces, and ordering or filtering of entry terms. This challenges the fundamental premise of research, which is to understand and control variation. Fortunately, a number of data quality assessment reporting tools (Zozus et al. 2014) and data quality methods (Weiskopf and Weng 2013) provide ways for researchers to compare metrics (between providers) to detect this variation between different data sources or study sites.

Variability of Data Documentation and Clinical Phenomena Across Providers and Sites

Because the structure and representation of clinical data is imposed at the facility according to their standards for clinical documentation and business needs, they are subject to variation across sites. Local and regional variation for different health care activities, events, documentation, treatment patterns, or data collection is to be expected. In addition to regional variation on health care delivery and documentation patterns, there are natural variations across populations from one region to another. This variation, in turn, can be seen in the clusters that are a common design feature in pragmatic trials. For example, one Collaboratory trial involves randomization of clinics located in Hawaii, the Pacific Northwest, and Georgia—and these populations would be expected to be different across a number of variables, such as age, race, and risk exposures. Consequently, differences between clinics could reflect pre-existing regional differences in patient populations, regional differences in practice patterns, or “true” differences attributable to study interventions. However, it is possible to use facility/provider fixed effects to reduce the effect of the provider on measurement IF the investigator believes that the demographic differences in the populations across providers are real and can be adjusted for.  The real problem is IF the measurement error is systematic and biased in one provider vs. another because this makes it challenging to differentiate the effect of the intervention vs. the provider effect.

The quality and completeness of EHR data will vary by EHR, by institution and their workflows, by features of patients and their conditions, and over time. (For example, providers may struggle to assess and document all visit data for complex patients with many chronic conditions.) In multisite trials, different organizational and business approaches (e.g., reimbursement policies, EHR systems) may introduce new sources of variation in addition to provider preferences or treatment variation across providers. Thus, each study may encounter more or less utility from a given set of EHR data depending on the objectives and clinical topic of the study, the design of the trial, and whether the organization collected relevant data during the given study time period. In the design of PCTs, researchers need to identify these types of variation and evaluate the potential impact on the trial.

Bastarache et al. (2021) summarized that “Decisions about data included in the EHR are rarely, if ever, neutral...the nature of what is recorded is highly dependent on who cares for the patient, the intended use of the information, and why they are seeking care in the first place.”

In the sections that follow, we present the process of designing pragmatic trials that include the use of EHR data. The error and bias that are inherent to EHR data limit the use of these data for many research purposes, and the variation that is inherent across sites needs to be assessed and reported. Potential investigators must understand the nature, limitations, and structure of EHR data they plan to (or consider to) use for their research, and this must be done in the context of a specific research question. Therefore, we start with developing and refining the research question and then explore specific uses of EHR data based on the major activities of a trial.