Many pragmatic clinical trials involve outcomes or endpoints that are available as part of routine care and are obtained using real-world data, including data from EHRs or claims data. Although the use of real-world data in pragmatic trials can improve the relevance and efficiency of the research, the use of these secondary research data, or data not originally designed for research, can also present challenges for data monitoring. Below, we describe 2 relevant considerations for DSMBs when monitoring pragmatic trials that use real-world data: data quality and data timeliness. (See also the Assessing Fitness for Use of Real-World Data Sources chapter of the Living Textbook.)

Data Quality

Unlike in explanatory trials, in which the research team often controls the entire chain of custody for outcome data, directly ascertaining study outcomes that are then recorded in study-specific databases, outcome data in pragmatic clinical trials are instead often extracted from real-world data generated from routine healthcare operations (Simon et al 2019).

The use of real-world data in pragmatic trials raises several challenges relevant for DSMBs. First, data generated from routine healthcare operations, such as those reported by clinicians during routine clinical interactions, may have more variability than those generated in explanatory research contexts, which typically rely on strict reporting protocols using separate systems for documentation (such as case report forms). Monitoring for data quality and integrity issues related to inconsistent measurement across study arms or sites early in the trial may be needed to help ensure that the trial can generate valid results.

Data in pragmatic trials may also be more heterogenous across sites, because of real-world variations in populations and approaches used by healthcare delivery organizations and their practicing clinicians. Consequently, DSMBs may need to pay close attention to site-specific data, to inform assessments of whether an emerging result is generalizable or may instead be attributable to a limited number of sites (Ellenberg et al 2015). For example, EMBED (Melnick et al 2019; Melnick et al 2022) researchers found that coding practices related to patient race and ethnicity varied not only from system to system but also within systems, which could undermine validity and the ability to assess whether the data are generalizable (Curtis et al 2025). To monitor for the issue, EMBED researchers increased the number of data pulls for review(Curtis et al 2025). As another example, the ACP PEACE trial planned to use the electronic health record to examine documentation of advance care planning (Volandes et al. 2025). However, chart review revealed that fewer than 60% of patients had correct documentation, forcing investigators to pivot to another method (natural language processing) to detect documentation (Curtis et al 2025).

A related challenge is that of data completeness. Unlike traditional explanatory trials, which may involve intense follow-up visits and extensive data collection, some pragmatic trials involve no more follow-up than would be normal in usual care, or may allow flexibility in follow-up according to standard clinical practices at the participating sites. The DSMB, investigators, and study sponsor will need to agree on necessary data for follow-up in accordance with the risks of the trial intervention. Variations in follow-up practices across sites may also need to be accounted for in the randomization process to ensure that sites with more frequent follow-up practices are balanced across study arms.

Finally, data recording practices (or even entire data systems) may change during the course of the trial, due to changes in the business practices or EHR systems of the participating trial sites (Simon et al 2019). In such cases, DSMBs should be prepared to collaborate with investigators and sponsors to assess how best to mitigate threats to study validity.

Data Timeliness

Data obtained from sources such as claims data, state mortality data, or even the EHR will often have a delay and not be available in real time. In some trials, sites may perform data analysis locally due to privacy concerns, then submit results for centralized analysis and aggregation. This process, along with necessary data quality checks, can result in additional time before data are available for review (Ellenberg et al 2015). In some cases, these delays may limit the feasibility of interim analyses to inform decisions about early termination based on considerations of futility, safety, or effectiveness. DSMBs and study investigators should prospectively consider the implications of these delays when developing plans for data monitoring.

Case Example

In a traditional study of a mental health intervention, a suicide attempt or suicide death would be considered a serious adverse event, calling for immediate reporting to the DSMB and a determination by the DSMB regarding the relatedness of the event to study treatment. However, in the SPOT suicide prevention trial, an NIH Pragmatic Trials Collaboratory Trial, information on deaths was obtained from state mortality data, which were delayed up to 16 months. Death was also an expected occurrence in this population at high risk for suicide. In this situation, it would not be sensible to stop the trial for investigation of a death 16 months after the death occurred. These special circumstances required negotiation with the DSMB to work out a more practical monitoring plan.