As described in the previous section, the FDA has provided initial guidance on assessing dataset fitness, outlining the broad concepts of relevance (eg, do the data apply to the question of interest and can they be used in the proposed analysis) and reliability (eg, are the data accurate and complete, is provenance known, and are the data traceable). It is essential to know that there is not a single approach to assessing dataset fitness because it depends on the study and the context in which the data are used.

The authors of a recent empirical study conducted a series of interviews and surveys with PCT study teams to explore their concerns, practices, and decision-making around the fitness for use of real-world data (RWD), with a focus on EHR data. Their analysis showed that, while many PCTs conducted fitness-for-use assessments, less than a quarter did so before choosing a data source. Fitness-for-use activities, findings, and resulting study design changes were not often publicly documented, and overall costs were barriers to assessments. From their analysis, the study authors developed considerations that could help researchers improve the characterization of RWD in PCTs, including (1) defining and articulating how study-specific fitness for use will be assessed, (2) conducting fitness-for-use assessments before the trial begins, and (3) sharing the results of fitness assessments and relevant challenges and facilitators.

An article in the Journal of the American Informatics Association (JAMIA) provides more detail on evaluating fitness-for-use of electronic health records in pragmatic clinical trials (Rahman et al, 2022).

The following case study illustrates the challenges associated with getting approvals for and obtaining RWD for prospective studies (especially internationally, for US-based researchers), and evaluating the general fitness-for use of RWD for trials.

Case Study: Harmony Outcomes EHR Ancillary Study (eHARMONY)

The Harmony Outcomes trial was designed to determine the effect of albiglutide, when added to standard blood glucose-lowering therapies, on major cardiovascular events in patients with type 2 diabetes. eHARMONY was an ancillary study conducted alongside the Harmony Outcomes trial with 3 objectives related to understanding the potential of RWD in clinical trials:

• Understand how EHR data are used to facilitate trial recruitment and the barriers to that use
• Evaluate the fitness of RWD data for use in populating baseline characteristics in the electronic case report form (eCRF)
• Evaluate the fitness of RWD data for use in identifying clinical endpoints (Hammil et al. 2022)

Getting approvals for and obtaining RWD for prospective studies

Originally, this multinational ancillary study planned to include RWD from the United States and several European countries, including the National Hospital Discharge registry (Sweden); the National Health Service Register (Denmark); and the National Health Service hospital discharge data (UK). However, beginning in 2018, the General Data Protection Regulation (GDPR) placed new restrictions on the movement of individual-level data outside the European Union. As a result, planned approaches to international data flow in the study were no longer feasible. Instead, the ancillary study had to rely on Medicare insurance claims data and EHR data from selected US study sites. For the EHR strategy to work, it required sites with a data warehouse based on EHR data; the ability to organize EHR data into a common data format; and an integrated clinical, operational, and technical team. Many of the sites approached for this ancillary study chose not participate because they knew they could not perform this work.

In the eHARMONY ancillary study, assessing a site’s technical capabilities and the quality of their EHR data up front was often not possible. In the end, not all selected sites were able to contribute meaningful data to the study. Among the lessons learned were:

• Standalone clinical research sites had very little extractable EHR data about patients.
• Most lab results and medications were either not extractable or not mapped to a useful terminology.
• Many sites did not have the ability to transform their data into a common format and had to send rudimentary data extracts to the ancillary study coordinating center. Sites participating in other research networks, such as PCORnet, had no difficulty with this task.

Evaluating the general fitness for use of RWD for trials

The study team compared EHR data and Medicare claims data with the trial-collected data and found that agreement varied by data domain.

The study authors had further recommendations for future studies considering incorporating RWD as a data source:

• Define inclusion/exclusion and outcome concepts to be more RWD-friendly. Prevalent disease (eg, cerebrovascular disease, coronary artery disease) is easier to identify than historical clinical events (eg, stroke, myocardial infarction). Focus on what’s available in structured data (eg, hospitalization with primary diagnosis of myocardial infarction), avoiding detailed clinical results (eg, ECG findings).
• When studies have both purpose-collected data (ie, trial database) and RWD, they should perform validation studies as they are able, to contribute to the evidence base for RWD in clinical research.

Watch the Grand Rounds presentation with Q&A: Leveraging RWD in a Multinational Trial: Results from the other eHARMONY (Harmony Outcomes EHR Ancillary Study).