Living Textbook

Acquiring Claims Data and CMS Research-Identifiable Files

CHAPTER SECTIONS

Acquiring Real-World Data


Section 5

Acquiring Claims Data and CMS Research-Identifiable Files

Contributors

Administrative claims are another secondary data source for collecting event information from healthcare systems. PCTs embedded in health insurance plans can be used to answer specific questions, such as whether an intervention is effective in different geographical locations, populations, and multiple complex organizations (Cocoros et al. 2023). These trials are best suited to studies that require large sample sizes. Health insurance data can be used to identify eligible individuals, facilitate patient and provider contact, and/or analyze the study outcomes.

“There are unique opportunities related to the design and conduct of pragmatic trials embedded in health insurance plans, which have longitudinal data on member/patient demographics, dates of coverage, and reimbursed medical care, including prescription drug dispensings, vaccine administrations, behavioral healthcare encounters, and some laboratory results.” (Cocoros et al. 2023)

Although trials embedded in health insurance plans hold the potential to generate evidence to improve care and population health, there are special challenges that must be considered in the planning, implementation, and analytic phases (Cocoros et al. 2023). Important logistical challenges require careful planning, including planning for timing (plan enrollment and disenrollment is typically at the beginning and end of a calendar year), lag time for data availability, and engagement of staff from health plans and providers. The intervention itself must also be fairly simple, as interventions will be disseminated through health plans.

In addition, the accuracy of billed diagnoses in identifying potential events has been shown to be less reliable than physician-adjudicated events (Guimarães et al 2017). For example, in the Treatment With Adenosine Diphosphate Receptor Inhibitors: Longitudinal Assessment of Treatment Patterns and Events After Acute Coronary Syndrome (TRANSLATE-ACS) trial, investigators compared the 1-year incidence of events after acute myocardial infarction as identified by medical claims or physician adjudication. They found modest agreement for myocardial infarction and stroke and poor agreement for bleeding (Guimarães et al 2017).

There are several sources of claims data that can be used for research:

  • Medicare claims data, which include data from Medicare beneficiaries who enroll in the traditional fee-for-service Medicare program (and do not include data from patients who enroll in Medicare Advantage plans).
  • Medicare Advantage data, which include both Part A and Part B claims but are provided by private insurance companies and, therefore, are not included in the data sources described below.
  • Claims from participants enrolled in Medicaid or the Children’s Health Insurance Program (CHIP).
  • Collected bills from private insurance companies. For example, in the ADAPTABLE study (Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness), investigators engaged with 2 large, national insurance companies to support record linkage for participating members. (See the Using Electronic Health Record Data chapter of the Living Textbook for more information.)
  • Collected bills from a patient’s inpatient care facilities. (See the TRANSLATE-ACS case study.)

It is important to note that, for the data sources described below, patients provide consent or otherwise authorize data to be provided to a study. This is different from some workflows in which EHR data are used for research, which may not require direct consent from the patient.

Research-Identifiable Files

The traditional method for obtaining Centers for Medicare & Medicaid Services (CMS) data for research is a formal request process and shipment of data files. Data can be requested from the Medicare program, which covers 95% of people aged 65 years and older, or the Medicaid program, which covers children from low-income families, pregnant women, people with disabilities, and some elderly and nonelderly adults, although coverage differs by state, since Medicaid programs are state-run. Researchers can request the public-use data set, a limited data set with deidentified data, or a research-identifiable file with individual-level data. For use in trial follow-up, research-identifiable files are the only option; to obtain research-identifiable files for individuals, an investigator must obtain protected health information, such as Social Security number and date of birth, and send it to the CMS data distributor for linking. This adds difficulty to the process. Although these data are well curated, gaining access to the data can be expensive, it can be time consuming to go through the CMS request process, and data latency can be an issue (Marsolo 2019).

Application Programming Interfaces

As with EHRs, many administrative claims sources provide the ability for participants to obtain copies of their data. Blue Button was created by the US Department of Health and Human Services as an online tool that allows patients to view, print, and download their medical records (Turvey et al 2014) and was intended to help with coordination of care. Blue Button is available on the patient portal for Medicare beneficiaries (MyMedicare.gov), for veterans (MyHealtheVet), and on the patient portals of those practices and clinics that choose to use it. Medicare beneficiaries can download 3 years of claims data, and veterans can download “demographic information (age, gender, ethnicity and more), emergency contacts, a list of their prescription medications, clinical notes, and wellness reminders.” (From https://www.healthit.gov/topic/health-it-initiatives/blue-button). With Blue Button, the patient provides the data for research; however, the completeness of the data varies by site and EHR. The document is a structured, text-based document (an XML file) and needs to be parsed through an app to be used for research (such as Hugo), and a patient would need to request a file from each site where they receive care (Marsolo 2019).

The CMS Blue Button 2.0 application programming interface (API) enables Medicare beneficiaries to authorize third parties to obtain and use their Part A, Part B, and Part D claims data directly from CMS (as opposed to through the Blue Button patient portal), for coordinating care, services, and research. It uses the FHIR-standard API. A Final Rule from the ONC and CMS mandates that CMS-regulated payers make claims available via FHIR.

Collected Participant Bills

Explanatory trial economic and quality-of-life studies frequently collect and abstract participant bills from study sites. This process is expensive and requires specially trained individuals with expertise in hospital billing and accounting systems. Hence, it would not be preferred for a large pragmatic trial, though it may be necessary if the relevant information cannot be obtained in another way.

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

Resources

Data Infrastructure for Implementing the PROVEN Trial
NIH Collaboratory EHR Workshop video module. Vince Mor of Brown University describes the PROVEN trial’s use of electronic health records linked with Medicare claims to measure outcomes of a nursing home–based intervention.

Screenshot of Grand Rounds presentation
PCORnet COVID-19 Common Data Model Design and Results
NIH Pragmatic Trials Collaboratory PCT Grand Rounds; June 5, 2020

REFERENCES

back to top

Beckman AL, Gupta S. Empowering people with their healthcare data: an Interview with Harlan Krumholz. Healthc (Amst). 6:238-239. doi:10.1016/j.hjdsi.2018.08.002. PMID: 30143459.

Dhruva SS, Mena-Hurtado C, Curtis J, et al. 2019. Learning how to successfully enroll and engage people in a mobile sync-for-science platform to inform shared decision making. J Am Coll Cardiol. 73:3039. doi:10.1016/S0735-1097(19)33645-9.

Guimarães PO, Krishnamoorthy A, Kaltenbach LA, et al. 2017. Accuracy of medical claims for identifying cardiovascular and bleeding events after myocardial infarction : a secondary analysis of the TRANSLATE-ACS study. JAMA Cardiol. 2(7):750-757. doi: 10.1001/jamacardio.2017.1460. PMID: 28538984.

Marsolo K. 2019. Approaches to Patient Follow-Up for Clinical Trials: What’s the Right Choice for Your Study? NIH Pragmatic Trials Collaboratory PCT Grand Rounds; March 1, 2019. Available at: https://rethinkingclinicaltrials.org/news/approaches-to-patient-follow-up-for-clinical-trials-whats-the-right-choice-for-your-study-keith-marsolo-phd/. Accessed October 14, 2022.

back to top

McCarthy EP, Chang CH, Tilton N, Kabeto MU, Langa KM, Bynum JPW. 2022. Validation of claims algorithms to identify Alzheimer's disease and related dementias. J Gerontol A Biol Sci Med Sci. 77(6):1261-1271. doi: 10.1093/gerona/glab373. PMID: 3491968.

Turvey C, Klein D, Fix G, et al. 2014. Blue Button use by patients to access and share health record information using the Department of Veterans Affairs' online patient portal. J Am Med Inform Assoc. 21(4):657-63. doi: 10.1136/amiajnl-2014-002723. PMID: 24740865.


Version History

October 14, 2022: Made nonsubstantive changes to the text, added an image to the Resources sidebar, added Seils as a contributing editor, and reordered the section within the chapter as part of the annual content update (changes made by D. Seils).

February 7, 2022: Added example sentence on using CMS data to identify patients with dementia (changes made by K. Staman).

January 15, 2021: Moved this section from “Inpatient Endpoints” to “Acquiring Real-World Data”, added EHR Workshop video module to resource bar, updated proposed rule to final rule (changes made by K. Staman).

Published June 19, 2019.

current section :

Acquiring Claims Data and CMS Research-Identifiable Files

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

Acquiring Electronic Health Record Data

CHAPTER SECTIONS

Acquiring Real-World Data


Section 4

Acquiring Electronic Health Record Data

Contributors

One problem with relying on electronic health record (EHR) data as part of a longitudinal data collection strategy is obtaining data from all the sites where a patient is treated. How does the study team obtain EHR data from a hospital that is outside the participating healthcare system? If a patient has a heart attack while on vacation, how will the study team capture that information? It is generally recommended to use multiple mechanisms to obtain secondary use data on longitudinal outcomes, such as directly from patients via a mobile app or call center, or through sources like administrative claims.

An important factor with EHR data, compared with sources like administrative claims, is that there is considerable variation in the ways data are captured in the EHR, as well as the terminologies used to represent those data. This challenge makes harmonization a key step in the use of EHR data. Although the Office of the National Coordinator for Health Information Technology (ONC) is creating a framework for greater standardization and research capacity for EHR systems, the use of these standards is variable across the healthcare industry (ONC 2022).

The “Data Formats” section of this chapter describes some of the common data formats that are used to represent real-world data. This section describes some of the challenges that arise in obtaining EHR data from the enrolling healthcare system and other healthcare systems for 3 of the most frequently used approaches: (1) database extraction from individual sites, (2) use of a common data model (CDM), and (3) use of an application programming interface (API).

Database Extract

One of the more straightforward approaches for obtaining EHR data is to work with a site’s IT staff or an EHR vendor–based IT resource to generate a database extract at a given site. These extracts, which often are delivered as flat files or database tables, can be large in scale and automated, but complex queries rely on the skills of the local analyst, and extracts are more difficult to do at smaller sites that may not have the resources (Marsolo 2019). Hence, while this method requires fewer site resources than something like a CDM, it still may have limited applicability in multicenter pragmatic trials.

Common Data Model

Information can be recorded in different ways across sites (and across diagnoses). Sites that participate in distributed research networks (such as PCORnet and Sentinel) and national registries (such as the American College of Cardiology’s National Cardiovascular Data Registry) have agreed to harmonize their source data in a prespecified way by using a CDM. CDMs that are populated from sources like EHRs and administrative claims typically contain data that are captured in those sources in a structured format (such as diagnoses, procedures, laboratory results, medication orders, and medication administrations). Information captured in an unstructured format or with varying reliability may not be present in a CDM. For example, if a cause for an event is needed, the level of specificity required may or may not be available in the CDM because of variability in how that information is documented in the source system. In such cases, manual chart review may be a more efficient way of obtaining the cause of an event.

CDMs often use standard controlled terminologies to represent data, such as ICD-10 codes for diagnoses or RxNorm or the National Drug Code (NDC) for medications. However, these terminologies alone do not guarantee that data from different facilities are comparable. For example, different site coding practices can cause differences in whether patients are counted as having one condition vs another. Moreover, CDMs require mapping data from an EHR repository or institutional data warehouse to the target definition. In many cases, the latter is a more abstract model, and useful context and other detail can be lost in mapping the data to the CDM (Garza et al 2016). As an example, the encounter endpoints that include hospitalization for PCORnet and Sentinel’s CDMs are described below.

Sentinel is a US medical product surveillance system designed to monitor medical products regulated by the US Food and Drug Administration. The Patient-Centered Outcomes Research Institute (PCORI) funded the National Patient-Centered Clinical Research Network (PCORnet) to build a network-of-networks to support clinical research. Their CDM was built based on the Sentinel model but was extended to support many of the data elements found in EHRs. The table below shows the “type of encounter” definitions for the Sentinel CDM and PCORnet.

“Type of Encounter” Definitions for the Sentinel CDM and PCORnet
Name Definition Sentinel PCORnet
Ambulatory Visit (AV) Includes visits at outpatient clinics, same day surgeries, urgent care visits, and other same-day ambulatory hospital encounters, but excludes emergency department encounters. x x
Emergency Department (ED) Includes ED encounters that become inpatient stays (in which case inpatient stays would be a separate encounter). Excludes urgent care visits. ED claims should be pulled before hospitalization claims to ensure that ED with subsequent admission won't be rolled up in the hospital event. x x
ED Admit to Inpatient (EI) Emergency Department Admit to Inpatient Hospital Stay: Permissible substitution for preferred state of separate ED and IP records. Only for use with data sources where the individual records for ED and IP cannot be distinguished. x
Inpatient Hospital (IP) Includes all inpatient stays, same-day hospital discharges, hospital transfers, and acute hospital care where the discharge is after the admission date. (PCORnet only: Does not include observation stays, where known.) x x
Observation Stay (OS) Hospital outpatient services given to help the doctor decide if the patient needs to be admitted as an inpatient or can be discharged. Observations services may be given in the emergency department or another area of the hospital.” Definition from Medicare, CMS Product No. 11435, https://www.medicare.gov/Pubs/pdf/11435.pdf x
Institutional Professional Consult (IC) Permissible substitution when services provided by a medical professional cannot be combined with the given encounter record, such as a specialist consult in an inpatient setting; this situation can be common with claims data sources. This includes physician consults for patients during inpatient encounters that are not directly related to the cause of the admission (e.g. a ophthalmologist consult for a patient with diabetic ketoacidosis) guidance updated in v4.0). x
Non-Acute Inst. Stay (IS) Includes hospice, skilled nursing facility (SNF), rehab center, nursing home, residential, overnight non-hospital dialysis and other non-hospital stays. x x
Other Ambulatory (OA) Includes other non overnight AV encounters such as hospice visits, home health visits, skilled nursing facility visits, other non-hospital visits, as well as telemedicine, telephone and email consultations. (PCORnet only: May also include "lab only" visits [when a lab is ordered outside of a patient visit], "pharmacy only" [e.g., when a patient has a refill ordered without a face-to-face visit], "imaging only", etc.] x x
Other (OT) x
Unknown (UN) x
No Information (NI) x

There are many possible encounter types, and it is important for investigators to understand encounter type definitions and to harmonize them across sites if possible. As an example, in 2014-2015, the University of California Research Exchange (UCReX) harmonized data from EHR sources across 5 medical campuses of the University of California to establish a common definition against which a single query would return patient counts against the geographically distributed but federated system architecture (Gabriel et al 2014). The data harmonization team discovered that there were 60 unique encounter types across the sites contributing EHR data extracts (Gabriel et al 2014). Many EHRs have 100 to 200 encounter types, which leads to 2 important considerations: (1) whether sites can correctly map their encounter types to the specified CDM value set; and (2) whether those value sets contain enough granularity for the research question. In some cases, there may be benefit in using the “raw” encounter types instead of the harmonized encounter types. If this occurs frequently, however, it would be better to expand the value set of the underlying CDM to better support such research questions.

Application Programming Interfaces

As noted previously, Fast Healthcare Interoperability Resources (FHIR) is emerging as a standard to obtain data from EHRs (Garza et al 2019; Duda et al 2022). Many data collection tools, including REDCap, have developed or are developing middleware services that allow data to be pulled from FHIR resources and to populate a study database or case report form. Some of these solutions are not compliant with CFR 21 Part 11 and may not be appropriate for all trials (Campion et al 2017).

While FHIR-based methods of data acquisition hold great promise, there are several caveats. First, because EHR data are not collected in a standard way, there is potential for mapping discrepancies (Marsolo 2019). Two sites with the same EHR may map FHIR resources in slightly different ways, and the same FHIR request could result in 2 slightly different data sets. Second, many sites have limited experience delivering data in this way, and the skill set to develop, maintain, and deliver data through data exchange is highly specialized (Marsolo 2019). Some of these issues are being mitigated through the US Core and Argonaut (consensus mappings) and by EHR vendors implementing these mappings as a part of their products. Nonetheless, facility-specific implementation decisions will affect the EHR vendor standard mappings. A final caveat is that all ONC-certified EHRs are supposed to allow patients to request copies of their data via FHIR APIs (see the “Participant-Reported Data” section of this chapter). These are nominally the same APIs as those on the “clinical” side, but there may be restrictions on the data that are available via participant-facing APIs. For example, a query for real-time laboratory results would yield different results if a healthcare system delays information for clinician review before releasing it to a patient.

See the Using Electronic Health Record Data chapter of the Living Textbook for more information about interoperability, data as a surrogate for clinical phenomena, and uses of EHR data in pragmatic clinical trials.

Previous Section Next Section

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

Resources

Screenshot of Grand Rounds presentation
PCORnet COVID-19 Common Data Model Design and Results
NIH Pragmatic Trials Collaboratory PCT Grand Rounds; June 5, 2020

EHR-Based Outcome Measurement in the LIRE Trial
NIH Pragmatic Trials Collaboratory EHR Workshop video module. Dr. Jerry Jarvik of the University of Washington summarizes the technical and cultural challenges of embedding a radiology reporting intervention into the EHRs of multiple healthcare systems in the LIRE NIH Collaboratory Trial.

EHR Pragmatic Innovation Beyond Follow-up in the TSOS Study
NIH Pragmatic Trials Collaboratory EHR Workshop video module. Dr. Doug Zatzick of the University of Washington describes the unique challenges of using electronic health records in the TSOS NIH Collaboratory Trial.

REFERENCES

back to top

Campion TR Jr, Sholle ET, Davila MA Jr. 2017. Generalizable middleware to support use of REDCap dynamic data pull for integrating clinical and research data. AMIA Jt Summits Transl Sci Proc. 2017:76-81. PMID: 28815111.

Gabriel D, Meeker D, Bell D, Matheny M. 2014. Data Harmonization and Synergies: OMOP, PCORnet CDM and the CTSA cohort identification models. Presented at: Data Integration, Analysis & Sharing Symposium; La Jolla, California; September 16, 2014.

Duda SN, Kennedy N, Conway D, et al. 2022. HL7 FHIR-based tools and initiatives to support clinical research: a scoping review. J Am Med Inform Assoc. 29:1642–1653. doi: 10.1093/jamia/ocac105. PMID: 35818340.

Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. 2016. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform. 64:333–341. doi:10.1016/j.jbi.2016.10.016. PMID: 27989817.

back to top

Garza M, Myneni S, Nordo A, Eisenstein EL, Hammond WE, Walden A, Zozus M. eSource for standardized health information exchange in clinical research: a systematic review. Stud Health Technol Inform. 2019;257:115-124. PMID: 30741183.

Marsolo K. 2019. Approaches to Patient Follow-Up for Clinical Trials: What’s the Right Choice for Your Study? NIH Pragmatic Trials Collaboratory PCT Grand Rounds; March 1, 2019. Available at: https://rethinkingclinicaltrials.org/news/approaches-to-patient-follow-up-for-clinical-trials-whats-the-right-choice-for-your-study-keith-marsolo-phd/. Accessed October 14, 2022.

Office of the National Coordinator for Health Information Technology (ONC). Interoperability Standards Advisory (ISA). https://www.healthit.gov/isa/. Accessed July 21, 2022.


Version History

October 14, 2022: Made updates to the subsection “Application Programming Interfaces (APIs),” made nonsubstantive changes to the text, added an image to the Resources sidebar, added Seils as a contributing editor, and reordered the section within the chapter as part of the annual content update (changes made by D. Seils).

January 25, 2022: Added a link to the June 5, 2020, PCT Grand Rounds to the Resources sidebar as part of the annual content update (changes made by D. Seils).

January 15, 2021: Moved this section from “Inpatient Endpoints” to “Acquiring Real-World Data”, added EHR Workshop video module to resource bar (changes made by K. Staman).

Published June 19, 2019.

current section :

Acquiring Electronic Health Record Data

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

Acquiring Patient-Reported Data

CHAPTER SECTIONS

Acquiring Real-World Data


Section 6

Acquiring Patient-Reported Data

Contributors

This section details methods for obtaining data from participants. See the Patient-Reported Outcomes chapter of the Living Textbook for information about how patient-reported outcomes can be used as part of core outcome sets in pragmatic trials and strategies for choosing appropriate measures.

The participant can be a valuable source of data. Patient-reported data offer a relatively low-cost option and help keep participants engaged in the research (Marsolo 2019). However, the accuracy of patient-reported outcomes, including outcomes as notable as hospitalizations, has been questioned (Klungel et al 1999; Barr et al 2009; Krishnamoorthy et al 2016). One study compared the cumulative incidence of rehospitalizations identified by patient report vs medical bills and found that 10% of patients overreported and 18% underreported hospitalizations (Krishnamoorthy et al 2016). Of note, there were significant differences in patient characteristics associated with under- or overreporting: underreporting was more likely among patients who were older, women, African American, unemployed, or non–high school graduates; overreporting was more likely among patients who were women and unemployed (see figure below).

The authors of this study concluded that pragmatic research studies should not rely on patient report alone to identify rehospitalizations, and they suggested that additional mechanisms are needed. All sources of data have flaws—this is especially true of real-world data. Using multiple sources is a common technique for addressing this.

A major challenge of pragmatic research trial design is to navigate the tension between the uncertain accuracy of patient self-report and the need to efficiently utilize study resources and lower trial costs. (Krishnamoorthy et al 2016)

There are a number of methods to obtain information about patient-reported hospitalizations, and some are more passive (and less expensive) than others. The 3 most frequently used methods are patient portals, mobile apps, and call centers.

Portal

As part of a clinical study, participants may be asked to self-report events through either a portal or a mobile app. For example, the ADAPTABLE trial studied the use of low-dose aspirin vs a regular dose of aspirin in individuals with heart disease on outcomes such as myocardial infarction, mortality, and hospitalizations. To report their health status, participants were prompted to login to a patient portal. However, because this method might miss some patients, a call center contacted patients who did not login to the portal (National Patient-Centered Clinical Research Network [PCORnet] 2015).

Mobile Apps

Mobile apps can be used as standalone methods for patient data collection or in combination with other technologies. See the Mobile Health chapter of the Living Textbook for more information about opportunities and advantages and considerations for using mobile apps in clinical trials.

Some “enhanced recovery after surgery” programs use text messaging for extended surveillance. For example, text messaging has been used for home-based surveillance after colorectal surgery, where text message questionnaires led to follow-up care, including  in-hospital care, rehospitalizations, and unplanned surgeries (Carrier et al 2016).

Smartphones and other mobile technologies can be used to report events. For example, the National Evaluation System for Health Technology (NEST) was developed to ensure the postmarket safety and effectiveness of medical devices (Fleurence and Shuren 2019). A NEST feasibility/pilot NIH Collaboratory Trial was conducted to evaluate a mobile health platform for obtaining real-world postmarket surveillance data for patients after either bariatric surgery (sleeve gastronomy or gastric bypass) or catheter-based atrial fibrillation ablation (Dhruva 2020). Sixty participants were enrolled at their preoperative appointment for 8 weeks of postprocedural follow-up. The investigators used the Hugo app to obtain and aggregate electronic medical record data, pharmacy data, personal digital device data, and patient-reported outcomes through questionnaires. Hugo uses the sync-for-science model: people grant permission for applications like Hugo to access their her and pharmacy data and to sync these data (and their wearable device data) for research purposes through a common platform (Dhruva 2018).

It is important to note that a comprehensive picture can only be obtained if patients link their data from all the places where they receive care, and vendor support may be required the first time a connection is made to a given healthcare system. Prior to about 2020, the EHR data that could be consumed in an app like Hugo were summary documents in a structured format, such as Continuity of Care Documents. As more healthcare systems make EHR information available via Fast Healthcare Interoperability Resources (FHIR) application programming interfaces (APIs) and the associated technology becomes more widespread (for example, integrated into iOS and Android smartphones), app-based methods of obtaining data from participants may become more feasible.

Call Center

In a pragmatic trial, a call center can also be used as backup for other secondary information sources (as with the ADAPTABLE example above), or it can be the primary source of information (as with the TRANSFORM-HF example). The important point is that a call center can be used to validate inpatient events that are triggered by other means. For example, the Influenza Vaccine to Effectively Stop Cardiothoracic Events and Decompensated Heart Failure (INVESTED) trial was a pragmatic trial comparing the effectiveness of high-dose flu vaccine vs the standard dose in patients with a history of recent heart failure or myocardial infarction hospitalization (Vardeny et al 2021). Participants were asked to inform site personnel of hospitalizations and were then called during influenza season and during the summer to ascertain outcomes.

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

Resources

Collecting and Sharing Patient-Reported Outcomes in the PPACT Trial
NIH Collaboratory EHR Workshop video module. Dr. Lynn DeBar of the Kaiser Permanente Washington Health Research Institute describes how her team collected and shared patient-reported outcomes in the PPACT trial.

REFERENCES

back to top

Barr EL, Tonkin AM, Welborn TA, Shaw JE. 2009. Validity of self-reported cardiovascular disease events in comparison to medical record adjudication and a statewide hospital morbidity database: the AusDiab study. Intern Med J. 39(1):49-53. doi:10.1111/j.1445-5994.2008.01864.x. PMID: 19290982.

Carrier G, Cotte E, Beyer-Berjot L, Faucheron JL, Joris J, Slim K; Groupe Francophone de Réhabilitation Améliorée après Chirurgie (GRACE). 2016. Post-discharge follow-up using text messaging within an enhanced recovery program after colorectal surgery. J Visc Surg. 153(4):249-52. doi: 10.1016/j.jviscsurg.2016.05.016. PMID: 27423211.

Dhruva S. 2018. Using a Novel mHealth Platform to Obtain Real-World Data for Post-Market Surveillance: A NEST Trial. NIH Pragmatic Trials Collaboratory PCT Grand Rounds; August 10, 2018. https://rethinkingclinicaltrials.org/news/august-10-2018-using-a-novel-mhealth-platform-to-obtain-real-world-data-for-post-market-surveillance-a-nest-demonstration-project-sanket-dhruva-md-mhs/. Accessed October 14, 2022.

Dhruva SS, Ross JS, Akar JG, et al. 2020. Aggregating multiple real-world data sources using a patient-centered health-data-sharing platform. NPJ Digit Med. 3:60. doi: 10.1038/s41746-020-0265-z. PMID: 32352038.

Fleurence RL, Shuren J. 2019. Advances in the use of real-world evidence for medical devices: an update from the National Evaluation System for Health Technology. Clin Pharmacol Ther. 106(1):30-33. doi: 10.1002/cpt.1380. PMID: 30888048.

Klungel OH, de Boer A, Paes AH, Seidell JC, Bakker A. 1999. Cardiovascular diseases and risk factors in a population-based study in the Netherlands: agreement between questionnaire information and medical records. Neth J Med. 55:177-183. doi: 10.1016/s0300-2977(99)00045-5. PMID: 10555434.

back to top

Krishnamoorthy A, Peterson ED, Knight JD, et al. 2016. How reliable are patient-reported rehospitalizations? Implications for the design of future practical clinical studies. J Am Heart Assoc. 5(1):e002695. doi: 10.1161/JAHA.115.002695. PMID: 26811163.

Marsolo K. 2019. Approaches to Patient Follow-Up for Clinical Trials: What’s the Right Choice for Your Study? NIH Pragmatic Trials Collaboratory PCT Grand Rounds; March 1, 2019. Available at: https://rethinkingclinicaltrials.org/news/approaches-to-patient-follow-up-for-clinical-trials-whats-the-right-choice-for-your-study-keith-marsolo-phd/. Accessed October 14, 2022.

National Patient-Centered Clinical Research Network (PCORnet). 2015. Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness (ADAPTABLE) Study Protocol.

Vardeny O, Kim K, Udell JA, et al. 2021. Effect of high-dose trivalent vs standard-dose quadrivalent influenza vaccine on mortality or cardiopulmonary hospitalization in patients with high-risk cardiovascular disease: a randomized clinical trial. JAMA. 325(1):39-49. doi: 10.1001/jama.2020.23649. PMID: 33275134.


Version History

October 14, 2022: Made nonsubstantive changes to the text, added Seils as a contributing editor, and reordered the section within the chapter as part of the annual content update (changes made by D. Seils).

January 15, 2021: Moved this section from “Inpatient Endpoints” to “Acquiring Real-World Data”, added EHR Workshop video module to the resource bar (changes made by K. Staman).

Published June 19, 2019

current section :

Acquiring Patient-Reported Data

  1. Introduction
  2. Common Real-World Data Sources
  3. Data Formats
  4. Acquiring Electronic Health Record Data
  5. Acquiring Claims Data and CMS Research-Identifiable Files
  6. Acquiring Patient-Reported Data
  7. Gaining Permission to Use Real-World Data
  8. Methods of Access
  9. Case Study: The IMPACT-AFib Trial

NIH HEAL, FDA, and Other Core Outcome Sets

CHAPTER SECTIONS

Real-World Evidence: Patient-Reported Outcomes (PROs)


Section 4

NIH HEAL, FDA, and Other Core Outcome Sets

Contributors

Core Outcome Sets

Variability in outcomes measures across clinical trials hinders between-trial comparisons and future evaluations of efficacy and effectiveness of interventions and treatments (Dworkin et al. 2005). Use of a standard set of outcome measures—or core outcome sets—can “facilitate the process of developing research protocols, encourage development of multicenter projects in which all participating facilities agree to include these measures, provide a basis for determining the treatment outcomes that constitute clinically important differences, permit pooling of data from different studies, and provide a basis for meaningful comparisons among treatments of the clinical importance of their outcomes” (Dworkin et al. 2005).

Core outcome sets have been developed for different domains to help standardize outcome reporting in clinical trials (see Table), and investigators of ePCTs should consider if any of these core outcome sets could be used to address their specific research question. Of note, the Helping to End Addiction Long-term Initiative, or NIH HEAL Initiative, has created core outcome sets to facilitate cross-study comparisons for pain, improve interoperability of findings for patient-reported outcomes, and compare results across trials. The HEAL core outcome sets were released in January 2020 and represent PROs that HEAL pain clinical trials are required to collect. The NIH Pragmatic Trials Collaboratory currently supports 6 pragmatic NIH Collaboratory Trials that are a part of the HEAL Initiative: the PRISM studies (Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing).

Table 1 shows core outcome sets for NIH HEAL, FDA, and others. Where possible, a link to a sample measure is provided as well as a link to the instructions.

Table 1. Core Outcome Sets by Indication, Domain, and PRO Measure

Outcome Set
 Core Outcome Domains Suggested PRO Measures (with links to samples)
 Measure Instructions
NIH HEAL Initiative CDE for adult acute pain Pain intensity, pain interference, physical functioning/QOL, sleep, pain catastrophizing, depression, anxiety, global satisfaction with treatment, substance use screener
NIH HEAL Initiative CDE for adult chronic pain Pain intensity, pain interference, physical functioning/QOL, sleep, pain catastrophizing, depression, anxiety, global satisfaction with treatment, substance use screener
  • Pain, enjoyment, and general activity scale (PEG)
  • PROMIS Physical Functioning Short Form 6b
  • PROMIS Sleep Disturbance 6a + Sleep Duration Question
  • Pain Catastrophizing Scale – Short Form 6
  • PHQ-2
  • GAD-2
  • PGIC
  • TAPS 1
NIH HEAL Initiative CDE  for pediatric acute and chronic pain Child: Pain intensity, pain interference, physical functioning/QOL, sleep, pain catastrophizing, depression, anxiety, global satisfaction with treatment, substance use screener

Parent: pain catastrophizing, depression, anxiety

 Child:

  • BPI Short Form
  • Pediatric Quality of Life (PedsQL) Inventory
  • Adolescent Wake Sleep Scale + Sleep duration items
  • Pain Catastrophizing Scale for Children
  • PHQ-2
  • GAD-2
  • PGIC
  • National Institute on Drug Abuse (NIDA) Modified ASSIST Tool - 2

Parent:

  • Pain Catastrophizing
  • PHQ-2
  • GAD-2
 Child:

Parent:
NIH HEAL Initiative Back Pain Consortium (BACPAC) Minimum Dataset Pain intensity, pain interference, physical function/QOL, sleep, pain catastrophizing, depression, anxiety, global satisfaction with treatment, substance use screener, pain location, widespread pain, chronic lower back pain, opioid use
  • PEG Scale
  • PROMIS Physical Functioning Short Form 6b
  • PROMIS Sleep Disturbance 6a + Sleep Duration Question
  • Pain Catastrophizing Scale – Short Form 6
  • PHQ-2
  • GAD-2
  • PGIC
  • TAPS 1
  • Abbreviated Pain Somatization Adapted from NIH Research Task Force Minimum Dataset
  • Pain location determined by Radicular Pain Questions Adapted from NIH Research Task Force Minimum Dataset
  • Chronic lower back pain: 2 Items (low-back pain duration and frequency) from NIH Research Task Force Minimum Dataset
  • Single-item current opioid use
 BACPAC dataset requirements and recommendations
An international multidisciplinary panel (Chiarotto et al. 2018) for nonspecific low-back pain Physical functioning, pain intensity, and health-related quality of life (HRQoL)

 
Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) (Dworkin et al. 2005): Chronic pain Pain, physical functioning, emotional functioning, participant ratings of improvement and satisfaction with treatment, symptoms and adverse events and participant disposition
  • Numerical or categorical rating scale of pain intensity
  • Multidimensional Pain Inventory Interference Scale
  • BPI interference items
  • Beck's Depression Inventory
  • Profile of Mood States
  • PGIC
  • Symptoms and adverse events
  • Passive capture of spontaneously reported adverse events and symptoms and use of open-ended prompts
Veterans Health Administration Work Group (Kroenke et al. 2019): Musculoskeletal pain

 

 Pain intensity

Pain interference
  • Single-item numerical rating scale
  • BPI interference scale
 The Brief Pain Inventory User Guide
The U.S. Government, including individuals from the Office of Hematology and Oncology Products, Office of New Drugs, Center for Drug Evaluation and Research, and the U.S. Food and Drug Administration (Kluetz et al. 2016) developed these common measures for cancer. Symptomatic adverse events, physical function, disease-related symptoms
The Center for Medical Technology Policy (CMTP) (Basch et al. 2012): cancer Symptoms, HRQoL  

 

 

Previous Section Next Section


SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. How Are PRO Measures Used?
  3. Best Practices for Collecting PRO Measures in Pragmatic Clinical Trials
  4. NIH HEAL, FDA, and Other Core Outcome Sets
  5. Choosing PRO Measures
  6. Cultural Adaptation and Linguistic Translation
  7. Case Study From Guiding Good Choices for Health
  8. Incorporating PRO Data Into the EHR
  9. Patient-Focused Drug Development
  10. Additional Resources

Resources

HEAL Cooperative Agreement Presentation at 2020 PRISM Onboarding Meeting

 

REFERENCES

back to top

Basch E, Abernethy AP, Mullins CD, et al. 2012. Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology. J Clin Oncol. 30(34):4249-4255. doi:10.1200/JCO.2012.42.5967. PMID: 23071244.

Chiarotto A, Boers M, Deyo RA, et al. 2018. Core outcome measurement instruments for clinical trials in nonspecific low back pain. Pain. 159(3):481-495. doi:10.1097/j.pain.0000000000001117. PMID: 29194127.

Dworkin RH, Turk DC, Farrar JT, et al. 2005. Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. Pain. 113(1):9–19. doi:10.1016/j.pain.2004.09.012. PMID: 15621359.

 

Kluetz PG, Slagle A, Papadopoulos EJ, et al. 2016. Focusing on core patient-reported outcomes in cancer clinical trials: symptomatic adverse events, physical function, and disease-related symptoms. Clin Cancer Res. 22(7):1553-1558. doi:10.1158/1078-0432.CCR-15-2035. PMID: 26758559.

Kroenke K, Krebs EE, Turk D, et al. 2019. Core outcome measures for chronic musculoskeletal pain research: recommendations from a Veterans Health Administration Work Group. Pain Med. 20(8):1500-1508. doi:10.1093/pm/pny279. PMID: 30615172.

Kroenke K, Spitzer RL, Williams JB. 2001. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.x. PMID: 11556941.

 

 

 


Version History

September 14, 2022: Updated as part of annual update (changes made by K Staman).

Published May 30, 2020

Updated November 17, 2020: Moved the Common Data Elements to a new section (changes made by K Staman).

current section :

NIH HEAL, FDA, and Other Core Outcome Sets

  1. Introduction
  2. How Are PRO Measures Used?
  3. Best Practices for Collecting PRO Measures in Pragmatic Clinical Trials
  4. NIH HEAL, FDA, and Other Core Outcome Sets
  5. Choosing PRO Measures
  6. Cultural Adaptation and Linguistic Translation
  7. Case Study From Guiding Good Choices for Health
  8. Incorporating PRO Data Into the EHR
  9. Patient-Focused Drug Development
  10. Additional Resources

Embedded ePCT Team Composition

CHAPTER SECTIONS

 

Building Partnerships and Teams to ensure a successful trial

Section 6

Embedded ePCT Team Composition

Contributors

Although a research team typically designs a pragmatic study, it is the healthcare system partners who actually deliver the intervention. Many different people are involved in the conduct of an ePCT, and when designing a trial, an investigator will need to carefully consider the roles of potential team members, including which roles are essential, and who will fill the roles. Because ePCTs are embedded in health systems, team members from both the research team and the health system must understand and respect each other’s workflow, culture, priorities, and responsibilities (Johnson et al. 2014). Communication is essential, and having a team member with both research and health system expertise can help ensure the success of a PCT (Johnson et al. 2014).

 

GRACE Case Study

In some cases, it may be necessary for study team members to perform certain functions to reduce patient and clinician burden and to streamline operations. The GRACE Trial tests the effectiveness of guided relaxation and acupuncture for chronic sickle cell disease pain at University of Illinois Chicago (UIC), Duke University, and University of Florida. The GRACE Trial is a hybrid 1 effectiveness-implementation trial, which means we are first and foremost testing effectiveness of the two interventions and are beginning to identify facilitators and barriers to implementing the interventions. At all 3 study sites, the individual teams developed customized processes for scheduling and tracking treatments at the different sites where acupuncture is delivered. UIC does not have an integrative medicine clinic, therefore acupuncture treatments are being administered at the complementary integrative health research lab at the College of Nursing. The study acupuncturist schedules all acupuncture treatments. At Duke, acupuncture treatments are being provided at a well-established integrative medicine clinic and at private acupuncture clinics. The participants are permitted to schedule appointments directly through the clinics to provide optimal flexibility. However, study staff have established processes to work with clinic staff and participants to send reminders for upcoming acupuncture treatments and document participant attendance, address problems the clinic may have reaching the participant, and are available to answer questions related to the research protocol. At the University of Florida, the acupuncture is provided by community acupuncturists in private practice. The study staff collaborates with the acupuncturists and the patient to schedule the first acupuncture study visit. Thereafter, the acupuncturist and the patient work with their schedules to schedule the rest of the study visits. The patient contacts the acupuncturist directly to reschedule or cancel an appointment. The acupuncturists contact the study staff for any matters or difficulties they encounter in working with the patients to complete the acupuncture study visit. The acupuncturists document all patient encounters in REDCap per GRACE trial protocol.

Assessing Clinicians’ Views on Participating in Pragmatic Clinical Trials

A 2020 qualitative study was conducted to better understand the views of clinicians—including physicians, nurses, and other care providers—about their potential role in research conducted in the context of healthcare delivery. A goal of the study was to develop guidance for how study teams can best engage front-line clinicians to participate in ePCTs. Among the study’s findings is the importance of involving clinicians early in planning the study, including helping select the research topic. Key reasons for early engagement include that front-line care providers can provide insights to researchers on issues of clinical relevance, specific populations, and gaining trust in research. Also, because every clinic has its own workflow, clinicians are aware of constraints and potential disruptions that a trial might bring to clinic operations. Last, clinicians need to believe in the importance of the study so they can translate that to patients and staff.

The authors describe three levels of PCT engagement with clinicians, characterizing the trials as being “low touch, medium touch, or high touch”:

  • A low-touch PCT would be one that involves direct recruitment and enrollment of eligible patients by members of the study team with no need for clinician involvement in delivering the study intervention or conducting follow-up.
  • A medium-touch PCT would be one where there are sufficient numbers of patients at all study sites, a functional EHR system, site-level study champions, and adequate space to accommodate research activities without impeding workflow.
  • A high-touch PCT would be one in which a clinician plays an active role in patient identification, recruitment, delivery of the intervention, data collection, and patient follow up.

While it has been reported that clinicians hold generally favorable views of research, there are barriers to participation such as lack of time and competing priorities. Study authors also noted that there are clinicians who are interested in research but do not know how to get involved. Expanded opportunities for clinicians and researchers to collaborate via multiple platforms and modalities should be encouraged.

“With increasingly high demands on clinician time and the fact that their first priority is, as it should be, providing the best possible patient care, research teams should take on as much of the burden of trial implementation as possible.” –Tambor et al. 2020

Watch the video module: Engaging With Stakeholders in Pragmatic Clinical Trials

Potential ePCT team members can include:

  • Principal investigator and co-investigator
  • Health system leader or executive
  • Biostatistician
  • Lead clinician (e.g., epidemiologist, behavioral specialist, radiologist, pharmacist, physical therapist)
  • Clinical staff (e.g., nurse, operations manager, business manager)
  • IT specialist for EHR data extraction or clinical decision support tool design
  • Professional society leader
  • Site champion/liaison
  • Practice facilitator
  • Research assistant
  • Project coordinator
  • Communications specialist
  • Research participant, patient, or patient advocate

These team members will fulfill different roles throughout the course of the trial, and could assist with trial design, budgeting, site and participant recruitment, communications with patients, trial conduct, and dissemination and implementation of results (Dolor et al. 2014). Early engagement of the team will help identify workflow issues, ensure the budget is feasible, and help with recruitment of sites and clinicians.

Roles and Expertise

The expertise necessary for a trial will depend on the specific study aims and how the intervention will be embedded in the healthcare system workflow. A simplified example of the roles the team members could fulfill is shown in the table.

Role Priority Study question refinement Protocol development Conduct
Researcher Improve public heath by answering research questions Develops and procures funding for research question Drafts protocol Ensures sustainability; communicates with team; helps solve issues as they arise
Patient Improve individual health or the health of others with a similar condition Provides feedback Provides feedback May participate in study
HCS Leader Improve patient outcomes, reduce costs, and increase efficiency Ensures question aligns with priorities and mission Provides feedback, promotes and supports study Promotes and supports study
IT Staff EHR features that support patient care and billing May need to adapt EHR for data collection. Can advise on feasibility Responsible for data collection and/or data pulls. Will help mitigate problems with data quality (consistency, anomalies, errors, etc)
Frontline and Operational Staff Patient care Consult on protocol so it fits into workflow with a little burden as possible. Advise on feasibility Responsible for data collection and/or implementing interventions
Clinic Champion Improving patient outcomes Refines question with researchers and HCS leaders Serves a liaison and helps integrate protocol into workflow Ensures sustainability; helps to solve issues as they arise

Finding the Right Biostatistician for the ePCT Team

A biostatistician will be an essential member of the ePCT study team, and ideally would be engaged from the earliest phase of developing the research question and proposal through the study design, statistical analysis, and final results dissemination. It will be useful for this team member to have experience with pragmatic trial interventions as well as familiarity with the PRECIS-2 tool, used to evaluate the degree of pragmatism along 9 domains. An understanding of the PICOT (population, intervention, comparison, outcome, time) framework is helpful in developing the research question, as the question will drive the design, and the design will drive the analysis. Other skills for the biostatistician include relevant experience in the design and analysis of randomized clinical trials (RCTs) with clustering, such as cluster-randomized trials (CRTs), stepped-wedge design, and individually-randomized group treatment trials. Familiarity with the Consolidated Standards of Reporting Trials (CONSORT) statements, including extensions for cluster trials and pragmatic trials, will be helpful knowledge for the team as they prepare to report their findings.

Other Considerations

Clinical and healthcare system staff and team members tend to have high turnover rates, which creates challenges for ePCTs. Developing good documentation and training practices for new team members helps ensure sustainability in case of turnover. Perhaps more importantly, NIH Collaboratory PIs have found that engaging staff throughout the trial and developing clear methods for multidirectional communication can help identify and mitigate potential issues before they become acute. For example, the STOP CRC trial implemented a well-validated quality improvement approach called Plan-Do-Study-Act, or PDSA. The use of PDSA helped to identify implementation issues and unintended consequences and empowered clinics to actively address local conditions.

A directory of team members is advisable; for example, the Health Care Services Research Network (a network of healthcare systems that participate in research) provides a directory on their website with key administrative contacts by site.

At the end of the trial, team members, including participating sites, physicians, and clinics, should have the opportunity to receive, interpret, and implement findings from the trial. Likewise, where applicable, findings should also be communicated to patients or patient advocacy groups.

To determine what skills are needed, consider the following key questions in the planning phase:

        • What clinical specialties will be needed to carry out the intervention?
        • What roles will support clinic operations?
        • Who will be the liaison between healthcare system departments for interventions that are multidisciplinary?
        • What aspects of the trial will require IT staff expertise?
        • Will the trial need training videos, online materials, or toolkits?

 

Previous Section Next Section

SECTIONS

CHAPTER SECTIONS
sections

  1. Introduction
  2. Engagement Across Settings and Populations
  3. Deciding Who to Engage
  4. Partner Engagement Throughout the PCT Life Cycle
  5. Advice From Healthcare System Leadership
  6. Embedded ePCT Team Composition
  7. Learning Health Systems and Embedded Clinical Trials
  8. Framework for Health Systems to Assess PCT Participation
  9. FAQ

Resources

Considerations for Training Front-Line Staff and Clinicians on Pragmatic Clinical Trial Procedures

This working document helps ePCT teams plan training for study procedures that involve frontline clinicians and staff.

A guide to research partnerships for pragmatic clinical trials (Johnson KE et al. 2014)

Publication describing how to build strong research partnerships between health researchers and healthcare systems.

REFERENCES

Dolor RJ, Schmit KM, Graham DG, Fox CH, Baldwin LM. 2014. Guidance for Researchers Developing and Conducting Clinical Trials in Practice-based Research Networks (PBRNs). J Am Board Fam Med. 27(6):750–758. doi:10.3122/jabfm.2014.06.140166. PMID: 25381071

back to top

Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, Martin PJ, Richards J, Rosenthal G, Septimus E, et al. 2014. A guide to research partnerships for pragmatic clinical trials. BMJ. 349(dec01 7):g6826–g6826. doi:10.1136/bmj.g6826. PMID: 25446054

 


Version History

October 3, 2022: Added new sections “GRACE Case Study” and “Assessing Clinicians’ Views on participating in Pragmatic Clinical Trials” (changes made by K. Staman and L. Stewart)

January 22, 2021: Added embedded video (change made by G. Uhlenbrauck).

August 27, 2020: Relocated the “Embedded PCT Team Composition” chapter to “Building Partnerships and Teams to Ensure a Successful Trial,” and added a subsection on how to find the right biostatistician as part of the annual content update (changes made by L. Wing).

Published Feb 25, 2020

current section :

Embedded ePCT Team Composition

  1. Introduction
  2. Engagement Across Settings and Populations
  3. Deciding Who to Engage
  4. Partner Engagement Throughout the PCT Life Cycle
  5. Advice From Healthcare System Leadership
  6. Embedded ePCT Team Composition
  7. Learning Health Systems and Embedded Clinical Trials
  8. Framework for Health Systems to Assess PCT Participation
  9. FAQ

Recruitment Highlights

CHAPTER SECTIONS

Participant Recruitment


Section 5


Recruitment Highlights

Contributors

Recruitment strategies in the Collaboratory ePCT Trials have varied widely, both in terms of their plans and ultimately, for a few, in the actual scenarios that evolved. The table below briefly describes some of the trials. During the conduct of an ePCT, all may go as originally designed. In other ePCTs, circumstances may change and additional, relevant discoveries made. In these situations, study teams may wish to adapt the recruitment process and/or materials to optimize recruitment approaches.

 

Trial Recruitment Strategy
ABATE The ABATE trial recruited hospitals within a health system and was informed by a prior trial’s experience, resulting in a mostly seamless recruitment. All recruitment was performed within 2 to 3 months using recruitment webinars and leveraging support from the health system’s corporate and division leadership on regular calls. Recruitment was remote but heavily reliant on internal leadership to encourage participation. Participating hospitals were required to have a commitment letter signed by a member of the executive team (i.e., C-suite).
Nudge Patients eligible for the Nudge study were identified through the EHR and received opt out consent packets via the United States Postal Service. Thus, the study required minimal participation from clinicians and executive leadership at the clinic level. However, the Nudge study team believed providing education to the clinicians and leadership at each clinic was vital to ensure patients could be met with an additional layer of support with the intervention if needed.

The Nudge Study team first introduced the study to internal research committees, provider and healthcare leadership advisory groups, and patient stakeholder panels at the healthcare system level. Upon receiving approval from these groups, the Nudge Study team approached the clinic leads from 16 potential clinics across the 3 participating health care systems by providing an electronic summary of the study and scheduling a presentation during provider meetings. The in-person presentation provided background information about the study and copies of the study literature, including consent forms and sample “nudges” (text messages to participants delivered as part of the intervention). The team also offered to provide a list of potentially eligible patients from the clinic, allowing clinicians to note if there were patients who they would not recommend enrolling in the study. During these sessions, clinicians and leadership were encouraged to ask questions, provide input, and seek clarification from the study team. Of the 16 clinics approached across the 3 healthcare systems, all clinics participated in the Nudge study presentation and agreed to support the study
PROVEN The PROVEN trial did not require active recruitment. The video intervention was introduced as a new program in the participating nursing homes, and all patients in the facilities during the implementation period were eligible and targeted to receive the intervention. Consent was waived.
SPOT  

In the SPOT trial, participants were automatically enrolled prior to any consent or contact with the study. Those assigned to intervention conditions were contacted and offered intervention services. Participants were enrolled in the trial regardless of whether they accepted the offer of intervention services.

 
TiME  

Enrollment for the TiME trial was through an opt-out mechanism. The study team received information every 2 weeks about anyone who opted out. Few people opted out. The recruitment strategy during the trial did not require any changes or enhancements.
TSOS  

The TSOS trial was on track for 70%-80% recruitment of the targeted 960 patients. In 2018, due to regulatory issues, recruitment was suspended for all sites for an extended period of months. This recruitment suspension prevented the study team from meeting the original recruitment target, such that fewer patients were enrolled in the study. Because the recruitment pause occurred in the final wave of the trial, when all sites were recruiting intervention patients, the study is imbalanced with regard to intervention versus control patients.

 

 

 

Previous Section Next Section

DISCLAIMER: The views expressed in this chapter are those of the contributors and do not necessarily represent the views of the National Heart, Lung, and Blood Institute; the National Institutes of Health; or the U.S. Department of Health and Human Services. 

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Questions to Consider
  3. Case Study: ICD-Pieces™
  4. Payments and Incentives
  5. Recruitment Highlights
  6. Clinician Recruitment Case Study: HERO
  7. Additional Resources

Resources

More on recruitment in the Living Textbook:


Version History

September 27, 2023: Added new case study on the Nudge project (changes made by E. McCamic).

Published August 25, 2020

current section :

Recruitment Highlights

  1. Introduction
  2. Questions to Consider
  3. Case Study: ICD-Pieces™
  4. Payments and Incentives
  5. Recruitment Highlights
  6. Clinician Recruitment Case Study: HERO
  7. Additional Resources

Operationalizing Fitness-for-Use Assessments

CHAPTER SECTIONS

Assessing Fitness for Use of Real-World Data Sources


Section 8

Operationalizing Fitness-for-Use Assessments

Contributors

Best practices for operationalizing assessments of the fitness for use of real-world data are still evolving (Berger et al 2017; Daniel et al 2018; Mahendraratnam et al 2019). In general, approaches fall into 2 categories: single-stage and 2-stage. In a single-stage process, once a data source (or sources) has been selected, a cycle of cleaning, transformation, and linkage steps (if necessary) are applied to the "raw" real-world data, generating an output dataset that has been deemed fit for use. In a 2-stage process, the raw real-world data go through the cleaning, transformation, and linkage cycle to ensure they meet a set of predefined baseline quality characteristics. This baseline level of quality is described as "research ready," but an additional cycle of cleaning, transformation, and linkage may be required to determine that the dataset is fit for use for a given analysis. The Figure characterizes the process of making a real-world data set fit for purpose (Daniel et al 2018).

Figure. The Process of Making a Fit-for-Purpose Real-World Dataset

Fitness for use figure 1

Fitness for use fig 2
From the Duke Margolis Center for Health Policy white paper on Characterizing RWD Quality and Relevancy for Regulatory Purposes (Daniel et al 2018).

The decision to use a single-stage or 2-stage approach will depend on a number of factors. If a real-world data source is being used for a study or a standard with a well-defined purpose (eg, a device registry), it may be more efficient to use a single-stage process. If the data are to be used for a large number of to-be-defined studies with different objectives or are to be linked with many other sources, a 2-stage process may be more appropriate. Data within a given domain can meet basic levels of quality (eg, laboratory records are mapped to LOINC and have a specified result and unit of measure) with study-specific assessments to ensure that the results are suitable for the proposed analysis (eg, all patients in the proposed cohort have a result where one is expected, and the overall distribution of values is plausible). Study teams can focus on the factors that are most appropriate to their project and avoid most of the heavy lifting that occurs in initial transformation and cleaning steps.

Many DRNs use a 2-stage process with foundational data characterization routines that run against the CDMs of each partner and provide descriptive statistics such as summaries of missing values, outliers, and frequency distributions, as well as results from the network's data quality checks. The information from these routines is reviewed and those that successfully pass are able to respond to basic queries, with additional assessments issued prior to specific study-based analyses. This should be an iterative process, with characterization activities helping improve the level of data quality over time. In addition, the study-specific analyses would be expected to uncover certain quality issues, and if pervasive, some of these could be pushed down into the foundational process. This is especially true during the first studies that use a given dataset for research or the first time a data domain is harmonized or made available in a CDM. But in this manner, the gap between the foundational process and the study-specific assessments is narrowed, reducing the amount of work for each study team to complete their suitability assessments. The details of the data characterization processes of several DRNs was outlined in the NEST Data Quality Framework and is replicated in the Table

Table. Data Characterization Processes for Selected Distributed Research Networks

Network Collaborators Approach to Data Characterization
Healthcare Systems Health Plans
HCSRN X X Detailed checks look at ranges, cross-field agreement, implausible data patterns, and cross-site comparisons. Partners execute a data characterization package each time the data are refreshed. Results are returned to the HCSRN Coordinating Center. Potential quality issues are flagged and mitigated at the partner level (HCSRN).
Sentinel X X Detailed checks look at ranges, cross-field agreement, implausible data patterns, and cross-site comparisons. Partners execute a data characterization package each time data are refreshed. Results are returned to the Sentinel Coordinating Center. Potential quality issues are flagged and mitigated at the partner level (Sentinel Initiative; Sentinel Operations Center 2017).
PCORnet X X Includes a foundational data curation process, which establishes a baseline level of research readiness for all network partners to support preparatory-to-research queries, and a study-specific data curation process, which includes assessments of outcomes/variables or other derived concepts for the cohort under study (PCORnet; PCORnet Distributed Research Network Operations Center; Qualls et al 2018).
OHDSI X X Optional. Each "datamart" can generate a standardized data profile that is viewable through a web-based tool (Achilles or Data Quality Dashboard). Institutions can choose whether to share these profiles or retain them locally (OHDSI).
ACT X Under development (Visweswaran et al 2018).

Curtis et al 2019. From the NEST Data Quality Framework. ACT = Accrual for Clinical Trials; HCSRN = Health Care Systems Research Network; OHDSI = Observational Health Data Sciences and Informatics; PCORnet = National Patient-Centered Clinical Research Network.

The National Evaluation System for health Technology Coordinating Center (NESTcc) Data Quality Framework, describes topics around the capture and use of real-world data—primarily from EHRs—for the postmarket evaluation of medical devices. Even if a project was not directly focused on devices, many of the steps are relevant for other types of research using real-world data and can serve as a foundation from which to develop or adopt processes for quality assessment. An additional defining feature of the NESTcc Data Quality Framework is that it also includes a Data Quality Maturity Model, which organizations can use to benchmark themselves, identify gaps and opportunities, and develop operational plans on how to progress through the different stages of organization maturity. Variations of this model have been proposed elsewhere (Callahan et al 2017). While these efforts are nascent, attestations about organizational maturity may be able to help satisfy the FDA's fitness-for-use requirements related to reliability, as they can address aspects of quality control (data assurance) and workflow (data accrual).

One issue with the generalizability of all data checks and fitness-for-use assessments thus far is that there is an underlying assumption that there has been a large-scale transformation of the source data that would support population-level quality analyses (eg, assessing missing values, identifying outliers, examining frequency distributions, etc.). This model is less well suited for sites that provide data for a specific trial or study, but otherwise do not keep their data in a "research ready" format. Quality assessments are more challenging when data are only provided for a few dozen study participants. Certain data checks may still be applicable, but others may need to be addressed through other means, such as site surveys. As the number of pragmatic trials continues to grow, we expect this to remain an active area of research.

Conclusion

Given that the data in most real-world data sources were not directly collected for the study in question, it is important to assess the suitability or fitness of a dataset before using it in an analysis. The FDA has provided general guidance on how to approach these tasks, but consensus-based best practices have yet to emerge. In the meantime, study teams that wish to perform their own assessments can consider looking to the procedures developed by DRNs as a source. Sharing the details around fitness-for-use assessments is not widespread, but we expect those efforts to increase as the use of real-world data sources increases.

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Resources

Calibrating Real-World Evidence Against RCT Evidence: Early Learnings from RCT-DUPLICATE  Grand Rounds, February 26, 2021

Characterizing RWD Quality and Relevancy for Regulatory Purposes; Duke Margolis Center for Health Policy; 2020

A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks; EGEMS (Wash DC). 2017;5(1):8.

REFERENCES

back to top

Berger ML, Sox H, Willke RJ, et al. 2017. Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Value Health. 20:1003-1008. doi:10.1016/j.jval.2017.08.3019. PMID: 28964430.

Curtis LH, Brown J, Laschinger J, et al. 2019. National Evaluation System for Health Technology Coordinating Center (NESTcc) Data Quality Framework. https://nestcc.org/data-quality-and-methods/. Accessed Aug 25, 2020.

Daniel G, Silcox C, Bryan J, McClellan M, Romine M, Frank K. Characterizing RWD Quality and Relevancy for Regulatory Purposes, 2018. https://www.semanticscholar.org/paper/Characterizing-RWD-Quality-and-Relevancy-for-Duke-Margolis/fd8731fcb5db8b78b8498e2b66102830c7350be5. Accessed Aug 25, 2020.

Mahendraratnam N, Silcox C, Mercon K, et al. Determining Real-World Data’s Fitness for Use and the Role of Reliability: Robert J. Margolis, MD, Center for Health Policy at Duke University, 2019. https://healthpolicy.duke.edu/publications/determining-real-world-datas-fitness-use-and-role-reliability. Accessed Aug 25, 2020.

Observational Health Data Sciences and Informatics. ACHILLES for data characterization. Available at: https://www.ohdsi.org/analytic-tools/achilles-for-data-characterization/. Accessed January 26, 2018.

PCORI. 2012. Standards in the Conduct of Registry Studies for Patient-Centered Outcomes Research.

PCORnet. PCORnet Common Data Model (CDM). Available at: https://pcornet.org/data-driven-common-model/. Accessed January 26, 2017.

 

back to top

PCORnet Distributed Research Network Operations Center. PCORnet Data Curation Query Package. Available at: https://github.com/PCORnet-DRN-OC/PCORnet-Data-Curation. Accessed January 26, 2018.

Qualls LG, Phillips TA, Hammill BG, et al. 2018. Evaluating Foundational Data Quality in the National Patient-Centered Clinical Research Network (PCORnet(R)). EGEMS (Wash DC). 6:3. doi:10.5334/egems.199. PMID: 29881761.

Sentinel Initiative. Distributed Database and Common Data Model. Available at: https://www.sentinelinitiative.org/sentinel/data/distributed-database-common-data-model. Accessed January 26, 2017.

Sentinel Operations Center. Sentinel Common Data Model - Data Quality Review and Characterization Process and Programs.  Program Package version: 3.3.4, 2017. https://www.sentinelinitiative.org/about/how-sentinel-gets-its-data. Accessed Aug 25, 2020.

Visweswaran S, Becich MJ, D'Itri VS, et al. 2018. Accrual to Clinical Trials (ACT): A Clinical and Translational Science Award Consortium Network. JAMIA Open. 1:147-152. doi:10.1093/jamiaopen/ooy033. PMID: 30474072.


Version History

August 26, 2022: Added Grand Rounds as part of annual review (changes made by K. Staman).

Published August 25, 2020

current section :

Operationalizing Fitness-for-Use Assessments

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Data Provenance

CHAPTER SECTIONS

Assessing Fitness for Use of Real-World Data Sources


Section 7

Data Provenance

Contributors

There is widespread variability in how information is captured in EHRs within and across healthcare systems. The same is true with how administrative claims are processed by health insurance providers. There is also variability in how data at sites are mapped between source systems and the value sets within a dataset or CDM. Knowledge about data collection practices and the decisions made in the source system–to–CDM translation can provide additional insight and context into the reliability of a dataset as it relates to data accrual (Johnson et al 2014). Many DRNs, for instance, ask their collaborators to complete surveys that describe the provenance of their data sources and to provide detail about the characteristics of their clinical workflows and/or source systems (Qualls et al 2018). Similarly, the NIH Collaboratory data quality white paper includes a recommendation that researchers explore the data collection and transformation procedures at each site, though this can be a highly manual process involving interviews and discussions between local data experts and the research team. It is important not to make this process overly burdensome, however, as these surveys should not be a onetime event, given that clinical systems change over time. Having multiple responses can provide a more complete longitudinal history. Provenance information can also be incorporated at the record level, for instance by assigning values as part of the data transformation process (eg, whether a diagnosis originated in a billing system, was entered by a clinician, or was derived from a note through natural language processing). These details are important because datasets that include records from one source and not the others or includes records from several sources that are not distinguished from one another will end up generating very different profiles.

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Resources

Evaluation System for Health Technology Coordinating Center (NESTcc) Data Quality Framework; NEST Coordinating Center; 2019.

REFERENCES

back to top

Johnson KE, Kamineni A, Fuller S, Olmstead D, Wernli KJ. 2014. How the provenance of electronic health record data matters for research: a case example using system mapping. EGEMS (Washington, DC). 2:1058. doi:10.13063/2327-9214.1058. PMID: Medline:25821838.

back to top

Qualls LG, Phillips TA, Topping J, et al. 2018. Evaluating Foundational Data Quality in the National Patient-Centered Clinical Research Network (PCORnet). eGEMs (Generating Evidence & Methods to improve patient outcomes.  Apr 13;6(1):3. doi:10.5334/egems.199. PMID: 29881761.

 


Version History

Published August 25, 2020

current section :

Data Provenance

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Data Quality Measures

CHAPTER SECTIONS

Assessing Fitness for Use of Real-World Data Sources


Section 4

Data Quality Measures

Contributors

Data quality is a multifaceted and context-dependent concept. Various groups that use secondary real-world data sources for research have extensive experience in assessing the quality of these data and have developed metrics and best practices for their use. Distributed research networks like the Health Care Systems Research Network (HCSRN; formerly the HMO Research Network) (Newton and Larson 2012; Ross et al 2014; Steiner et al 2014; Vogt et al 2004), the Center for Effectiveness & Safety Research (CESR), and Sentinel (Curtis et al 2012; Curtis et al 2014) developed processes to characterize administrative claims, with Sentinel informing much of the FDA's early work with real-world data and real-world evidence, particularly around pharmacoepidemiology (Carnahan and Moores 2012). The growing adoption of EHRs and the potential to use EHR data for research spurred a number of efforts to develop procedures that could assess EHR data quality, with 2 large-scale examples being the National Patient-Centered Clinical Research Network (PCORnet) and the Observational Health Data Science Initiative (OHDSI) (Califf 2014; Fleurence et al 2014; Hripcsak et al 2015).

Across these different approaches, there was varying use of terms to describe the purpose of each data quality measure (data check)—consistency, completeness, accuracy, precision, concordance, correctness, etc. This variation made it difficult to identify commonalities across initiatives or to compare the results of one check to another. As a result, Kahn et al (2016) developed a harmonized data quality framework that groups checks into 3 major categories: conformance (Does the format of the data adhere to the underlying model?), completeness (Are there values where we expect to see data populated?), and plausibility (Do the values that appear make sense?). Each check is further classified by its purpose, whether it is for verification or validation. Verification checks are used to determine whether data match or are consistent with internal expectations, and validation checks are used to compare against external "gold standard" benchmarks. The vast majority of checks that have been developed are verification checks. This is partly due to the difficulty of identifying gold standards to use as comparators, and also because validation is more straightforward against smaller cohorts (eg, patients with heart failure) as opposed to the entire population of a dataset, which may represent all patients who receive care within a healthcare system or all those enrolled in a health plan.

A given data check may apply to multiple fields or tables within a CDM (e.g., all required fields are populated), so networks will often talk about data check instances, or data check measures, which represent a specific instantiation of the check against a given table/field. In this manner, a handful of checks can lead to hundreds or thousands of data check measures. For instance, within PCORnet, 38 data checks translate into more than 1400 data check measures (PCORnet Distributed Research Network Operations Center) while OHDSI has 20 checks that resolve into more than 3300 data check measures. The vast majority of data checks tend to be related to conformance, so CDMs with a higher number of tables and fields will end up with more data check measures.

Example verification checks from several existing distributed research networks are provided in the table below. The source (network) of each check is listed, unless a similar version of the check is used by multiple networks. These examples were taken from publicly available material. While many networks and organizations that use real-world data have quality assurance processes, those that make the content and programs behind these processes freely available remain the exception rather than the norm. (Some groups will provide material only upon request.) For the field to advance and arrive at a consensus for a set of minimum necessary checks, greater sharing and transparency of assessment method are needed. This has become increasingly relevant during the COVID-19 pandemic, because of the potential misuse or misrepresentation of data—as in the case of Surgisphere, which analyzed and published information on hydroxychloroquine from EHRs but was unable to verify their data, resulting in a retraction of a peer-reviewed article (Ledford and Van Noorden 2020). There is also the need to rapidly run parallel analyses across a series of real-world data sources in a way that generates comparable results.

Table 1. Categories of Data Quality Checks and Examples From Distributed Research Networks

Category Subcategory Description Data Check Example
Conformance Value Determines whether the data conform to the formats of the data model used to store them Required fields do not contain values outside of the CDM specifications (multiple networks)
Relational Determines whether the data agree with the constraints imposed by the database used to store them (eg, primary or foreign key relationships) All fields are present in each CDM table (multiple networks)
Calculation Evaluates whether variables derived computationally yield valid results Enrollment periods do not overlap, and are not duplicates or subsets of one another (Sentinel)
Completeness Examines whether expected values are present (single time point or longitudinally) Fewer than 50% of patients with an encounter have diagnosis data in the CDM (PCORnet)
Plausibility Uniqueness Determines whether multiple values exist when only one value is expected Patient does not have multiple inpatient admissions to the same facility on the same day (CESR)
Atemporal Measures whether data agree with expected values More than X% of records fall into the lowest or highest categories of age, height, weight, diastolic blood pressure, etc (multiple networks)
Temporal Examines whether variables change as expected over a specified time period More than X% of records have illogical date relationships (eg, events before date of birth, events after date of death) (multiple networks)

The harmonized terminology proposed by Kahn et al (2016) provides a good organizing framework considering what to assess within a dataset. Yet there are a few gaps. For example, the concept of timeliness or data latency (ie, Given a dataset, how recent are the records?) does not exist in the framework. So, to avoid a new entry, those checks would need to be framed in terms of completeness or plausibility (eg, Does the number of records for a given month look "complete" given expectations? Is the monthly volume of lab records an outlier compared to recent trends?). In addition, persistence checks—those that assess changes in the dataset over time (eg, Is there a large drop in the number of records or number of patients from one refresh to the next?)—do not have a ready home in the framework. They can be considered a type of completeness check, but instead of record-level completeness, the comparison is between 2 versions of the same dataset from different time points. Given that some studies rely on a single data extract, persistence checks may not be as relevant to studies that use multiple data pulls over the course of several years. However, given that persistence issues can appear in any dataset, it is important that study teams have a sense of performance of this measure, even if they cannot measure it directly with the data they receive.

The NIH Collaboratory's Electronic Health Records Core Working Group developed a white paper and recommendations for assessing data quality in pragmatic clinical trials. The recommendations include a formal assessment against 3 domains: accuracy, completeness, and consistency for key data elements, such as endpoints. The report offers examples of metrics that can be used for each dimension, and the optimal metric depends on the nature of the data and the research study. (Note that the white paper was developed before the harmonized framework by Kahn et al, so consistency would now be referred to as conformance).

Data Quality Checks

The differences between the terminologies describing EHR and claims-based data checks and the language used by the FDA to describe whether a dataset is fit for use can lead to confusion when comparing and/or mapping between them. Using data checks to assess the quality of a dataset can be considered a type of data assurance. Such a process is a necessary component of demonstrating data assurance, but it is likely insufficient to satisfy all requirements in that area or demonstrate overall fitness for use. As noted above, most of the data checks developed thus far are for verification purposes, which speaks to the reliability of the data. However, it is possible that a dataset can satisfy all of these data checks and still be inappropriate for a specific research question because of relevance concerns. Validation checks could be considered as a way to assess the relevance of a dataset, since those checks would be more population-focused and targeted to specific outcomes or variables within them (eg, distribution of HbA1c values in a population of patients with diabetes), but many of these checks will need to be developed and evaluated in a study-specific context. One area of the FDA's description of fitness for use that is more difficult to handle directly with data checks are the concepts of provenance and traceability. For EHR and claims-based data sources, these concepts may be best handled through process documentation or otherwise describing steps of data transformation from the original source system to the final analytic dataset.

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Resources

Assessing Fitness-for-Use of Clinical Data for PCTs; 2-page handout for assessing data quality and fitness for use (2023)

Assessing Data Quality for Healthcare Systems Data Used in Clinical Research; NIH Collaboratory white paper (2015)

REFERENCES

back to top

Califf RM. 2014. The Patient-Centered Outcomes Research Network: a national infrastructure for comparative effectiveness research. N C Med J. 75:204-210. doi:10.18043/ncm.75.3.204. PMID: 24830497.

Carnahan RM, Moores KG. 2012. Mini-Sentinel's systematic reviews of validated methods for identifying health outcomes using administrative and claims data: methods and lessons learned. Pharmacoepidemiol Drug Saf. 21 Suppl 1:82-89. doi:10.1002/pds.2321. PMID: 22262596.

Curtis LH, Weiner MG, Boudreau DM, et al. 2012. Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf. 21 Suppl 1:23-31. doi:10.1002/pds.2336. PMID: 22262590.

Curtis LH, Brown J, Platt R. 2014. Four health data networks illustrate the potential for a shared national multipurpose big-data network. Health Aff (Millwood). 33(7):1178-1186. doi:10.1377/hlthaff.2014.0121. PMID: 25006144.

Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. 2014. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc. 21(4):578-582. doi:10.1136/amiajnl-2014-002747. PMID: 24821743.

Hripcsak G, Duke JD, Shah NH, et al. 2015. Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers. Stud Health Technol Inform. 216:574-578.  PMID: 26262116.

back to top

Kahn MG, Callahan TJ, Barnard J, et al. 2016. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 4(1):1244. doi:10.13063/2327-9214.1244. PMID: 27713905.

Ledford H, Van Noorden R. 2020. High-profile coronavirus retractions raise concerns about data oversight. Nature. 582(7811):160. doi:10.1038/d41586-020-01695-w. PMID: 32504025.

Newton KM, Larson EB. 2012. Learning health care systems: leading through research: the 18th Annual HMO Research Network Conference, April 29-May 2, 2012, Seattle, Washington. Clin Med Res. 10(3):140-142. doi:10.3121/cmr.2012.1099. PMID: 22904375.

Ross TR, Ng D, Brown JS, et al. 2014. The HMO Research Network Virtual Data Warehouse: a public data model to support collaboration. EGEMS (Wash DC). 2(1):1049. doi:10.13063/2327-9214.1049. PMID: 25848584.

Steiner JF, Paolino AR, Thompson EE, Larson EB. 2014. Sustaining research networks: the twenty-year experience of the HMO Research Network. EGEMS (Wash DC). 2(2):1067.  PMID: 25848605.

Vogt TM, Lafata JE, Tolsma DD, Greene SM. 2004. The role of research in integrated health care systems: the HMO Research Network. Perm J. 8(4):10-17.  PMID: 26705313.


Version History

August 26, 2022: Updated as part of annual review (changes made by K. Staman).

Published August 25, 2020

current section :

Data Quality Measures

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Defining Fitness for Use

CHAPTER SECTIONS

Assessing Fitness for Use of Real-World Data Sources


Section 2

Defining Fitness for Use

Contributors

Given the widespread adoption of electronic health records (EHRs) and the general availability of administrative claims in electronic formats, there is a corresponding interest in leveraging these data sources for research (Bayley et al 2013; Botsis et al 2010; Coorevits et al 2013; Etheredge 2007; Friedman et al 2010; Hersh et al 2013; Jensen et al 2012; Weiner and Embi 2009). This interest has spurred the development of approaches to assess the underlying data quality, often by defining data checks that can be executed against a dataset (Kahn and Todd 2008; Kahn et al 2010; Kahn et al 2012; Khare et al 2017; Qualls et al 2018; Rogers et al 2019).

Data checks, or metrics, can describe characteristics of a dataset, including missing values, outliers, and frequency distributions. However, determining whether the result or value of a particular metric is good or bad depends on the needs of the research project and the intended use of the data. For example, in determining eligibility for a study, it may be sufficient to simply know if a patient has an available laboratory result, as study coordinators will need to complete a screening form that involves chart review. In this case, using the presence of a result (regardless of unit or value) is an adequate filter. However, if a lab result were going to serve as the biomarker endpoint for a trial, more rigorous thresholds might be needed. For example, each result may need to have an actual value and a unit of measure and a measure of confidence in the accuracy of the result. In other words, when it comes to using real-world data in clinical research, datasets must be considered in the context of a specific project or analysis to determine whether they are suitable or fit for use.

“Fitness-for-use” is a nebulous concept, and defining it is more art than science, with few hard and fast rules established thus far. When it comes to the use of real-world evidence derived from real-world data for regulatory decision-making, the US Food and Drug Administration (FDA) has provided guidance through the recommendations contained in the Framework for FDA's Real-World Evidence Program (FDA 2018) and further highlighted in Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products (FDA 2021). The FDA defines fitness for use in terms of relevance and reliability. Relevance “includes the availability of key data elements (exposures, outcomes, covariates) and sufficient number of representative patients for the study,” while reliability is focused on “data accuracy, completeness, provenance and traceability” (FDA 2021). The definitions of these terms are described in more detail below. The FDA has stated that it is able to respond to study teams regarding whether a specific set of assessments is sufficient to determine the fitness for use of a real-world data source for a given study or analysis. The FDA has not yet approved a given assessment package that will always be sufficient for determining suitability for all studies that use real-world data. As the field gains more experience and confidence with the use of real-world data and real-world evidence, we expect more refinement in this area.

Key Point: On a study-by-study basis, FDA will work with stakeholders to evaluate whether a given assessment is suitable for a particular research question.

Relevance

A real-world data source is said to be relevant if:

  • the data apply to question at hand;
    • For example, the data contain sufficient detail to capture the use or exposure of the the product or device and/or the outcome of interest.
  • the data are amenable to sound clinical and statistical analysis; and
    • For example, the data can be used to answer the specified question using the proposed statistical plan.
  • the data and evidence the source provides are interpretable using informed clinical and statistical judgement.
    • For example, the use of a device or product in a real-world population is representative of what is captured in the data source, is generalizable to the relevant population under study, etc (FDA 2018).

The "sufficient detail" needed to capture use or exposure may depend on the intended use case. For instance, medication prescription data may be sufficient if an investigator is planning to screen patients for a trial, while dispensing or medication administration data would be a more reliable indicator for exposure as part of an outcome or endpoint. Along the same lines, "amenable to analysis" means that the data are specific enough to support the study question. For example, having death data may be sufficient for one study but not another if it is important to distinguish between all-cause mortality and mortality due to a specific cause.

Investigators will often have a general sense of the relevance of a data source before attempting to use it as part of a study, but it may be necessary to include additional analyses to better demonstrate applicability. This is particularly true for real-world data sources like EHRs. While administrative claims tend to have complete capture of all medically attended events during a given enrollment period, the same concept does not exist within EHRs. Encounters may occur outside a given health system; even within a health system, data collection within the EHR can be variable, particularly for workflows that are not tied to reimbursement. Practices vary by hospital, clinic, and/or provider, and the availability of data for longitudinal analysis may be affected by when the EHR or other clinical information system was deployed across the health system. All of these factors should be taken into account when assessing fitness for use, particularly for studies that rely on EHR data from different healthcare systems.

Reliability: Data Accrual

Data accrual relates to aspects of how the data in the source are collected or captured. Reliable documentation of data accrual methods for a real-world data source includes:

  • an operational manual that pre-specifies the data elements to be collected;
  • the definitions of those data elements;
  • methods of data aggregation, transformation, and documentation; and
  • a relevant time window, etc (FDA 2018).

This information is expected for real-world data sources like patient or device registries (Agency for Healthcare Research and Quality 2010; International Medical Device Regulators Forum Group 2015; Krucoff et al 2015; Patient-Centered Outcomes Research Institute 2012), as well as those that collect data directly as part of a study (eg, patient-reported outcomes or patient-generated data). Secondary real-world data sources like EHRs and administrative claims lack many of these characteristics, though it is possible to approximate some aspects through items like data dictionaries or data model specifications, provenance surveys that detail the source of certain data elements, and workflow descriptions that describe how data elements were captured over time, including any changes or modifications (eg, patient-reported outcomes initially being captured in an in-clinic kiosk in the waiting room and later supporting the ability to have patients complete from home via questionnaires delivered through the healthcare system’s patient portal). Documentation of the procedures and specifications used to translate EHR data from the source system to the target database (eg, a common data model or database extract) can provide further insight into the practices of data accrual.

Reliability

For secondary data sources like EHRs and administrative claims, data reliability concerns aspects of data quality and provenance over the “life cycle” of the data, or the steps that occur as data are curated and transformed from initial capture in the source system(s) to data repositories/common data models to a final analytic dataset. Activities to ensure data reliability include the execution of data checks that can describe the completeness, conformance, and plausibility of the data (see Section 4), and documentation of the data quality processes for the various transformation steps along the data life cycle to ensure the overall validity and integrity of the data (see Section 7).

Real-world data sources like patient or device registries (Gliklich et al 2010; International Medical Device Regulators Forum Group 2015; Krucoff et al 2015; Patient-Centered Outcomes Research Institute 2012), as well as data sources that are collected directly as part of a study (eg, patient-reported outcomes or patient-generated data), can provide a template for the types of information that should be documented to demonstrate reliability. While secondary real-world data sources like EHRs and administrative claims lack some of the characteristics of these data sources, it is possible to approximate some aspects through items like data dictionaries or data model specifications, provenance surveys that detail the source of certain data elements, and workflow descriptions that describe how data elements were captured over time, including any changes or modifications (eg, patient-reported outcomes initially being captured in an in-clinic kiosk in the waiting room and later supporting the ability to have patients complete from home via questionnaires delivered through the healthcare system’s patient portal). Documentation of the procedures and specifications used to translate EHR data from the source system to the target database (eg, a common data model or database extract) can provide further insight into the practices of data accrual.

 

SECTIONS

CHAPTER SECTIONS

sections

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments

Resources

Leveraging RWE to Support Regulatory Decisions–An Update on Efforts to Inform Policy; NIH Collaboratory Grand Rounds; March 15, 2019

Expanding Use of Real-World Evidence: A National Academies Workshop Series; NIH Collaboratory Grand Rounds; April 27, 2018

REFERENCES

back to top

Gliklich RE, Dreyer NA, eds. 2010. Registries for Evaluating Patient Outcomes: A User's Guide. Rockville, Maryland: Agency for Healthcare Research and Quality. https://effectivehealthcare.ahrq.gov/products/registries-guide-4th-edition/. Accessed August 24, 2020.

Bayley KB, Belnap T, Savitz L, Masica AL, Shah N, Fleming NS. 2013. Challenges in using electronic health record data for CER experience of 4 learning organizations and solutions applied. Med Care. 51:S80-S86. doi:10.1097/MLR.0b013e31829b1d48. PMID: 23774512.

Botsis T, Hartvigsen G, Chen F, Weng C. 2010. Secondary use of EHR: data quality issues and informatics opportunities. Summit Transl Bioinform. 2010:1-5. PMID: 21347133.

Coorevits P, Sundgren M, Klein GO, et al. 2013. Electronic health records: new opportunities for clinical research. J Intern Med. 274:547-560. doi:10.1111/joim.12119. PMID: 23952476.

Etheredge LM. 2007. A rapid-learning health system. Health Aff (Millwood). 26(2):w107-w118. doi:10.1377/hlthaff.26.2.w107. PMID: 17259191.

FDA. 2018. Framework for FDA's Real-World Evidence Program. https://www.fda.gov/media/120060/download. Accessed August 25, 2020.

FDA. 2021. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision- Making for Drug and Biological Products. https://www.fda.gov/media/152503/download Accessed August 26, 2021.

Friedman CP, Wong AK, Blumenthal D. 2010. Achieving a nationwide learning health system. Sci Transl Med. 2:57cm29. doi:10.1126/scitranslmed.3001456. PMID: 21068440.

Hersh WR, Cimino J, Payne PRO, et al. 2013. Recommendations for the use of operational electronic health record data in comparative effectiveness research. EGEMS (Washington, DC). 1(1):1018. doi:10.13063/2327-9214.1018. PMID: 25848563.

International Medical Device Regulators Forum Group. 2015. Patient Registry: Essential Principles. https://www.imdrf.org/sites/default/files/2021-09/imdrf-cons-essential-principles-151124.pdf. Accessed December 3, 2025.

Jensen PB, Jensen LJ, Brunak S. 2012. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 13:395-405. doi:10.1038/nrg3208. PMID: 22549152.

back to top

Kahn MG, Todd J. 2008. Comparative quality measures: putting evidence above expediency. Pediatrics. 122(1):182-183. doi:10.1542/peds.2008-1042. PMID: 18596002.

Kahn MG, Eliason BB, Bathurst J. 2010. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proc. 2010 Nov 13:356-360. PMID: 21347000.

Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF. 2012. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med Care. 50 Suppl(0):S21-S29. doi:10.1097/MLR.0b013e318257dd67. PMID: 22692254.

Khare R, Utidjian L, Ruth BJ, et al. 2017. A longitudinal analysis of data quality in a large pediatric data research network. J Am Med Inform Assoc. 24(6):1072-1079. doi:10.1093/jamia/ocx033. PMID: 28398525.

Krucoff M, Normand S, Edwards F, et al. 2015. Recommendations for a National Medical Device Evaluation System: Strategically Coordinated Registry Networks to Bridge Clinical Care and Research. https://www.fda.gov/media/93140/download. Accessed August 24, 2020.

Patient-Centered Outcomes Research Institute. 2012. Standards in the Conduct of Registry Studies for Patient-Centered Outcomes Research. PCORnet. PCORnet Common Data Model (CDM). https://pcornet.org/data-driven-common-model/. Accessed January 26, 2017.

Qualls LG, Phillips TA, Topping J, et al. 2018. Evaluating foundational data quality in the national Patient-Centered Clinical Research Network (PCORnet). EGEMS (Wash DC). 6(1):3. doi:10.5334/egems.199. PMID: 29881761.

Weiner MG, Embi PJ. 2009. Toward reuse of clinical data for research and quality improvement: the end of the beginning? Ann Intern Med. 151(5):359-360. doi:10.7326/0003-4819-151-5-200909010-00141. PMID: 19638404.


Version History

December 3, 2025: Updated hyperlinks (changes made by G. Uhlenbrauck).

August 26, 2022: Updated as part of annual review (changes made by K. Staman).

Published August 25, 2020

current section :

Defining Fitness for Use

  1. Introduction
  2. Defining Fitness for Use
  3. Evaluating Fitness for Use
  4. Data Quality Measures
  5. Use of Medicare Data in PCTs
  6. Data Source Accuracy: Case Study from TRANSLATE-ACS
  7. Data Provenance
  8. Operationalizing Fitness-for-Use Assessments