Acquiring Real-World Data
Section 9
Case Study: The IMPACT-AFib Trial
Whereas most of the pragmatic trials discussed in the Living Textbook—including the NIH Collaboratory Trials—are embedded within healthcare systems, the IMPACT-AFib trial (Implementation of a Randomized Controlled Trial to Improve Treatment With Oral Anticoagulants in Patients With Atrial Fibrillation; NCT03259373) was embedded in health insurance plans participating in the FDA's Sentinel System, a large distributed database of EHRs and administrative claims. Conducted between 2017 and 2019, IMPACT-AFib tested whether an educational mailing to patients and their clinicians improved dispensing of oral anticoagulants in a large population of patients with atrial fibrillation (Pokorney et al 2022). The FDA supported the trial as a proof of concept within FDA-Catalyst, which leverages the Sentinel System for research.
As the first to conduct a randomized clinical trial using the Sentinel platform, the IMPACT-AFib investigators encountered several challenges during the planning and implementation of the study (Cocoros et al 2019; Cocoros et al 2023; Garcia et al 2020). In this section, we summarize the challenges and the lessons the study team learned.
IMPACT-AFib and Sentinel
The FDA launched the Sentinel System in 2008 to support retrospective postmarket safety surveillance (Platt et al 2018). Participating "data partners" (including academic medical centers, healthcare systems, and health insurance companies) use the Sentinel Common Data Model to transform their medical billing data, administrative claims data, and EHR data into a standardized format and remove identifiable patient information. The resulting collection of datasets makes up a large, harmonized, distributed database that can be analyzed quickly through the execution of routine querying tools.
The IMPACT-AFib research team used this large distributed database to conduct a randomized clinical trial for the first time. In the planning phase of the trial, they retrospectively analyzed claims data from 5 large health plans to estimate the number of health plan members who might be eligible for inclusion in the trial. Partners at each participating health insurance company executed a software program distributed by the study team to query their local, analysis-ready dataset, which had been created using the Sentinel Common Data Model. Analysis of these data allowed the study team to understand baseline rates of treatment initiation and the outcomes of interest in the population. This information supported development of sample size estimates, statistical power calculations, and other features of the trial protocol and analysis plan. Also, because the analysis used Sentinel's distributed database approach, no patient-level data were shared outside the data partners’ firewalls.
In the implementation phase of the trial, each data partner supplemented its analysis-ready dataset with additional enrollment and pharmacy dispensing data for events that occurred after the data partner's most recent data in the Sentinel distributed database. (Sentinel incorporates a lag in the availability of routine data to ensure the claims are closed before they are used in analyses, whereas enrollment and pharmacy data are available sooner.) Each data partner created a crosswalk to link its analysis-ready dataset to internal records of patient and clinician contact information to allow for mailing the educational materials that constituted the study intervention. After the trial's follow-up period, the data partners submitted deidentified, aggregate results to the trial's coordinating center for analysis.
Lessons From ADAPTABLE
Another large-scale clinical trial that relied on data from health insurance plans, ADAPTABLE (Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-Term Effectiveness; NCT02697916) compared the effectiveness of 2 different daily doses of aspirin commonly used to prevent heart attack and stroke in patients with established cardiovascular disease. In scenarios where researchers had a relationship with patients via a study or research registry, they were able obtain consent and health plan authorization, allowing them to collect administrative claims data. By linking patient-reported outcomes with clinical data from the EHR, they created an integrated patient dataset. Acquiring and using data from health plans posed some challenges. For example, gaps in claims data can arise due to short durations of enrollment follow-up time or a lack of deep clinical detail. In ADAPTABLE, it was difficult to identify patient information such as aspirin allergies or a history of gastrointestinal issues that might be more easily identified in medical records rather than administrative claims. Overall, the trial outcomes revealed that collecting health plan data enabled reliable identification of eligible participants for pragmatic trials. This approach significantly enhanced participant recruitment efforts and provided a longitudinal electronic method for ascertaining endpoints. Moreover, patient engagement played a vital role in facilitating health plan outreach. Learn more about health plan–based recruitment and follow-up in the ADAPTABLE trial.
Challenges
The IMPACT-AFib study team encountered several challenges in acquiring and using real-world health plan data as the basis for the trial. The challenges are described below.
- In addition to providing an analysis-ready Sentinel dataset, it was necessary for each data partner to identify an in-house investigator who "brought essential expertise regarding the existence, availability, interpretation of, and access to the electronic health [data] that was the foundation of this study" and to "a wide array of health plan specific operational data systems" (Garcia et al 2020). These investigators stewarded the project through their organization and operated behind their organizations' firewalls to work with data not available to the trial’s other investigators.
- As noted above, each data partner had to supplement its analysis-ready dataset with "fresh" enrollment and pharmacy data before the study intervention to identify patients who initiated oral anticoagulation therapy after the data partner’s most recent update to the Sentinel distributed database. The process for extracting the fresh data varied slightly between the data partners, necessitating quality-assurance checks, both automated and manual.
- Data for some medications are not captured completely in health plan claims. In IMPACT-AFib, the investigators anticipated that some patients would pay out of pocket for warfarin, an anticoagulant, because of its relatively low cost. Thus, they used a proxy for warfarin therapy in the claims data (claims for international normalized ratio [INR] tests or test results) to identify patients already receiving treatment.
- More than one-third of the provider identifiers in the health plan data mapped to physician groups, facilities, or institutions rather than to individual providers. This challenge required the study team to develop an alternative plan for identifying these clinicians in the data and for rolling out the intervention, a process that added several months to the study timeline.
- Some clinicians have multiple provider identifiers both within and between health plans, making identification of unique clinicians difficult. Because IMPACT-AFib used individual-level randomization of the patients, and because removing duplicate clinicians from the data would have added considerably to the study's workload, the study team's ability to account for clustering of patients within these providers was limited.
Lessons Learned
The IMPACT-AFib research team shared several important lessons from their experience with study planning and implementation (Cocoros et al 2023).
- In general, a limitation of studies based on administrative claims data is that they are "subject to misclassification due to the use of diagnostic and procedure codes to capture conditions, exposures, and outcomes of interest" (Cocoros et al 2019). Planning for these studies requires substantial time and effort to identify valid proxy variables and to devise robust inclusion and exclusion criteria and minimize such misclassification.
- Involving staff from the participating health plans throughout the study, including during the planning phase, is critical to a trial’s success.
- The research team should conduct feasibility analyses and other preliminary work to aid in calculating sample size estimates, establishing robust methods for identifying study cohorts, and developing the trial protocol and statistical analysis plan.
- If the study intervention will require contact with patients or clinicians, as in IMPACT-AFib, the research team must establish agreements with health plan staff ahead of time to specify how they will use their internal source data to identify study participants and their contact information.
SECTIONS
Resources
IMPACT-AFib: An 80,000 Person Randomized Trial Using the Sentinel Initiative Platform
NIH Pragmatic Trials Collaboratory PCT Grand Rounds; January 5, 2018
ADAPTABLE Recruitment and Follow-up: Health Plan Research Network Engagement
NIH Pragmatic Trials Collaboratory PCT Grand Rounds; September 13, 2019
REFERENCES
Cocoros NM, Gurwitz JH, Cziraky MJ, et al. 2023. Pragmatic guidance for embedding pragmatic clinical trials in health plans: Large simple trials aren't so simple. Clin Trials. 20(4):416-424. doi: 10.1177/17407745231160459. PMID: 37322894.
Cocoros NM, Pokorney SD, Haynes K, et al. 2019. FDA-Catalyst—Using FDA's Sentinel Initiative for large-scale pragmatic randomized trials: Approach and lessons learned during the planning phase of the first trial. Clin Trials. 16(1):90-97. doi: 10.1177/1740774518812776. PMID: 30445835.
Garcia CJ, Haynes K, Pokorney SD, et al. 2020. Practical challenges in the conduct of pragmatic trials embedded in health plans: Lessons of IMPACT-AFib, an FDA-Catalyst trial. Clin Trials. 17(4):360-367. doi: 10.1177/1740774520928426. PMID: 32589056.
Platt R, Brown JS, Robb M, et al. 2018. The FDA Sentinel Initiative - an evolving national resource. N Engl J Med. 379(22):2091-2093. doi: 10.1056/NEJMp1809643. PMID: 30485777.
Pokorney SD, Cocoros N, Al-Khalidi HR, et al. 2022. Effect of mailing educational material to patients with atrial fibrillation and their clinicians on use of oral anticoagulants: a randomized clinical trial. JAMA Netw Open. 5(5):e2214321. doi: 10.1001/jamanetworkopen.2022.14321. PMID: 35639381.