In a study supported by the NIH Collaboratory, researchers developed and validated a new sample size formula for detecting heterogeneity of treatment effect in cluster randomized trials. The work was published this month in Statistics in Medicine.
Cluster randomization is frequently used in pragmatic clinical trials embedded in healthcare systems. Although cluster randomized trials are typically designed to evaluate the overall treatment effect in a study population, investigators are increasingly interested in studying differential treatment effects among subgroups.
The NIH Collaboratory investigators used extensive computer simulations to validate the new formula. They illustrate the procedure in a dataset from a large clinical trial.
In a previous study published last year, the same research team used computer simulation models validated by real-data simulations to reveal the influence of baseline covariate imbalance on treatment effect bias.
This work was supported within the NIH Collaboratory by the NIH Common Fund through a cooperative agreement from the Office of Strategic Coordination within the Office of the NIH Director, and by a research supplement from the NIH Common Fund to promote diversity in health-related research.
Jon Tilburt, MD Professor of Medicine and Biomedical Ethics Mayo Clinic
Andrea Cheville, MD Professor of Physical Medicine and Rehabilitation Mayo Clinic
Topic
Learning While Sprinting: A One-Year Retrospective from the NOHARM Pragmatic Trial
Keywords
PRISM; NIH Heal Initiative; NOHARM; Postoperative care; Nonpharmacologic pain care (NPPC); Stepped wedge; Cluster-randomized trial; Electronic health records (EHRs); Patient engagement; Clinical decision support
Key Points
The Nonpharmacologic Options in Postoperative Hospital-based and Rehabilitation Pain Management (NOHARM) NIH Collaboratory Trial is completing its pilot phase. This embedded, stepped-wedge PCT will test a sustainable strategy in perioperative, nonpharmacologic pain management that preserves patient function, honors patient values, and maintains the availability of opioids as a last resort.
NOHARM is a pragmatic, EHR-integrated intervention that bundles a portal-based conversation guide that captures patient preferences for postsurgical pain care and a clinician-directed decision support tool.
Nonpharmacologic pain care management options include walking, yoga, tai chi, acupressure, massage, meditation, and relaxation.
Discussion Themes
Opioids are insufficient in postsurgical care. Guidelines recommend nonpharmacologic pain care (NPPC), but there have not been studies showing how to make NPPC more viable.
The COVID-19 pandemic caused disruption in scheduled surgeries and also air travel, which precluded on-the-ground support at two study sites. However, the team was able to adjust recruitment methods during the pilot phase.
What was the team’s proactive process in working with the IRB in order to obtain a waiver of consent?
The NOHARM intervention has sustained high-level institutional support despite the impact of COVID-19.
Fibromyalgia (FM) is a condition of widespread pain that is worsened with physical activity. It involves chronic musculoskeletal and visceral pain and is often accompanied by fatigue, depression, or anxiety.
Transcutaneous electrical nerve stimulation (TENS) is a technique that uses a device to deliver an electric impulse through the skin. Treatment with TENS has been shown to improve resting and movement-evoked pain and fatigue.
While physical therapists generally are trained in the use of TENS, the technique remains underused in clinical practice.
The goal of the FM TIPS pragmatic trial is to determine, in a real-world clinical setting, whether physical therapy combined with TENS for patients with FM is more effective than physical therapy alone. The study is being piloted in 24 sites across 5 physical therapy health systems.
Discussion Themes
While COVID-19 has had an impact on piloting the FM TIPS study, some kind of physical therapy will be possible through telemedicine. Other challenges include that conducting embedded research in physical therapy clinics is new, and there are multiple different EHR systems in use across the partnering clinics.
The recently published results from Fibromyalgia Activity Study With TENS (FAST) showed that TENS can be safely used in addition to other treatments to improve pain and fatigue in women with fibromyalgia in the setting of a randomized controlled trial.
FM TIPS is one of the NIH HEAL Initiative’s PRISM (Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing) studies.
A co–principal investigator of the Fibromyalgia TENS in Physical Therapy Study (FM TIPS), an NIH Collaboratory Trial supported by the NIH HEAL Initiative, will be featured in the next session of NIH Collaboratory Grand Rounds on July 24. The project is the first of the NIH Collaboratory Trials funded through the Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing (PRISM) program to be featured in Grand Rounds.
FM TIPS will use a cluster randomized trial design to assess the feasibility of using transcutaneous electrical nerve stimulation (TENS) in addition to physical therapy for the treatment of fibromyalgia. The study will also determine whether the use of TENS improves symptoms of fibromyalgia, increases adherence to physical therapy and the likelihood of meeting therapeutic goals, and reduces medication use. FM TIPS is currently in the planning phase, which involves recruiting physical therapy sites into the embedded pragmatic clinical trial, understanding usual practice to inform trial processes, and ensuring the adequacy of trial infrastructure.
FM TIPS is supported by the NIH through the NIH HEAL Initiative under an award from the National Institute of Arthritis and Musculoskeletal and Skin Diseases.
Rachel Richesson, PhD, MPH Associate Professor, Informatics Duke University School of Nursing
Devon Check, PhD Assistant Professor, Population Health Sciences Department of Population Health
Topic
Choosing What to Measure and Making it Happen: Your Keys to Pragmatic Trial Success
Keywords
Measuring outcomes; Phenotypes; Data quality; Data linkage; Endpoints; Patient-reported outcomes (PROs)
Key Points
Endpoints and outcomes for embedded pragmatic clinical trials (ePCTs) should be meaningful to providers and patients and be relatively easy to collect as part of routine care. Endpoints and outcomes also should be clearly defined and reproducible.
Patient-reported outcomes (PROs) are often the best way to measure quality of life, but come with challenges in that they are not routinely or consistently used in clinical care nor are regularly recorded in the EHR.
To fully capture all care—complete longitudinal data—it is often necessary to link research and insurance claims data.
Discussion Themes
Data in EHRs are an important component of ePCTs. While ePCTs strive for efficiency, there remain tradeoffs. Sometimes it will be necessary to collect data outside of the EHR to ensure important and compelling results.
It is also important that the endpoint that is conveniently available will also be accepted as influential for stakeholders when the trial results are disseminated.
In the future, it is essential that more meaningful data as well as more patient-reported outcomes are routinely collected and incentivized.
In 2012, the NIH Common Fund established the NIH Health Care Systems Research Collaboratory. The goal of the program is to improve the way clinical trials are conducted by creating an infrastructure for collaborative research with healthcare systems. The NIH Collaboratory launched with a Coordinating Center, Core Working Groups, and NIH Collaboratory Trials to conduct embedded pragmatic clinical trials (ePCTs) in partnership with healthcare system leaders and to work collaboratively with the NIH to solve problems as they arise, develop best practices, and share lessons and resources to with others conducting ePCTs.
Collaboratory Mission: Strengthen the national capacity to implement cost-effective large-scale research studies that engage healthcare delivery organizations as research partners.
With the first round of NIH Collaboratory Trials nearing completion, the project teams are beginning to publish results and share lessons with other researchers. We asked the principal investigators of the most recently completed projects to share insights about the important contributions of their studies.
Congratulations on finishing your NIH Collaboratory Trial: What do you think is the most important contribution of your study?
ABATE was conducted to determine whether routine bathing and showering with chlorhexidine soap would reduce multidrug-resistant organisms and bloodstream infections compared with usual care. The trial was conducted in 53 HCA Healthcare hospitals (194 non–critical care units)and included 340,000 patients in the intervention period.
“We found that there was no overall benefit to universal antiseptic bathing in non–intensive care units (ICUs). This is in stark contrast to the huge benefit demonstrated in ICUs in the REDUCE-MRSA trial, and may reflect the fact that non–critical care patients stay only a few days in the hospital and are less likely to develop infection. Nevertheless, we did find that antiseptic bathing and nasal decolonization reduced bloodstream infections and antibiotic-resistant organisms by over 30% in patients with devices outside of the ICU. This is important because they are 10% of the non-ICU population, but responsible for over half of bloodstream infections. They provide a valuable targeted population who appear to benefit from this intervention.”
LIRE was conducted to test the effectiveness of a simple and inexpensive intervention: inserting epidemiologic benchmarks into lumbar spine imaging reports. The goal of the trial was to reduce subsequent tests and treatments, including cross-sectional imaging (such as magnetic resonance imaging and computed tomography), opioid prescriptions, spinal injections, or surgery.
“I think that one of the most important contributions of the LIRE trial was demonstrating the feasibility of randomizing hundreds of thousands of patients to receive or not receive an intervention that we inserted into the radiology report. Before our trial began, there was a fair amount of skepticism about whether radiologists would accept routinely inserting prevalence information into their reports on a wide scale. We showed without a doubt that it was feasible.”
PPACT was designed to assess the potential benefit of helping patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting in three Kaiser Permanente (Northwest, Georgia, and Hawaii) involving approximately 800 patients.
“We started a trial when everybody was still uncertain about what the trade-offs between external validity (and real-world issues that are important for implementation) and the rigor of internal validity. I don’t know if we got that right. There was an assumption that the trial needed to be cluster randomized, and I think it’s informative that only 1 of the 11 NIH-DOD-VA Pain Management Collaboratory trials was cluster randomized. We needed to be able to incubate, have embedded teams stay over time, and really shift the culture. Patients needed to get used to the idea of non-pharmacotherapy over several months, and we may have had more success if we had individually randomized our cohort. I learned a lot in this process.”
PROVEN was designed to evaluate the effectiveness of advance care planning video tools in the nursing home setting by partnering with 2 large healthcare systems that operate 492 nursing homes nationwide.
“PROVEN found an ACP Video Program did not significantly impact hospital transfers, burdensome treatments, or hospice enrollment among nursing home residents with advanced illness, however intervention fidelity was low. Nonetheless, PROVEN was one of the first large pragmatic trials conducted in US nursing homes. Thus, I feel its greatest contribution was setting a foundation of knowledge for the field in terms of methodologies that enable pragmatic trials in this setting and challenges to overcome.”
STOP CRC was conducted to determine whether EHR-embedded tools and clinic staff training in how to implement a mailed fecal immunochemical test (FIT) outreach program could increase colorectal cancer screening uptake among patients with historically lower CRC screening rates and worse CRC outcomes, such as those with low income, or who are on Medicaid or underinsured. STOP CRC was conducted in 26 Federally Qualified Health Centers (FQHCs) in Oregon and California and involved approximately 41,000 patients.
“The ability to work with FQHCs and their new electronic data systems was an important contribution. FQHC settings are not organized healthcare systems, such as Kaiser Permanente, where research is more routine. I think we contributed to the success of this type of research and enabled the FHQCs’ ability to do more of it.” — Dr. Beverly Green
“Our study designed real-time electronic health record tools to allow clinics to mail cancer screening tests to adults who were overdue. We learned a lot about the challenges clinics faced in implementing the program. We shared our learnings with hundreds of additional community clinics in Washington, Oregon, California to help them anticipate and overcome these challenges.”— Dr. Gloria Coronado
TiME was conducted to determine whether treatment with hemodialysis sessions that are longer than many patients in the United States currently receive reduces the high rate of mortality among people being treated with thrice-weekly maintenance hemodialysis. The trial was conducted in 2 large US dialysis provider organizations, DaVita, Inc. and Fresenius Medical Care – North America, and included 266 outpatient dialysis facilities with 7035 patients.
“TiME established a model for conducting real-world research for a group of patients for whom there is very little clinical trial data. Many of the approaches and lessons from TiME are now being applied to a new set of pragmatic trials in dialysis that are being conducted in the US and internationally. In my view, TiME’s greatest contribution was to create a foundation for ongoing efficient and rigorous evidence generation in dialysis.”
Data and resources from the NIH Collaboratory Trials are posted on the NIH Collaboratory’s Data and Resource Sharing page in the coming months. As part of the program’s commitment to sharing, all NIH Collaboratory Trials are expected to share data and resources, such as protocols, consent documents, public use datasets, computable phenotypes, and analytic code.
Congratulations to Drs. Vincent Mor, Susan Mitchell, and Angelo Volandes and their team for the publication of their article in JAMA Internal Medicine reporting the primary results of the Pragmatic Trial of Video Education in Nursing Homes (PROVEN), an NIH Collaboratory Trial.
Dr. Vincent Mor, Co-PI of PROVEN
Dr. Susan Mitchell, Co-PI of PROVEN
Dr. Angelo Volandes, Co-PI of PROVEN
PROVEN was the first large-scale embedded pragmatic clinical trial conducted in nursing homes. The intervention was an advance care planning video, meant as an adjunct to first-person discussions with the clinical care provider to help people understand their options for end-of-life care, including life prolongation, limited care, and comfort care. PROVEN’s primary outcome was the number of transfers to the hospital from the nursing home over 12 months among long-stay residents with advanced illness. Secondary outcomes included hospital transfers, burdensome treatments, and hospice use among residents with or without advanced illness.
Although the video program was effective in previous small randomized trials, it was not effective in the PROVEN trial in reducing either the primary or secondary outcomes. The authors suggest that implementation error may explain the findings: intervention fidelity was low, only 1 in 5 targeted residents were shown the video, and implementation of the intervention was highly variable across facilities. The authors state that the results are “sobering,” that creative approaches are needed to change care in nursing homes, and that the perspectives of key stakeholders should be considered.
“For pragmatic trialists and implementation scientists focused on the nursing home setting, the highest level of health care system readiness and endorsement from senior and local leadership must be present before embarking on [pragmatic clinical trials]; otherwise, low implementation fidelity may compromise interpretation of its findings.” — Mitchell et al. 2020 JAMA Internal Med
Drs. Mitchell and Mor are also co–principal investigators of the National Institute on Aging (NIA) IMPACT Collaboratory (Imbedded Pragmatic Alzheimer’s Disease [AD] and AD-Related Dementias [AD/ADRD] Clinical Trials), which is similar to the NIH Collaboratory in that it aims to build the nation’s capacity to conduct pragmatic clinical trials embedded within healthcare systems. The population of interest for the IMPACT Collaboratory is people living with dementia and their caregivers.
Data and resources from PROVEN will be posted on the NIH Collaboratory’s Data and Resource Sharing page in the coming months. As part of the program’s commitment to sharing, all NIH Collaboratory Trials are expected to share data and resources, such as protocols, consent documents, public use datasets, computable phenotypes, and analytic code.
Focus on the research question, because that will drive the design, and the design will drive the analysis.
Select design features with analysis in mind, and collaborate early with a statistician. Weigh statistical choices against the challenges of implementation.
If possible, choose individual randomization. However, sometimes there is a strong rationale for choosing cluster/group randomization. Clustering must be accounted for in both design and analysis for CRTs and individually randomized group treatment (IRGT) trials.
The intraclass correlation coefficient (ICC) is a common measure of outcome clustering. Estimating the ICC is needed for study planning and power.
Increasing the number of clusters has more impact on power than increasing the number of patients per cluster.
Discussion Themes
With the move to virtual healthcare, the boundaries between clinic-based clusters have become more fluid. What approaches should trials use to describe contamination and estimate the impact of contamination on outcomes?
Read more about ICC in a Living Textbook resource and visit the Training Resources page for practical help on how to plan and conduct ePCTs.
Learn more in the Living Textbook about considerations for trial design and analysis for ePCTs.
The Methods: Mind the Gap series explores research design, measurement, intervention, data analysis, and other methods of interest in prevention science. The July 14 session will address the stepped-wedge cluster randomized design, which has received increasing attention in pragmatic clinical trials (PCTs) and implementation science research. Since the design’s introduction, a variety of mixed-effects model extensions have been proposed for the design and analysis of PCTs. Dr. Li will provide a general model representation and discuss model extensions as alternative ways to characterize secular trends, intervention effects, and sources of heterogeneity. He will also review key model ingredients and clarify their implications for the design and analysis of stepped-wedge cluster randomized trials.
Register in advance to join the online presentation. Registration is required.
Join us Friday, June 19, for “Demystifying Biostatistical Concepts for Embedded Pragmatic Clinical Trials,” the fourth session in our special 5-part Grand Rounds series focused on the Living Textbook. NIH Collaboratory investigators Drs. Liz Turner, Patrick Heagerty, and David Murray will discuss statistical design considerations, choosing the right design, and implications for the analysis. Topics covered will include:
RCTs, CRTs, and IRGTs: selecting the right trial design
Clustering and statistical power
Other analytical issues
See below for the full schedule of Living Textbook sessions and a special message from Dr. Kevin Weinfurt.
In a study supported by the NIH Collaboratory, researchers developed and validated a new sample size formula for detecting heterogeneity of treatment effect in cluster randomized trials. The work was published this month in Statistics in Medicine.
