Delineating the roles of all stakeholders to determine training needs

Data Analysis

Delineating the roles of all stakeholders to determine training needs


Charles Cannon, MD, PHD,

Carol S. Channing MVP, MD, DDS

Delineating the roles of all stakeholders to determine training needs

Consider the roles of different stakeholders at each trial site and the training needed for frontline staff and operational personnel. Depending on the complexity of the intervention, this may involve clinicians (physicians, specialists, nurses), senior management, business operations personnel, IT staff, researchers, clinic champions, and practice facilitators. Consider how existing procedures in each setting will need to be adapted. The intervention will generate new care delivery processes and workflows. Training for embedded pragmatic trials is optimally conducted within routine care and operations using the healthcare system’s existing training structure. Also, system change over time may need to be accommodated, for example, to orient new leaders joining the system or to account for changes in electronic tools for clinical decision support.

The following checklist presents considerations for designing the training. For more guidance and examples, read Training Frontline Staff and Clinicians.

*With [LW1] respect to hiring, consider providing funds through the study to pay the personnel who are directly responsible for study procedures, which both prioritizes the study procedures and gives more control to investigators. For example, an approach taken by one of the demonstration projects was to provide funds to hire personnel to do the research but not to provide care. In instances in which additional work was anticipated because of the intervention, the healthcare system used study funds to directly hire the personnel needed, or to cover the effort of existing personnel if involved in the study.

Pilot-testing for feasibility

This phase involves evaluating the context, capabilities, and challenges of the partner healthcare system and testing key elements of the intervention and collection and transfer of data from the EHR. Feasibility assessment specific to embedded PCTs may be associated with:

·      Biostatistical challenges: Study teams should work early on with their statistician to anticipate gaps or issues around cluster randomization, including sample size and potential for contamination, intraclass correlation, varying cluster size, the need for stratification or matching, and potential for missing follow-up data.

The Collaboratory’s Biostatistics and Study Design Core provides resources to help address challenges related to:

o   Statistical lessons learned

o   Constrained randomization

o   Small-sample robust variance correction

o   Frailty models

o   Unequal cluster sizes

o   Pair-matching versus stratification

o   Intraclass correlation coefficient

o   Extracting usable data from EHRs

·      Secondary use of EHR data: Using EHR data for research is fundamentally different from using prospectively collected data. Several aspects of EHR data drive these differences, including the lack of control over data definitions and data collection processes in healthcare facilities, procedures for access to the data, frequent dependence on record linkage, the need for computable definitions for cohorts and outcomes of interest, and the intricacies of demonstrating that data are of adequate quality to support research conclusions.

Consider how your intervention will use existing EHR data for cohort identification, recruitment, sample size estimates, population screening, collection of embedded patient-reported outcome (PRO) data, and so on. If there will be different EHR systems, how will they be linked? Pilot-test your data collection procedures and any web-based tools developed specifically for the study.

The Collaboratory’s Phenotypes, Data Standards, and Data Quality Core and Electronic Health Records Core offer PCT-specific guidance for study teams on:

o   Acquiring and using EHR data

o   EHR-based phenotyping

o   NIH Collaboratory Distributed Research Network

o   EHR data FAQs

·      Capabilities and readiness[TL2]  [JJ3] of the partner healthcare system: Consider whether there is a gap in existing services for the target population, or whether the system has had difficulties successfully addressing patient needs. Evaluate how effective the system or clinic will be as a research partner. One[MV4]  way is for the study team to pilot the programming capacity of a site (giving it sample programming code) to determine if the site is able to fully participate. Researchers could devise a set of criteria for site inclusion. For example, in a trial in which epidemiological benchmarks are inserted into radiology imaging reports, the study team wanted to verify at each site that the intervention text could be successfully included in the report based on a specific CPT code, modality, patient age, and date.

·      Integrate the study into the clinical workflow: Test how smoothly the embedded intervention will be incorporated into the site’s existing system infrastructure. Sites have different workflows and available resources, which can complicate the process. In pragmatic research, it is important that study procedures mimic routine practice. Consider how individual healthcare providers will be engaged in the intervention and how burdensome it may be for them. Make an effort to simplify and streamline the study to make integration as user-friendly as possible. Be open to adapting the study based on stakeholder input or changes in the care delivery system. Study teams are also encouraged to pilot the effectiveness of trial-related materials such as informed consent forms and processes, training videos, brochures and placards, and toolkits.


Conduct Resources

These articles by members of the Collaboratory’s Health Care Systems Interactions Core contain tips and guidance for pragmatic trial study teams:

·       A guide to research partnerships for pragmatic clinical trials (KE Johnson et al. BMJ 2014).

Trials without tribulations: minimizing the burden of pragmatic research on healthcare systems (EB Larson et al. Healthcare 2015).


[LW1]This footnote combines comments from Jerry and Miguel.
[TL2]Might mention assessing the best timing in case there are EHR changes or other potential disruptions (e.g, acquisitions)
[JJ3]May also want to add that it’s important to anticipate changes, such as switching EMR platforms.
[MV4]May want to mention that it is key to assess how the interventions in the trial may align with goals of the HCS to achieve practice metrics.


Delineating the roles of all stakeholders to determine training needs: Delineating the roles of all stakeholders to determine training needs. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials. Bethesda, MD: NIH Health Care Systems Research Collaboratory. Available at: Updated March 8, 2017.