The design and implementation of CDS tools should not only include careful consideration of their content and purpose, but their method of monitoring for success in terms of outcomes, functionality, and how they fit in the context of other CDS tools that are currently in place (see Section 5 for further details).

In 2022, the Food and Drug Administration (FDA) released a Guidance for Industry and Food and Drug Administration staff about Clinical Decision Support Software (CDS) (FDA 2022). CDS is a broad term and encompasses a range of different functions, including “computerized alerts and reminders for providers and patients, clinical guidelines, condition-specific order sets, focused patient data reports and summaries, documentation templates, diagnostic support, and contextually relevant reference information (FDA 2022).” Some CDS software meet the regulatory definition of a medical device, and therefore, that CDS software is subject to regulatory oversight by FDA.

The FDA guidance outlines criteria for determining which CDS are considered non-devices based on criteria in the 21st Century Cures Act. The CDS software is a not a device if the following are all true:

1. it is not intended to acquire, process, or analyze a medical image, for example, from an computed tomography (CT), x-ray, ultrasound, or magnetic resonance imaging (MRI) OR a signal from an in vitro diagnostic device or a pattern or signal from a signal acquisition system from within, attached to, or external to a body.
2. it is intended for the purpose of displaying, analyzing, or printing medical information about a patient or other medical information (such as peer-reviewed clinical studies and clinical practice guidelines)
3. it is intended for the purpose of supporting or providing recommendations to a health care professional about prevention, diagnosis, or treatment of a disease or condition
4. it is intended for the purpose of enabling a health care professional to independently review the basis for any recommendations and not have them rely primarily on such recommendations when making diagnosis or treatment decisions about a patient.

If a software is the focus of FDA’s regulatory oversight, the next step is a determination of whether an investigational device exemption (IDE) is required. This determination is based on risk, and evaluating the risk of a software is based on its function, who it is intended for, and what the user is supposed to do with the information provided by the software. Note that a device being non-significant risk does not mean the research is minimal risk; those determinations are distinct.

The FDA encourages engagement early and often to understand how the regulations and policies apply to a particular product or technology. For specific questions, the presenters encouraged researchers to reach out to the FDA’s digital health experts at DigitalHealth@fda.hhs.gov.

In addition to the CDS guidance, there are other policies that may apply to a digital health technology, and the FDA created a Digital Health Policy Navigator to help people find the relevant guidance for their situation.

Resources Needed

To understand what resources are needed to build CDS implementations for PCTs, potential researchers must consider the EHR systems, workflows, policies, and personnel at each research site. The work required to implement CDS can vary by system depending upon what data they already collect and how their system is configured. Some sites may not have the technical staff required to integrate new CDS into the EHR or may have policies that prohibit the addition of new CDS (to prevent “alert fatigue” experienced by many clinicians). The cost of developing CDS can also be quite significant, as many organizations will charge fees for building CDS, and there are expenses for consultation, development, coding, quality assurance, data pulls, and numerous other services. When developing a PCT using CDS, it is important to budget for these expenses, investigating the potential feasibility and costs with the health system that you will be working with ahead of time. Also often underestimated, there may be a considerable waiting period prior to the completion of the build, depending on organizational priorities. This is in reference to not only the time of the individuals on the development team, but also time in a general sense; the development, testing, and implementation may take many months. For this reason, it is recommended that the timeline for the PCT is flexible and allows for adequate time for development and validation.

To reduce build time, a detailed explanation of the required data elements should be considered prior to the request. For example, if a specific diagnosis is required, it would be beneficial to identify the list of ICD-10 codes that would correctly identify the needed patient population. Pointing the builder to a value set or predefined phenotype would save time in communicating specific data needs. It also helps to put the capabilities of the EHR into perspective. For instance, creating a list of all the data requirements for the recruitment tool and mapping them to existing sources of data collection in the EHR will allow the research team to assess the feasibility and level of specificity that is possible by an EHR-based recruitment tool. From there, it may be possible to further assess for data quality and different workflow considerations to determine feasibility.

Most importantly, building, measuring, and maintaining a CDS tool requires a team. There will be a number of individuals external to the PCT team that will be involved depending on particular domains of expertise that are required. First, there must be technical and informatics experts available to build the tool into the EHR (or desired channel) using specifications provided by the research team. They may also give guidance to facilitate interoperability and determine if the requested data are available for the tool. A data manager will provide additional insight into data collection for multisite trials and facilitate trial evaluation (i.e. surveys, interview data, and heuristics). Another important individual to consult may be a clinical expert, or an informaticist well-versed in the appropriate workflow targeted by the PCT. Understanding workflow is essential, as integrating CDS tools with workflow has been found to be a predictor of their success (Khalifa 2014). In addition, there is a noted synergy between workflow and data experts; understanding when the data will be available contributes greatly to determining the feasibility of building a CDS tool. The required data may exist, but it might not be available when it is needed. Finally, involving key stakeholders in the development and implementation process is crucial, including end-users. Participatory design in CDS development has been found to be an effective method to assure adherence and functionality of CDS (Jeffery et al 2017). Other stakeholders will be dependent on the PCT, and may also involve leadership, financial services, and patients. These examples are not meant to be exhaustive, as there may be many other consultants who are necessary to engage. For instance, security experts may be needed when developing a CDS intervention using a mobile app. Always consider what resources you need prior to conducing a trial, and the information provided from these experts can save a lot of time and frustration when developing a complex CDS tool.

Specificity and Utility

Paramount to the functionality of CDS is the ability to target the intended recipient of the intervention. EHR data are complex and variable, and defining the patient in terms of these data can be difficult. A phenotype (the data that define the characteristics of a specified cohort) is important to define for EHR-based CDS. Identifying a specific phenotype allows for consistent targeting of a heterogeneous population, making a CDS functional and plausible to implement. However, there are challenges when selecting or creating a phenotype. In a 2013 study, Richesson et al. examined a number of validated phenotypes for diabetes mellitus, finding that these different definitions yielded meaningfully different results (Richesson et al 2013). When selecting a phenotype, especially if there are multiple choices for a similar population, it is important to select the definition that best coincides with the research being conducted, practice setting, and the purpose of the CDS tool. This may also mean selecting a more specific set of inclusion criteria, depending on the required specificity. (See the Electronic Health Records–Based Phenotyping chapter of the Living Textbook.)

To determine the level of specificity required for the CDS, workflow should be closely examined with an outlined use case being developed for this initial analysis. Foremost, a decision of the frequency of the alert should be made. Should it fire during a certain time in the workflow? Does it warrant multiple warnings to the same clinician? Is it critical enough to yield false positives, or would this result in alert fatigue? By asking questions similar to these, the criteria for the CDS tool may be developed or modified from the original phenotype.

Hypothetical Example: A Screening Tool for Lung Cancer Implemented in a Primary Care Setting

Using guidance from the American Cancer Society, an alert to suggest screening to the clinician is instituted if the patient is age 55 to 74 and is a current smoker. Prior testing of the alert is done before implementation, and 70% of the patients at this clinic would be recommended for screening, which is determined to not be specific enough for practical use. To target patients who are at highest risk, only those with a 30 pack-year smoking history will be included for the alert, dropping the population being captured by this alert to an acceptable number.

From this example, initial criteria are tested and determined to not be specific enough, so further criteria are added. This required change could also manifest as modifying a diagnosis list, redefining upper or lower limits of a laboratory value, or including additional criteria such as family history. There are many options, and these criteria are only limited to the available data. Also seen in the example is the use of pretesting prior to implementation. This is highly recommended, as simulating the CDS tool prior to implementation can save time and frustration. If the CDS targeting criteria are too specific, criteria must be removed or changed. Regardless, there should be an expectation of the specificity of the alert, whether it is derived from previous evidence or from the data of the health institution in which the tool will be implemented. If the specificity of the tool is not calibrated, it will not have the desired effect. This could result in the tool being retired or removed because it is not usable, or even detrimental effects if recommendations are given to a patient who should not have met criteria.