Postmarket surveillance

Grand Rounds December 6, 2024: Opportunities and Challenges in the Use of Large Language Models for Post-Marketing Surveillance of Medical Products (Michael E. Matheny, MD, MS, MPH)

Posted on by Sophia Ramirez

Speaker

Michael E. Matheny, MD, MS, MPH
Director, Center for Improving the Public’s Health Through Informatics
Professor of Biomedical Informatics, Biostatistics, and Medicine
Vanderbilt University Medical Center
Staff Scientist, Geriatrics Research Education and Clinical Care Service
Associate Director, VA ORD VINCI
Tennessee Valley Healthcare System VA

Keywords

Artificial Intelligence; Large Language Models; Surveillance; Medical Products

Key Points

  • Increasingly, leaders in many disciplines are finding new applications for Artificial Intelligence (AI). Within healthcare, this technology is being used to support clinical decision-making; imaging processing; drug discovery; clinical trials; and as Ambient and Autonomous AI.
  • Large Language Models (LLMs) are a subset of generative AI. Since 2012, LLMs have emerged as a promising new technology with rapid growth, evolution of capacity and reach, and many potential applications in healthcare and clinical research.
  • There is significant interest in using LLMs to assist with patient trial matching, clinical trial planning, and the development of trial protocols and consent documents.
  • Another key area that LLMs could provide support in is medical product safety surveillance, with potential applications in adverse event detection, probabilistic phenotyping, and information synthesis.
  • The post-marketing surveillance space utilizes an ecosystem of healthcare data, imaging, radiology reports, insurance, structured data, medical literature, and social media. These sources could be integrated to conduct LLM reasoning and extractions.
  • Key challenges in safe and effective use of LLMs for this purpose include the lack of evaluation for medical product surveillance, the complexities of prompt engineering, hallucination risk (i.e., false positives), and the fact that evolving models over time challenge stable performance estimates.

Discussion Themes

 A segment of the clinical workforce could be trained to be “super users,” partnering with development teams in order to make sure that these tools are working appropriately in a clinical environment.

There is substantial interest in using LLMs to support clinical decision-making. However, studies have shown that the quality of the AI output can influence the performance of the clinicians. Especially in high-risk clinical environments, any drift in those algorithms could result in adverse clinical outcomes. The life cycle approach to conceptualization, development, implementation, surveillance, and maintenance will be necessary to achieve and maintain performance.

Task Force Releases Recommendations for National Medical Device Evaluation System

Posted on by Gina Uhlenbrauck

A new report (PDF) containing recommendations for the creation of a national registry system for evaluating and monitoring medical devices has been released for public comment today. The report, a joint project of the Medical Device Registry Task Force and cover_19aug2015 the Medical Device Epidemiology Network (MDEpiNet), is available on boh the US Food and Drug Administration (FDA) website and on  the MDEpiNet website.

The report reflects the results of a year-long effort, prompted by the FDA’s Center for Devices and Radiological Health (CDER), that  is focused on fostering a national system for monitoring the use of medical devices in the “real-world” setting of patient care, once the devices have been approved for the market (known as “postmarket surveillance”).

The term “medical devices” encompasses a wide range of technologies, including implantable pacemakers, cardiovascular stents, robotic surgical devices, and artificial joint replacements, among many others. At present, information about the use of these devices in routine care settings, including safety issues reported by doctors and patients, is collected in a variety of registries and health record systems. A  networked national system, such as the one described in the task force report, would be able to unite and build upon both existing and novel data resources, thereby improving safety monitoring and accelerating the development of new devices:

“Task Force recommendations for [Coordinated Registry Network] CRN architecture, and thus for the National System, center on leveraging existing, self sustaining electronic resources, such as device registries, electronic health records, administrative data and even social media and personal mobile device sources.”

The Task Force Report offers recommendation in several key areas, including:

  • Establishing a national dialog about medical device evaluation that includes all stakeholders;
  • Leveraging existing efforts in the arena of device registries and electronic data systems;
  • Describing the desired characteristics of a national Coordinated Registry Network (CRN) for medical devices;
  • Outlining priorities for developing and refining medical devices in multiple therapeutic areas;
  • Identifying and improving methods for analyzing data on medical devices; and
  • Addressing network governance and issues related to patient privacy and informed consent.

Each of these key areas also features suggested pilot projects designed to inform ongoing efforts.

A related perspective article summarizing the National Registry System project has also been published online in the Journal of the American Medical Association.

Related Links