Using the interactive text messaging intervention from the LungSMART trial as a case study, researchers demonstrated the application of a proposed framework for assessing the fit of custom large language models (LLMs) in public health interventions.
The report was published online in the Journal of Medical Internet Research.
Integration of custom LLMs into health chatbots and other clinical and public health workflows is increasingly common. However, these novel tools can produce errors and unpredictable results and can be difficult to evaluate, raising ethical and practical challenges for teams seeking to use them.
Researchers from LungSMART, an NIH Collaboratory Trial, developed an “acceptance criteria framework” to guide assessments of an LLM’s performance before its deployment in real-world practice. They demonstrated the framework by applying it to an LLM being developed for the interactive chatbot component of the LungSMART intervention.
LungSMART, led by Dave Wetter, Guilherme Del Fiol, and Ken Kawamoto of the University of Utah, is testing a combination of text messages, an interactive chatbot, and educational videos delivered by text message to address barriers to completing lung cancer screening among patients receiving care in safety net community health centers.
LungSMART is supported within the NIH Pragmatic Trials Collaboratory by a grant from the National Cancer Institute. Learn more about LungSMART.