TU Dresden study analyzes risks of Large Language Models in medicine

An interdisciplinary research team from TU Dresden, the Else Kröner Fresenius Center for Digital Health, and Dresden University Hospital has published a study in the journal ‘Nature’ analyzing the risks of using Large Language Models (LLMs) in clinical practice.

The researchers found that the integration of LLMs into healthcare is progressing faster than the implementation of official safety and protective measures. Many systems are currently used informally without institutional guidelines, raising concerns regarding patient safety, data privacy, and accountability. Because many models are not operated locally, questions regarding data control and security are prominent.

The study identifies risks throughout the AI lifecycle, including model design, training data, and clinical application. Specific threats mentioned include ‘data poisoning’ and ‘prompt injections,’ where hidden instructions can lead to dangerous errors, such as the failure to detect a tumor.

To mitigate these risks, the authors recommend establishing secure development processes, rigorous training data preparation, and continuous model monitoring. They also suggest creating specialized ‘Security Operations Centers (SOCs) for AI’ within healthcare institutions to coordinate and oversee the use of these technologies.