API
< Back to all clusters
[HEALTH] · United States · 4 sources

started · updated

Healthcare professionals warn of risks in AI-driven medication and claims workflows

Healthcare organizations are rapidly adopting artificial intelligence to manage medication workflows and insurance claims, but experts warn that current oversight models may be insufficient. Research from Wolters Kluwer Health indicates that while 99% of healthcare professionals report their organizations are moving toward AI-enabled medication workflows, consumer trust remains low. Only 27% of surveyed consumers strongly trust automated systems to identify drug interactions without professional involvement.

In the insurance sector, where U.S. insurers process over 5 billion claims annually, AI systems generate thousands of daily recommendations regarding claim denials and eligibility. Experts caution that the “human in the loop” model often functions as symbolic reassurance rather than substantive protection. At high volumes, human reviewers may act as an “autopen,” approving batches of recommendations without sufficient time or context to independently verify the decisions.

Pharmacists have also highlighted risks such as automation bias, where users check a tool's work less frequently as they become accustomed to it. Experts emphasize that “bad data in means bad data out” and advocate for a “human-in-command” model. This approach requires mapping potential failure points, detecting uncertainty, and ensuring that professionals can explain and stand behind AI-generated outcomes to maintain clinical and operational accountability.

Entities

Academy of Managed Care Pharmacy · American Medical Association · CureIS Healthcare · Kaiser Family Foundation · The Lancet · Wolters Kluwer Health · World Economic Forum