AI agents' autonomy raises governance and safety challenges across industries

As AI systems become more autonomous in manufacturing and other enterprise settings, leaders stress that human judgment remains essential. In factories, the shift from "human in the loop" to "human on the loop" creates a risk of passive acceptance, where operators trust AI outputs without critical evaluation. Effective oversight, the ability to intervene, and accountability for errors are highlighted as key to preventing hidden failures.

At the corporate level, the rise of agentic AI—software that can act independently—has exposed major governance gaps. Only a minority of firms have formal oversight structures, while "shadow AI" use by employees threatens data security and compliance. Organizations face new cost models, such as token‑based pricing, and must establish clear guardrails to manage the increased complexity and liability of autonomous agents.

Experts advise that autonomy should be a design decision, not an unchecked goal. Bounded autonomy—restricting tools, tasks, and giving humans final approval for high‑impact actions—helps maintain predictability and trust. The appropriate level of autonomy depends on the specific problem, balancing capability against risk and the organization’s confidence in the system.