AI governance faces risks from intelligence explosions and inaccurate outputs

Experts are raising concerns regarding the risks of an ‘intelligence explosion’ and the limitations of current AI governance frameworks. As AI systems begin to assist in the development and evaluation of their own successors, there is a risk that flawed assumptions and biases could be amplified through successive generations, making future systems increasingly difficult to govern.

There is a critical distinction between governance and accuracy. A system can be strictly governed—with full audit trails and access controls—yet still produce incorrect or ‘hallucinated’ outputs. Because AI models are designed to produce plausible and fluent text, they can present wrong information with a level of confidence that makes errors difficult for human users to detect. This is particularly dangerous in professional sectors like conveyancing, where a well-governed but incorrect answer carries a false sense of credibility. Relying on AI output as a defense for errors is not considered a suitable position by regulatory bodies like the SRA.