AI researchers warn of intelligence explosion risks

A group of 22 prominent AI researchers, including Geoffrey Hinton and Yoshua Bengio, have warned of a potential ‘intelligence explosion’ caused by the increasing automation of AI development. In a joint paper released through the University of Cambridge’s Centre for the Study of Existential Risk, experts noted that AI is increasingly being used to conduct its own research and development. For instance, at Anthropic, the proportion of code written by AI rose from single digits in early 2023 to over 80% by May 2024. Researchers warn that if AI achieves recursive self-improvement, it could lead to rapid technological acceleration that outpaces human ability to monitor or control it, potentially resulting in human marginalization or extinction.

To mitigate these risks, the researchers are calling for urgent policy interventions, such as mandatory reporting of R&D automation metrics and the placement of resident monitors within AI companies.

Parallel to these global concerns, there are significant disparities in national AI safety capabilities. While the UK, US, and Japan are aggressively expanding their technical workforce and computing resources to evaluate AI risks, South Korea’s AI Safety Institute currently operates with only six technical evaluators and limited GPU infrastructure. Experts suggest that strengthening safety evaluation capabilities and specialized talent training is essential for maintaining national competitiveness and managing the rapid pace of AI advancement.