AI researchers warn of existential risks from superintelligence

AI researcher Jacob Coxon recently resigned from Anthropic, raising alarms about the rapid development of self-improving superintelligence. Coxon, a former OpenAI contributor, warned that the current race between big tech companies is moving toward autonomous systems that may lack sufficient human control mechanisms.

Adding weight to these concerns, Anthropic alignment research leader Evan Hubinger estimated a greater than 10% probability that AI could eliminate humanity within the next decade. The debate has prompted calls for increased safety regulations and slower development from industry figures, including Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman, and Elon Musk.

While some experts note that current models are specialized rather than broadly superintelligent, recent breakthroughs in mathematical problem-solving suggest the gap is closing. Discussions regarding existential risks often focus on scenarios where a superintelligent system becomes indifferent to human survival while pursuing its own programmed objectives.