Jacob Coxon resigns from Anthropic, warning of AI existential risks

Jacob Coxon, a former pre-training researcher at both OpenAI and Anthropic, has resigned from Anthropic, sparking intense debate regarding the safety of artificial intelligence development. In a viral announcement, Coxon accused both companies of failing to act responsibly, stating they are “racing straight to self-improving superintelligence and gambling with our lives.”

Coxon, who contributed to the development of GPT-4o, warned that while immediate AI takeover risks may be low, the danger lies in future systems capable of self-improvement. His claims have drawn reactions from across the political and industrial spectrum. In the United Kingdom, Prime Minister Andy Burnham noted that AI poses “risks to national security,” while members of the U.S. Congress have called for legislative action.

The warnings have also met significant skepticism. Critics and industry figures, including Elon Musk, have suggested that such alarms could be part of a sophisticated public relations effort. Some argue that emphasizing existential risks serves to drive public demand for regulation, which could ultimately benefit established frontier labs by creating high barriers to entry for new competitors.