YesWeHack tests AI agents on cybersecurity vulnerabilities

The French cybersecurity platform YesWeHack conducted a blind test to evaluate the capabilities of three AI agents—Codex, Claude Code, and Gemini CLI—in identifying and exploiting security vulnerabilities.

Using PortSwigger hacking labs and the Burp Suite tool, the autonomous command-line agents were tasked with resolving cross-site scripting (XSS) vulnerabilities without prior knowledge of the flaws. The results indicated that while these tools can accelerate cybersecurity workflows, they cannot yet replace human judgment.

Codex, powered by GPT-4o, demonstrated the highest level of autonomy, successfully exploiting an XSS flaw in under four minutes and bypassing a JWT authentication system. In contrast, Claude Code identified vulnerabilities but failed to complete the exploitation, while Gemini CLI declined to engage with the target.