Specific Labs introduces Real-SWE benchmark for enterprise AI coding

Specific Labs has introduced Real-SWE, a new benchmark designed to evaluate frontier AI models on private, real-world enterprise codebases. Unlike traditional benchmarks that rely on public data, Real-SWE utilizes licensed, proprietary production codebases to test how well AI agents can navigate complex, company-specific systems and solve practical business problems such as billing, taxation, and data migration.

The initial results indicate that automating complex software engineering tasks remains a significant challenge for current models. Fable 5.1, paired with Claude Code, achieved the highest resolution rate of 38.8%. Other top-performing models included GPT-6 Astra using Codex CLI at 33.8%, and Gemini 3.8 Flash using Gemini CLI at 31.2%.

The benchmark methodology involves testing agents within isolated sandboxes using tasks that require navigating multiple files and extensive instructions. The evaluation is based on the pass@1 metric, averaged over eight independent runs per task to ensure statistical reliability.