Microsoft and Hugging Face launch ThinkingBox AI benchmark

Microsoft and Hugging Face have introduced ThinkingBox, a new benchmark designed to evaluate AI agents based on the terminal backend state and side effects they leave behind in a database, rather than just the text they generate.

The benchmark tests agents across 507 stateful business workflows, with each task being run 20 times to measure consistency. In testing, Claude Opus 5.5 and Claude Opus 5 both successfully completed 241 of the 507 tasks across all 20 attempts. The estimated cost for a dependable task was $7.80 for Claude Opus 5.5 and $13.30 for Claude Opus 5.

ThinkingBox aims to identify gaps where an agent may perform well-formed tool calls and generate correct-sounding responses but fail to achieve the required end state in a database. For example, an agent might correctly follow a refund policy but fail to update a ticket status to ‘on hold’ when a carrier exception remains open. The benchmark is available through OpenEnv.