Microsoft releases ThinkingBox to test AI agent reliability

Microsoft has released ThinkingBox, an open-source sandbox framework designed to evaluate the reliability of AI agents. Unlike traditional evaluation methods that rely on transcript analysis, ThinkingBox verifies performance by checking actual changes made to back-end database records. This approach aims to address the ‘discovery-reliability gap,’ where agents may appear to complete tasks successfully in transcripts but fail to produce the correct data outcomes.

Initial testing using the ThinkingBox-Bench benchmark across 12 models revealed significant reliability issues. While the top-performing model achieved a 65.36% success rate on its first attempt, its ability to succeed across 20 repeated trials of the same task dropped to 25.25%.

Separately, experts warn that the use of sandboxed architectures in desktop AI chatbots does not inherently guarantee data privacy. While sandboxing provides isolated execution environments to prevent code from accessing local devices or networks, it does not prevent service providers from collecting, reviewing, or using prompts, files, and metadata for model training. Privacy remains dependent on specific provider policies, product tiers, and contractual terms rather than the sandbox itself.