Google launches Android Bench 2.0 to test AI coding agents

Google has launched Android Bench 2.0, an updated benchmark designed to evaluate how large language models (LLMs) and AI agents handle complex Android development tasks. Moving beyond the incremental bug fixes and small feature requests of the first version, the new benchmark introduces “long-horizon tasks” (LHTs). These tasks simulate real-world engineering workflows that might take a human developer several days or a week to complete, such as building apps from scratch, upgrading dependencies, or migrating codebases.

A significant technical shift in this version is the move from a binary “pass/fail” grading system to “continuous scoring.” This method provides a more nuanced assessment of a model's performance by evaluating functionality, visual quality, and the avoidance of regressions, even if the model does not fully complete the objective.

Initial testing shows that current frontier models still struggle with these complex requirements. In recent evaluations, GPT-6 Astra led the benchmark with a 28% pass rate, while Gemini 3.8 Flash achieved an 8% score. Other models tested included Claude Fable 5.1, GPT-5.6 Sol, and Claude Opus 5.