Google officially releases Android BenchThe new framework and leaderboard is designed to evaluate how Large Language Models perform on Android Development Tasks. Open-source datasets, methodologies, and test harnesses are now publicly available. GitHub.
Benchmark Methodology & Task Design
The nuances, platform-specific requirements and limitations of mobile application development are often missed by general coding benchmarks. Android Bench solves this problem by curating a set of tasks sourced from public GitHub Android repositories.
There are different levels of difficulty for the scenarios.
- Fixing broken changes in Android versions
- Wear OS Devices can perform domain-specific tasks such as network networking.
- Migration of code to the latest versions Jetpack Compose Android’s Modern Toolkit for Building Native User Interfaces
The framework asks an LLM for a fix to a reported problem and verifies it. fix using standard developer testing practices:
- Tests on Units The Android framework is not required to run tests that check small, isolated code blocks (such as a single class or function).
- Tests of instrumentation Testing that runs on an Android emulator or a real Android device to see how code interacts the Android APIs and system.
Mitigating Data Contamination
The challenge of evaluating benchmarks for developers is a significant one. data contamination. It happens when a LLM model is presented with evaluation tasks as part of its training, and the resultant results are that the model will memorize the answers instead of demonstrating real reasoning or problem solving abilities.
Google has implemented several measures to ensure that the Android Bench results are accurate.
- The manual review of agents’ trajectory: It is important that developers check the steps taken by the model to solve the problem.
- Canary String Integration: In the benchmark dataset, a string of unique text that can be identified is included. It is a way to tell web crawlers, data scrapers and AI companies that they should exclude the data in future training.
First Android Bench Results
The benchmark is designed to measure the performance of a base model only, and not include complex agentic workflows, or the use of tools.
It is important to note that the word “you” means “you”. Score This represents the percentage of each model’s 100 tests that are successfully solved over 10 independent runs. LLM outputs may vary from run to run, so the results also include an overview of each LLM model. Confidence interval (CI) To calculate a p-value
Models successfully complete between 16 and 72 percent of tasks in this initial release.
| Model | Score (%) | Range of CI (%) | Date |
| Gemini 3.1 Pro Preview | 72.4 | 65.3 — 79.8 | 2026-03-04 |
| Claude Opus 4.6 | 66.6 | 58.9 — 73.9 | 2026-03-04 |
| GPT-5.2-Codex | 62.5 | 54.7 — 70.3 | 2026-03-04 |
| Claude Opus 4.5 | 61.9 | 53.9 — 69.6 | 2026-03-04 |
| Gemini 3 Pro Preview | 60.4 | 52.6 — 67.8 | 2026-03-04 |
| Claude Sonnet 4.6 | 58.4 | 51.1 — 66.6 | 2026-03-04 |
| Claude Sonnet 4.5 | 54.2 | 45.5 — 62.4 | 2026-03-04 |
| Gemini 3 Flash Preview | 42.0 | 36.3 — 47.9 | 2026-03-04 |
| Gemini 2.5 Flash | 16.1 | 10.9 — 21.9 | 2026-03-04 |
Android Studio 3.0 and later versions of Android Studio include API keys which allow you to test all models.
What you need to know
- Specialized Focus Over General Benchmarks: Android Bench fixes the flaws of other generic benchmarks. This is done by measuring the ability of LLMs to handle Android’s APIs, dependencies, and unique complexity.
- Real-World Scenarios for Grounded Learning: In place of isolated algorithmic testing, the benchmark compares models to actual challenges pulled directly from public GitHub repositories. The tasks include resolving API breaking changes, migration of legacy UI codes to Jetpack composition, and device-specific network handling (e.g. on Wear OS).
- Verifiable, Model-Agnostic Testing: It is not the methodology that determines how code should be generated, but rather its functionality. The framework checks the LLM’s suggested fixes automatically using Android engineering standards: isolated unit testing and emulator-based instruments tests.
- Strict Anti-Contamination Measures: To ensure models are actually reasoning rather than regurgitating memorized training data, the benchmark employs manual reviews of agent reasoning paths and uses ‘canary strings’ to prevent AI web crawlers from ingesting the test dataset.
- Baseline Performance Established: This first version of the Leaderboard is based solely on model performance, without any external agents. Gemini 3.1 pro preview currently has a success rate of 72.4%. This highlights the wide variation in LLM capability (which ranges from 16.1% up to 72.4%).
Take a look at the Repo The following are some examples of how to get started: Technical details. Also, feel free to follow us on Twitter Don’t forget about our 120k+ ML SubReddit Subscribe Now our Newsletter. Wait! What? now you can join us on telegram as well.

