How can we test if large language models really understand Indian languages in context? OpenAI’s latest release has been released. IndQA, A benchmark for evaluating how well AI-models understand and rationalize questions relevant to Indian languages, across all cultural domains.
Why IndQA?
OpenAI reports that around 80 percent of the world’s population does not speak English. Most benchmarks for measuring non-English capabilities remain narrow, and rely heavily on multiple choice or translation formats.
At the top, benchmarks like MMMLU or MGSM have reached saturation. Strong models are clustered together with similar scores. The benchmarks are not able to test models’ ability to understand the local history, context and daily life.
OpenAI uses India to create new benchmarks based on region. India’s largest market is ChatGPT, which has a total of 22 official languages spoken by over 50 million Indians.
The Dataset of Languages and Domains
IndQA measures the level of knowledge about Indian culture, everyday life and Indian languages. The benchmark is 2,278 multiple-choice questions, spanning 12 languages and ten cultural domains. 261 experts in the respective domains from India created this benchmark.
Cultural domains include Architecture and Design and Arts and Culture. Other cultural areas are Everyday Life and Cuisine, History and Law, Media and Entertainment and Religion and Spirituality. Articles are available in Bengali as well as English, Hindi and Hinglish. It is important to include Hinglish in order to show the code-switching that occurs during Indian conversation.
Each datapoint has four elements: a culture-based prompt in a language Indian, an English version for auditability and rubrics for scoring, as well as an ideal response that encodes expectations of experts.
The Rubric Based Assessment Pipeline
IndQA does not use exact match accuracy but instead uses rubric-based scoring. Domain experts assign weights to the criteria they define for each question.
The model-based grader compares the response of the candidate to these criteria, and then marks those that are met. The final grade is calculated by dividing the possible total score by the weighted sum for all criteria that were met. It is similar to grading an exam question, as it allows for partial credit, and takes into account nuance, cultural correctness and not just surface token overlap.
Construction process and adversarial filtering
OpenAI describes four steps in the construction pipeline.
In the first step, they worked with organisations in India to find experts from 10 distinct domains. They are experts in their respective fields and native speakers of both the target language as well as English. These experts wrote complex, reasoning-heavy prompts that were anchored to regional contexts, like literature, food, history, law, or media.
They applied an adversarial filtering. Each draft question was evaluated using OpenAI’s most powerful models, GPT-4o at the time of creation, OpenAI O3, GPT-4.5, and partially, after launch, GPT-5. We kept only those questions for which a majority failed to provide acceptable answers. In this way, future improvements to the models will be visible on IndQA.
The experts also provided detailed criteria, which are very similar to the rubrics used in exams, for each of the questions. They are used whenever another model on IndQA is evaluated.
Fourth, the experts used ideal responses and English translations. They then underwent peer review and revisions iteratively until they approved of quality.
Measurement of Progress in Indian Languages
OpenAI relies on IndQA in order to assess the performance of recent models for Indian languages and track progress made over the past two years. IndQA’s model performance is reported to have improved considerably, yet there are still many areas for improvement. Results include comparisons with GPT-5 thinking high and are stratified according to language, domain and system.
The Key Takeaways
- IndQA has a strong cultural foundation in IndiaIndQA does not only test translation accuracy or the ability to answer multiple-choice questions. It also evaluates whether AI models can understand and rationalize about important Indian language questions, in culturally relevant domains.
- This dataset has been expertly designed and is reasonably largeThis benchmark contains 228 questions in 12 languages, and 10 domains of culture, which were developed with the help of 261 experts across India. The topics include architecture, daily life, food and history.
- It is not necessary to match exact.The model is based on a system of weighing expert-defined criteria and evaluating model outputs. This allows for partial credit as well as nuanced evaluations.
- OpenAI’s most powerful models are filtered in a negative wayDraft questions have been filtered through GPT 4o (OpenAI o3), GPT 4.5 (and partially GPT 5), and kept only the items for which most models failed.
IndQA was a good move because it addresses a real problem. Most existing benchmarks for multilingual languages overindex on English-language content, and translate style tasks. India is home to many different high-resource and low-resource languages. IndQA provides expert curated evaluations based on rubrics for Indian cultural questions. In addition, it uses adversarial to filter against GPT4o and OpenAI o3, GPT4.5 and GPT5 in order to maintain headroom for future frontier models. IndQA is now a useful tool for testing Indian language reasoning within modern AI systems.


