Google Research has proposed a way of building software that is accessible with Natively Adaptive Interfaces. This agentic framework uses a multimodal agent to become the primary interface. It adapts applications in real-time to suit the abilities and circumstances of each individual user.
NAI integrates accessibility directly into the architecture, rather than shipping a pre-defined UI. The agent explains, observes and modifies the interface. It moves from standard design to context-based decisions.
Natively Adaptive Interfaces (NAIs) and the Stack?
NAI is built on a very simple principle: accessibility can be managed by the multimodal agent that mediates an interface, instead of static menus and options.
Some of the key features include:
- It is important to note that the word “you” means “person”. multimodal AI agent The primary user interface surface. It can read text and images and listen to voice, as well as output speech or text.
- The agent has been designed with accessibility in mind It should be built in from the beginning and not added later. Agents are responsible for adapting the navigation style, content density and presentation to every user.
- This process of design is explicit user-centeredPeople with disabilities should be treated not just as a group of users, but as the ones who set standards for all.
The framework targets what Google team calls the ‘accessibility gap’– the lag between adding new product features and making them usable for people with disabilities. The goal of embedding agents in the interface is to close this gap. This will allow the system to adapt faster without the need for additional custom software.
Agent Architecture: Orchestrator & Specialized Tools
The UI under NAI is backed up by a system of multiple agents. This is the core design:
- The following are some of the ways to get in touch with each other Organist Agent maintains shared context regarding the app, user and task.
- Specialized sub-agents Use focused features, like summarization and settings adaptation.
- Set of Configuration patterns The document describes how to adjust the settings and fix queries that are not correct.
In NAI’s case studies on accessible video for example Google team highlights core agent abilities such as:
- What is the user’s intent?
- Manage context and refine queries across all turns.
- Consistently use tool and engineer prompts.
The static navigation trees are replaced by dynamic ones. dynamic, agent-driven modules. The ‘navigation model’ is effectively a policy over which sub-agent to run, with what context, and how to render its result back into the UI.
Multimodal Gemini and RAG Video and Environments
NAI was built explicitly on models such as Gemini or Gemma, which can handle voice, text and images within a single context.
Google describes two stages in the pipeline for accessible video:
- Indexing offline
- This system creates dense semantic and visual descriptors on the timeline of video.
- This index is indexed by content, time and key.
- Online retrieval and augmented generation (RAG).
- When a user is playing back a video, and asks questions such as “What is the character wearing right now?”The system returns relevant descriptors.
- The multimodal model uses these descriptors and the question in order to produce a short, concise answer.
The design allows for interactive questions during playback and not only pre-recorded tracks of audio descriptions. This pattern can be applied to scenarios involving physical navigation, where an agent must reason through a series of user questions and observations.
Concrete NAI Prototypes
Google NAI Research is based upon several piloted or deployed prototypes developed with partner organisations such as RIT/NTID The Arc of the United States RNID Team Gleason.
StreetReader AI
- Built to help blind or low-vision people navigate urban environments.
- The combination of two or more words. AI Description This software is used to process camera data and geospatial information. AI Chat interface for natural language queries.
- Maintains a temporal model of the environment, which allows queries like ‘Where was that bus stop?’ and replies such as ‘It is behind you, about 12 meters away.’
Multimodal Agent Video Player (MAVP)
- Video accessibility is the focus of this project.
- The Gemini RAG pipeline is used to deliver adaptive audio descriptions.
- The user can adjust the descriptive density and ask questions to interrupt the playback.
Grammar Laboratory
- The RIT/NTID bilingual learning platform (English and American Sign Language) was created with the support of Google.org.
- Uses Gemini to generate individualized multiple-choice questions.
- When presenting content, you can use ASL videos, English subtitles, spoken narration and transcripts. You can also adapt the difficulty and modality to suit each individual learner.
The design process and the curb-cut effect
The NAI document describes a structure process: Investigate, Build and Refine, then Iterate Based on Feedback. One case study, which focuses on accessibility for video content:
- Defined users on a continuum from blindness to sight.
- Co-designing and testing sessions were conducted with 20 people.
- The product has been iterated over 40 times, based on feedback from 45 sessions.
It is expected that the interfaces will produce a curb-cut effect. Features built for users with disabilities – such as better navigation, voice interactions, and adaptive summarization – often improve usability for a much wider population, including non-disabled users who face time pressure, cognitive load, or environmental constraints.
What you need to know
- Agent is the interface, not an Add-onNatively adaptive interfaces (NAIs) use a multimodal AI as their primary layer of interaction. This means that accessibility is managed by the AI directly within the core user interface and not through an overlay.
- Architectural architecture of Orchestrator plus subagentsNAI relies on an Orchestrator to maintain a common context. It then routes the work (such as summarization and settings adaptation) to specific sub-agents.
- Multimodal Gemini + RAG for adaptive experiences: Prototypes such as the Multimodal Agent Video Player build dense visual indexes and use retrieval-augmented generation with Gemini to support interactive, grounded Q&A during video playback and other rich media scenarios.
- Real systems: StreetReaderAI, MAVP, Grammar LaboratoryNAI has been implemented in concrete tools, including StreetReaderAI, MAVP, for video accessibility and Grammar Laboratory, for ASL/English language learning. All of these are powered by agents that can work across multiple modes.
- Designing for accessibility as a primary constraintThis framework encodes accessibility into configuration patterns, such as detecting intent, adding context and adjusting settings, and it leverages curb-cutting, which improves robustness, usability and reliability for a broader audience by solving problems for users with disabilities.
Take a look at the Technical details here. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.

