Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
  • Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI
  • OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min
  • Copywriting: The Full Information | Copyblogger
  • AI Governance in 2027: Accountability Strikes Into the Workflow – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»Nested Learning is a New Machine Learning Approach that views models as nested optimization problems to enhance long context processing.

Nested Learning is a New Machine Learning Approach that views models as nested optimization problems to enhance long context processing.

Tech By Gavin Wallace08/11/20256 Mins Read
Facebook Twitter LinkedIn Email
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
Share
Facebook Twitter LinkedIn Email

How can AI systems learn new information without losing what they have learned or having to retrain from the beginning? Google Researchers introduced Nested Learning. This is a machine-learning approach which treats models as collections of small nested problems rather than a single model trained with a loop. It is a way to combat catastrophic forgetting, and to move models towards continuous learning. This will be closer to the biological brain’s ability to manage memory over time.

https://abehrouz.github.io/files/NL.pdf

Nested learning?

Google’s research report ‘Nested Learning, The Illusion of Deep Learning Architectures’ A complex neural network can be modelled as a collection of coherence optimization problems. They may run simultaneously or nested and are optimised together. Every internal problem is unique, with its own flow of context, inputs or gradients that it observes and update frequency.

Nested Learning forces an order based upon the frequency of updates. The parameters that are updated frequently tend to be at the innermost levels while those with slower updates form the outermost levels. In this hierarchy, each level is defined as a Neural learning module. Each of the levels compresses their own context into parameters. The team of researchers shows that the view includes standard back-propagation using an MLP as well as linear attention as well as common optimizers, which are all examples of associative memories.

Associative memory in this context is defined as any operator which maps values to keys and has an objective that is internal. The team formalizes and formalizes Associative Memory. They then show that Back-Propagation is a gradient descent one-step update which learns an input mapping to local surprises signals.

https://abehrouz.github.io/files/NL.pdf

Deep Optimizers and Associative Memory

Nested Learning proposes redesigning optimizers with richer objectives. Standard momentum is a linear memory of past gradients that can be trained using a similarity dot objective. The internal objective results in a Hebbian update rule which ignores data sample dependencies.

They replaced this similarity object with an L2 regressive loss over gradient features. The result was an update rule which better managed memory capacities and memorized gradient sequences. The researchers then convert the linear momentum map into an MLP, and create Deep Momentum Gradient Descent. This is where a neural memory produces the state of momentum and it can be passed through non-linear functions such as Newton Schulz. In this perspective, the Muon Optimizer is also recovered as a particular case.

https://abehrouz.github.io/files/NL.pdf

Continuum Memory System

In most sequence models attention serves as the working memory of the window in which it is displayed, while feedforwards blocks are used to store long term memories that rarely change after training. Researchers at Nested Learning extend this binary perspective to a Continuum Memory System (CMS).

CMS is defined as a chain of MLP blocks, MLP(f₁) through MLP(fₖ), where each block has its own update frequency and chunk size. These blocks are applied sequentially to an input sequence. The parameters of each block are updated only every C^(ℓ) steps, so each block compresses a different time scale of context into its parameters. Special case: A Transformer standard with only one feedforward blocks is used with k=1.

Instead of one static feedforward level, this construction creates a range of long term memories that are based on frequency. This research relates this directly to the multi-time scale synaptic consolidation and brain processes, in which different parts of a system can learn at varying rates but share a similar architecture.

Hope, an architecture built on titans that is self-modifying

The research team created HOPE to demonstrate that Nested Learning can be applied in recurrent architectures. Titans is an architecture for long-term memories where the neural memory module can be taught to remember unexpected events and help attention pay attention to tokens from long ago.

Titans only has 2 levels of parameter updating, resulting in first-order context learning. Titans can be extended by HOPE in two different ways. It can be self-modifying. Through a self-referential process, it optimizes its memory and in theory, supports unbounded in context learning. Second, the Continuum Memory System is integrated so that updates to memory occur multiple times and are scaled up to larger context windows.

https://abehrouz.github.io/files/NL.pdf

Understand the results

The research team assesses the HOPE model and baselines in language modeling, common sense reasoning, at 3 scales of parameters, 340M 760M and 1.3B. For reasoning, benchmarks include ARC Easy (ARC Challenge), Social IQa and BoolQ for accuracy, as well as Wiki perplexity and LMB for language modeling. Below is Table 1 which reports the results of HOPE and Transformer++.

https://abehrouz.github.io/files/NL.pdf

The Key Takeaways

  1. Nested Learning uses nested problems to optimize a model, with different updating frequencies. It targets forgetting catastrophically in continuous learning.
  2. The framework redefines backpropagation as a set of associative memory units that can compress the context flow.
  3. Nested Learning’s deep optimizers replace the simple comparison of dots with more complex objectives, such as L2 regression. They also use neural memory to create rules that are context-aware and expressive.
  4. The Continuum Memory System is a model of memory that uses a spectrum MLP blocks which update at different speeds, creating’short, medium and long-range memory’ rather than relying on merely one feedforward static layer.
  5. The HOPE Architecture, a self-modifying Titans variant built on Nested Learning Principles, has shown improved performance in language modeling, context reasoning and continuous learning compared with strong Transformer baselines and recurrent baselines.

Nested learning is an effective way to reframe deep networks into Neural Learning modules that combine architecture and optimization in one system. Deep Momentum Gradient Descent (DMGD), Continuum Memory System and HOPE provide a tangible path for achieving enriched associative memories and continual improvement in learning. The work makes continual education a design priority.


Take a look at the Paper You can also find out more about the following: Technical Details. Check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter. Wait! Are you using Telegram? now you can join us on telegram as well.


Asif Razzaq, CEO of Marktechpost Media Inc. is a visionary engineer and entrepreneur who is dedicated to harnessing Artificial Intelligence’s potential for the social good. Marktechpost was his most recent venture. This platform, which focuses on machine learning and deep-learning news, is technically solid and accessible to a broad audience. Over 2 million views per month are a testament to the platform’s popularity.

🙌 Follow MARKTECHPOST: Add us as a preferred source on Google.

ar learning machine learning models x
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Mannequin Skilled With Reinforcement Studying Solely

20/11/2026
Top News

Who are the real losers in the Musk v. Altman Trial?

The Existential Threats of Wikipedia Feel More Than Ever

DOJ Lawyers Argue xAI Is ‘Vital’ for National Security in NAACP Lawsuit

Hackathon Man vs. Machine: A Look Inside

It’s a race to build the DeepSeek in Europe

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

Can OpenAI’s ‘Master of Disaster’ Fix AI’s Reputation Crisis?

22/05/2026

I trained a fly’s brain to generate WIRED story ideas

16/09/2026
Latest News

Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains

22/11/2026

Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI

21/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.