NVIDIA AI Releases DeltaNet-2 Gated: A Linear attention layer that decouples the Erase and Write of Delta Rule.

The unbounded cache KV of softmax attention is replaced by a fixed size recurrent state. The decoding is done in constant memory and the sequence mixing can be reduced to linear time. What to forget is not the hard part. How to edit compressed memories without destroying existing associations is the hard part.

NVIDIA releases new NVIDIA GPUs Gated DeltaNet-2The model decouples the active memory edit into two channel-wise gates. Model decouples active memory editing in two gate-by-channel gates. This model was trained with 100B FineWeb, Edu tokens and 1.3B parameter values. It is superior to Mamba-2 Gated, DeltaNet, KDA and Mamba-3 in the benchmark suite.

The problem of the scalar gates in delta rule models

The matrix is stored in a recurrent linear focus layer SThe t The query will then read it. DeltaNet subtracts the value associated with the key to add an active edit. The scalar size is used. βt to control how much to overwrite. MamThe ba-2 adds a daThe mA-dependenThe t scalar decay αt Global forgetting. Gated DeltaNet merged both operations. However, both gates were scalar for each head.

Kimi Delta Attention KDA refines the decay. It is a replacement for the scaler αt with A channel-wise vecThe tor. KDA uses a single vector βt for the active edit. That scalar controls two different things at once. It decides how much old content to erase on the key side. It also decides how much new content to commit on the value side. These two decisions act on different axes of the state. Tying them together is a modeling restriction, not a property of the delta rule.

Two gates in place of one: Gated Delta Rule-2

Gated DeltaNet-2 seperates the two decisions using Gated Delta Rule-2. The Gated DeltaNet-2 introduces an erase gate that is channel-based. The bThe t ∈ [0,1]DK On the key axis. The write gate is channel-wise. WThe t ∈ [0,1]The dV The value axis. These gates are generated by the sigmoidal projection of token representation. Update the decay after active editing.

When written in compact form, this recurrence would be:

SThe t = (I − kThe t (bThe t ⊙The kThe t)⊤"The t St−1 +The l (wThe t ⊙VThe t)⊤

You can find out more about this by clicking here. The following are some of the ways to get in touch with someone elseThe t = Diag(αThe t) The decay of the channels is carried over to KDA. The right factor in the erase matrix remains KThe tIt is important to maintain the Delta-rule when writing. The correct factor will become The bThe t ⊙The kThe tThis is achieved by making the channel selection for reading. Write term KThe t zThe t⊤ You can use The zThe t =T ⊙VThe tSelecting the channel for value updates.

Both gates will collapse at the same scaler βt, the update recovers KDA exactly. When the decay αt also collapses to a scalar, it recovers Gated DeltaNet. Both prior models are preserved as tied suThe bspaces of The the new updAThe ’te.

Gated Delta Rule-2, in fast-weight mode, is one online gradient on the local regression loss. While the decayed state remains close to memory during residual editing, gated targets are used for both read and write.

Backwards training with gate awareness and chunkwise training

This recurrence has a form WY that is similar to the KDA structure. Each rank-one delete absorbs the cumulative channel-wise decay. The per-chunk updating is a result of the asymmetric matrix of form I − k̄R ēr⊤. ThThe e-mail address you entered is not valid. implementation uses chunk size C = 64 with fused TRiton kernels.

KDA’s scalar-shortcut is not applicable to the reverse pass. On the write side, a diagonal gate is different over each value channel. On the eraser side, a diagonal gate is used to cover key channels. Gate factors are therefore required to appear within the dots products which accumulate gradients. This gate-aware Vector-Jacobian is derived explicitly in the paper. To avoid Triton WGMMA assertions on Hopper GPUs the fused WY forward kernel is limited to only two and four warps.

Hybrid block design

In a Transformer-style standard block, Gated DeltaNet-2 serves as the token mixer. The Query- and Key-Paths use linear project, short causal Convolution, L2 Normalization, and SiLU. Value path is linear project, SiLU, and short convolution. The decay αt, erAse gaThe te The bThe tWrite gate WThe t The linear branches come from different directions. The recurrent out is RMS-normalized and multiplied using a SILU output gate.

In a hybrid version, Sliding-Window Attention is added after the mixer. Gated-DeltaNet-2 contains an MLP with SWA and another MLP. SWA is used to handle local interactions that are exact, while the recurrent mix compresses histories. This hybrid maintains linear scale with a limited attention cache.

Results for 1.3B Parameter

The models all have 1.3B parameters and are trained using 100B tokens of FineWeb-Edu. All models have the same parameter size and count. Recurrent states hold 262,144 floating elements per layer and per batch element. The training length for hybrid models is 2K SWA and 4K Tokens. Mamba-3 MIMO uses rank R = 4.

Gated DeltaNet-2 is the most averaged model in terms of language modeling as well as commonsense reasoning. This model has an average of 53.11 for LAMBADA as well as the reasoning suite. It is higher than Mamba-3 MIMO (52.39) and KDA (52.28). Gated DeltaNet-2 is averaging 53.97 vs. Mamba-3 MIMO, which averages 52.72 in hybrid settings. As the size of the recurrent and the current states is the same, it’s the update rules that are responsible for the gains, not the memory.

The most obvious gains are seen in RULER’s long context retrieval. S-NIAH-2 rises to 93.0 in the recurrent mode. S-NIAH-3 jumps up from 63.2 to 89.8 (KDA). MK-NIAH-1 climbs to 37.8 (KDA) at 4K from 28.8 (KDA).

Gated DeltaNet-2 leads in real-world retrieval as well (SWDE SQuAD FDA TriviaQA NQ Drop DROP) for both settings. The hybrid average stands at 42.28 and the recurrent is 29.88.

Marktechpost’s Visual Explainer

git clone https://github.com/NVlabs/GatedDeltaNet-2.git
GatedDeltaNet-2

Build the Dockerfile environment.
Docker build -t Gdn2
docker run --gpus all -it —ipc=host -v $PWD:/workspace gdn2

Pre-train Python.. 
  --train_data_dir ${TRAIN_DATA} 
  --val_data_dir ${VALIDATION_DATA} 
  --output_root ${SAVE_DIR} 
  --exp_name ${NAME} 
  --model_name ${MODEL} 
  --train_config ${CONFIG} 
  --eval_iters ${EVAL_ITERS} 
  --learning_rate ${LR} 
  --micro_batch_size ${MICRO_BATCH_SIZE}

@article{hatamizadeh2026gdn2,
  title   = {Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention},
  author  = {Hatamizadeh, Ali and Choi, Yejin and Kautz, Jan},
  journal = {arXiv preprint},
  year    = {2026}
}

The Key Takeaways

• GaThe ted DeltaNet-2 splits the scalar β_t into a channel-wise erase gate The bThe t (key axis), and channel-wise write gates WThe t (value axis).
• When both gates collapse, the update will recover KDA and Gated DeltaNet.
• Triton is fused with a gate aware backward and an asymmetric erase factor.
• It has a better average than Mamba-2 and Gated DeltaNet.
• Largest gains come on RULER long-context retrieval — S-NIAH-3 at 2K rises 63.2 → 89.8 You can also find out more about the following: MK-NIAH-1 at 4K rises 28.0 → 37.8 over KDA (recurrent).

Check out the Paper and Repo. Also, feel free to follow us on Twitter Don’t forget about our 150k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.

You can partner with us to promote your GitHub Repository OR Hugging Page OR New Product Launch OR Webinar, etc.? Connect with us