Reusing large language models can help us to increase the speed of our model. “free” GPU computation without losing output quality on an autoregressive basis? Researchers at NVIDIA propose TiDAR, an hybrid sequence-level language model which samples tokens autoregressively and drafts them with diffusion in one forward pass. This research aims to achieve autoregressive qualities while increasing throughput significantly by utilizing free token slots in modern GPUs.
The quality of the problem, system motivation and free token slots
Autoregressive transformers decode one token per step. In realistic batch sizes decoding usually is memory bound because the latency of loading weights and cache KV is dominant, rather than floating point operations. The number of tokens that are in the input within the region where the decoding is memory bound does not affect the latency because the same parameters and KV cache are being used.
Models of masked diffusion already take advantage. With a prefix they can predict multiple tokens at once in a denoising process. This research team refers to these extra positions as free token slot, since profiling shows sending more tokens under this regime barely affects the forward time.
Diffusion LLMs, such as Dream or Llada, still perform worse than strong autoregressive bases on quality. These models sample tokens independently, when decoding multiple tokens at the same time, from a noisy context, and a marginal distribution. It is important to note that this intra-step independence of tokens can negatively impact sequence coherence. This removes the speed advantages of diffusion decoding in practice.
TiDAR was designed to maintain the computational efficiency of diffusion, while recovering autoregressive quality. It uses a common backbone with standard transformer infrastructure.
Dual mode architecture backbone with attention masks
TiDAR can be used to partition the sequence on a very high level. Three sections are divided for each step in the generation:
- Prefixes of tokens accepted
- The previous step involves the creation of tokens.
- Tokens mask that hold candidates pre-drafted for the next stage.
The model then applies a structured mask of attention to this sequence. As in an autoregressive transform, prefix tokens are causally attended to, which allows for chain-factorized prediction of the next token. Tokens within the block in the masking region and drafting region can be attended bi-directionally. This is a variation of Block Diffusion, in which only the decoding is bidirectional. The rest of sequences remain causal.

TiDAR increases the length of the sequence at the time of training to enable both modes. Original input is in the diffusion section and corrupted copies are located in the causal sections. Labels are moved by one token in the causal section to align with the token prediction goal. In the diffusion area, the labels align with the input position.
TiDAR is unique in that it uses a “full mask” strategy. In the diffusion section, all tokens are replaced with a mask token rather than sampling sparse patterns of corruption. In this way, the diffusion loss becomes dense. Also, the loss term numbers in the diffusion and autoregressive sections are kept equal. In most experiments, the research team used a weighting factor of 1.

Self-speculative Generation in One Forward Pass
The generation process is an evaluation of a network function per step.
Step 1TiDAR, when given a prompt, encodes the prefix in a causal manner and then performs diffraction over mask positions. This produces drafted blocks of tokens.
Step 2. During the next steps, every forward pass will perform Double operation
- Verification using Autoregressive Logits with an Extended Prefix and Rejection Sampling Rule, in the spirit of Speculative Decoding.
- Diffusion is used to pre-write the next step, depending on the outcome of all the possible steps.
Rejected tokens will be discarded and the cache entries for their KV will remain. Rejected tokens and cache entries of rejected tokens are removed. Since the drafting and the verification use the same attention mask and backbone, the diffusion computation is done using the tokens that are free in a single forward pass.
This model allows for two different sampling modes: trusting autoregressive prediction or trusting diffusion predictions. Both control how closely the final sample will follow each of the heads. For the 8B, experiments have shown that trusting in diffusion predictions, particularly on math benchmarks is beneficial.
From the system’s perspective, both the layout of the tokens and their number per step have been fixed. TiDAR initialises an attention block mask in advance and reuses slices from this mask throughout the decoding process using Flex Attention. Like Block Diffusion, the architecture also supports an exact KV Cache. It never performs a recompute of KV for tokens accepted and it does not add any extra hyperparameters to inference times.
Sizes and recipes for training models
Qwen2.5, 1.5B, Qwen3, 4B, and 8B models are continuously trained to create TiDAR. Training for the 1.5B variant uses 50B tokens in block sizes of 4, 8, and 16. The 8B version is trained with 150B blocks of size 16 and on 50B tokens. Both systems are based on a maximum 4096 sequence length, a cosine-learning rate schedule, BF16 distributed Adam framework, Torchtitan, LM Megatron, NVIDIA GPUs H100, and BM16.
Evaluation includes coding tasks HumanEval Plus (including MBPP Plus), math tasks GSM8K (including Minerva Math), factual and commonsense task MMLU, Hellaswag and Winogrande.
Results in terms of quality and output
TiDAR 1.5B has a high level of performance on generative code and math problems, while producing an average 7,45 tokens when compared to its autoregressive competitor. The quality of TiDAR’s 8B is not significantly affected by Qwen3-8B, but the generation rate increases to 8.25 tokens for each forward pass.
Because likelihood is calculated with a causal mask, the behaviours of TiDAR 8B and 1.5B are comparable to those observed in similar autoregressive model. Diffusion benchmarks like Dream, Llada, and Block Diffusion use Monte Carlo-based likelihood estimators that are both more expensive and not directly comparable.
When wall clock benchmarks are run on an H100 GPU, with batch sizes 1, TiDAR 1,5B achieved a 4.71-fold increase in the decoding rate compared to that of Qwen2.51.5B. TiDAR 8B is 5.91x faster than Qwen3-8B.
TiDAR is superior to diffusion LLMs for both accuracy and efficiency, as long as diffusion models are limited to decoding 1 token on each pass forward. TiDAR outperforms speculative models such as EAGLE-3, and training-matched block diffusion in efficiency by converting tokens into real tokens faster.
The Key Takeaways
- The TiDAR architecture is a hybrid sequence-level system that samples tokens autoregressively using an attention mask with a mix of causal and bidirectional areas.
- This design takes advantage of the free slots available on GPUs by adding diffusion-drafted tokens and masking them to prefixes. Many positions can be processed with a single forward pass, with a virtually unchanged latency.
- TiDAR uses self-speculative generation. The same core generates tokens in one step, and then verifies and rejects them using autoregressive logits. This eliminates the need for separate models and the overhead associated with classic speculative decoding.
- TiDAR can reach an autoregressive-level quality for coding and math benchmarks with continuous pretraining using Qwen2.5 1.5B.
- TiDAR’s 1.5B and 8B models deliver about 4,71 and 5,91 tokens/second more than the baseline autoregressive model in single GPU batch size 1. They also outperform diffusion LLMs such as Dream and Llada, and close the quality gap when using strong autoregressive model.
Compare
| Aspect | Standard autoregressive transformer | The Diffusion LLM (Class LLaDA, Dream) | Speculative decoding (EAGLE 3 class) | TiDAR |
|---|---|---|---|---|
| Core Idea | Predicts the exact next token with causal attention | This method predicts many parallel tokens by iteratively denoising masked or corrupted sequentials. | The target model accepts and verifies a subset of tokens that are proposed using a draft. | In the same forward passage, single backbone drafts and verifies using autoregression with diffusion. |
| Drafting mechanisms | No, each token is only produced by the model | Denoising by diffusion over masking positions. | The current state of the transformer is converted into draft tokens by a lightweight or truncated version. | The mask tokens are added after the prefix to create a one-step diffusion. |
| Verification mechanisms | The same causal forward is used for both sampling and logit. | No, most sampling relies on diffusion margins within each step. This can lead to a reduction in sequence coherence. | This model uses the target distribution to perform rejection sampling and recalculate logits. | The same backbone generates autoregressive logits for the prefix, which verify diffusion drafts by reject sampling |
| The number of models used for inference | Single model | Single model | The usual set-up includes at least one model for the target plus one model as a minimum. | No extra heads or networks beyond AR projections |
| Token parallelism per forward | Each network evaluation requires a token decoder to generate a single new token. | Several masked tokens are updated at the same time. The effective window is dependent on both schedule and masking policy | The final accept tokens are typically fewer in number than the draft ones | According to the setup, 1.5B would cost around 7.45 tokens and 8B would require approximately 8.25 tokens. |
| The typical single GPU decoding performance compared to AR (batch sizes 1) | Baseline refers to 1 times | Most tuned variants are capable of achieving 3x throughput over strong AR baselines. This is often achieved by making quality compromises in math or coding. | The empirical evidence shows that the throughput is around 2.5 to 2 times higher than native autoregressive. | The reported speedups were 4.71 for 1.5B and 591 for 8B in comparison to the matched auto-regressive Qwen base on one H100, batch size 1. |
| The quality of the AR base versus its strength | Refer to the benchmarks for coding, mathematics and knowledge. | When step counts are reduced for chase speed but the quality is still sensitive to schedule decoding, it can cause a drop in quality. | When acceptance rates are high, the quality of the model can be close to that intended. However, if the draft model is weak and misaligned it may degrade. | It closely matches the Qwen autoregressive baselines for coding, maths and knowledge tasks. But it achieves a higher throughput. |
| Likelihood evaluation support | Exact log likelihood under causal factorisation, standard lm eval harness compatible | Monte Carlo estimators and approximations are often needed to estimate the likelihood of a sequence. | It uses the original Autoregressive Model for Log Likelihood, so the evaluation is accurate but it does not use speed tricks. | The likelihood of a given event is calculated exactly as an auto-regressive transform |
| KV cache behaviour | Standard cache: reused to hold all previous tokens. Add one token per step. | The cache use is dependent on the diffusion method. Some methods rewrite segments repeatedly, increasing cache churn. | The KV caching system is needed for both target and draft models. Also, extra accounting for rejected and confirmed tokens. | Cache sharing between diffusion and autoregressive components, tokens accepted are never recomputed but cached only once, and rejected tokens evicted. |
The TiDAR model is useful for bridging the autoregressive and diffusion models by using a unified backbone. It increases tokens for network functions evaluation by utilizing free token slots, self-speculative generation and a similar method to Qwen’s baseline. Full mask diffusion and KV cache support make this a practical solution for serving production style on H100 GPUs. TiDAR’s overall result shows how diffusion drafting, autoregressive verification and a single LLM can exist in harmony.
Take a look at the PAPER. Please feel free to browse our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter. Wait! Are you using Telegram? now you can join us on telegram as well.
Asif Razzaq, CEO of Marktechpost Media Inc. is a visionary engineer and entrepreneur who is dedicated to harnessing Artificial Intelligence’s potential for the social good. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. This platform has over 2,000,000 monthly views which shows its popularity.

