“>
Unsloth, kernel-level improvements on one GPU
Unsloth’s published benchmarks The Llama dataset, batch size 2, and gradient accumulation 4 were used to set up the setup. Setup used Alpaca datasets, batch size 2, gradient accumulation 4, and the Alpaca dataset. QLoRA was ranked 32 for all layers.
It is important to note that the word “you” means “you”. MoE results are larger. Unsloth fine-tuned unsloth/gpt-oss-20b-BF16 On an NVIDIA b200. This game shows 712.33ms/step in 8K context versus 52226.86ms for Transformers 5. This is a gap of 7.3x. This gap increases to 4.82x at 4K, but only to 1.37x at 1K.
Unsloth argues that the trend is dependent on model, and this assertion applies to gpt. In this case, there is an increase in speed with increasing sequence length. Flex Attention and MoE kernels are credited for the increased performance.
The Qwen3-30B A3B runs in the opposite direction. Reports of its speedup have dropped from 1.7x in 1K down to 1.1x in 16K. Memory saving moves in the opposite direction. It goes from 2% up to 15%.
Qwen3-30BA3B for H100 can achieve up to 1.77x. The GLM-4.7 Flash on the RTX Pro 6000 can reach a maximum of 2.1x. A collaboration with AMD Llama-3.5-8B LoRA SFT was 2.07 seconds per step. FlashAttention-2 plus TRL took 2.87s/step. This is 1.39x the difference with identical loss curves.
Axolotl : borrowed kernels, native parallelism
Axolotl is added custom Triton kernels and autograd functions LoRA will be launched in February 2025 with Unsloth cited as an inspiration. The opt-in process is through lora_mlp_kernel, lora_qkv_kernel” lora_o_kernel.
The following are some examples of the use of release notes SonicMoE supports LoRA. This product delivers up 1.45x performance with 30% reduction in memory over the previous generation. grouped_mm baseline. The figure represents Qwen3.5 35B A3B LoRA in 8 bits on a H100SXM.
Axolotl ships FlashAttention 2/3/4 as well as xFormers. SageAttention is also available, along with Liger Kernel.
Baseline standard everyone uses
It is more important to compare TRL with raw GPU throughput than just the absolute number. The wide range of speed and memory levers is documented. Reducing Memory Usage The following are some examples of how to get started: Speeding Up Training.
These levers are packing, batching without padding, and truncation. Liger KernelGRPO has a sleep mode that uses vLLM. The TRL has also a first-party Unsloth integrationBoth are mutually inclusive.
LLaMA Factory: Speed by Delegation
LLaMA Factory does not create its kernels. The config flags expose other people’s code.
Setting Use_unsloth is true Unsloth Patch is activated. Project’s changelog indicates 170% relative speeds from that path. Unsloth lists 117% speed, and 50% memory for its long-sequenced training. The training also helps with memory. enable_liger_kernel: true FlashAttention-2 is available via flash_attn: fa2.
Memory floors
Unsloth publications a VRAM requirements table Sorted by number of parameters. There are 6 GB listed for an 80B model using 4-bit QLoRA, and 41 GB when it comes to 70B. LoRA 16-bit models cost 22 GB, and 164GB.
LLaMA-Factory’s README hardware table The same scheme is used for QLoRA 4-bit. There are 6 GB listed for 7B; 24 GB is given at 30B and 48 GB in 70B. The full bf16 tuning of 70B requires 600 GB.
The two tables below describe the minimums. Real number can be shifted by batch size and sequence length.
The context length makes a big difference
Unsloth’s peak VRAM in a context with a set budget is not as interesting as the context that a VRAM budget can support. Unsloth’s context length benchmarks The differences between Llama QLoRA 3.1 8B at batch size 1, rank 32, and Llama QLoRA 3.1 8B QLoRA 3.1 QLoRA at ranks 32 and 1 are striking.
| GPU VRAM | Unslotted context | Transformers + F A2 context |
|---|---|---|
| The 8GB | 2,972 | OOM |
| 16GB | 40,724 | 2,551 |
| The 24GB | 78,475 | 5,789 |
| 48 GBP | 191,728 | 15,502 |
| 60 GB | 342,733 | 28,454 |
Unsloth credits this to the combination of its gradient checkpointing algorithms and Apple’s Cut Cross Entropy. Llama 3 70B reports 89.389 tokens on an 80GB A100. FA2 is at 6,916.
MoE Memory Story
In 2026, the most significant change in memory behavior was seen with MoE. Unsloth Reports gpt-oss-20b The Qwen3-30BA3B 16-bit LoRA system requires 63GB for fine tuning.
Transformers V5 uses 73.80GB of memory in 8K. B200’s gpt oss used only 47.43GB. Transformers V5 ran out of memory at 16K and Unsloth consumed 55.13GB.
This mechanism uses a LoRA split formulation. PEFT multiplies all LoRA deltas across experts, before the MoE Matmul. Unsloth can reorder the operations, which avoids materialization but is identical mathematically.
Axolotl approaches the same issue differently. The same problem is approached differently by Axolotl. MoE expert quantization Quantizes the expert weights when loading models, releasing the original Bf16 Tensor instantly.
Transformers version 5 is the reason. The expert layers have moved to nn.Linear Fusion nn.Parameter 3D tensors. Bitsandbytes couldn’t quantize them during load. Axolotl documents report GLM 4.7-Flash QLoRA dropping roughly from 127 GiB in reserved memory to approximately 23 GiB. quantize_moe_experts: true.
Here is the point where ranking reverses. Unsloth’s one-GPU advantage does not translate.
Axolotl, the matrix with deepest parallelism
Axolotl’s multi-GPU guide There are three sharding options: DeepSpeed ZeRO, stages 1 through 3 (FSDP), and DDP. FSDP2 has become the preferred path. FSDP1 should no longer be implemented.
Then, there’s the newest addition to our list: N-D Parallelism guide PyTorch can be used to compose data, tensors and context, as well as expert parallelism. DeviceMesh. The documentation confirms the FSDP+TP (FSDP+TP+CP), HSDP+TP (FSDP+TP+CP), FSDP+CP and FSDP+EP.
In v1, two combinations are expressly unsupported. In v1, expert parallelism is not supported. They are also not compatible with pure DDP.
Axolotl’s sequence parallelism Uses of ring-flash-attention library. The library is a good example. published H100 benchmark QLoRA’s support for Llama 3 1 8 B QLoRA the tradeoff.
| SP degree | Max context | Context-scaling | Tokens/sec | Speedup / GPU count |
|---|---|---|---|---|
| 1 | 17,408 | 1.00x | 9,104 | 100.0% |
| 2 | 34,816 | 2.00x | 15,806 | 86.8% |
| 4 | 66,560 | 3.82x | 12,314 | 33.8% |
| 8 | 129,024 | 7.41x | 11,096 | 15.2% |
Context scaling is linear. Throughput efficiency falls. On the 4090s, at SP level 8, this benchmark shows a speedup of 0.88x. The training got slower as context tokens reached 32 768.
Axolotl supports multi-node and torchrun training. Ray.
Two sequence-splitting Backends
TRL’s distributed training guide Draws a clear distinction. Context Parallelism refers to Ring Attention for FSDP2. Sequence Parallelism refers to ALST/Ulysses for DeepSpeed.
Use Ring Attention Accelerate 1.1.0+ cp_size The following are some examples of how to use cp_backend="torch"A SDPA-only solution is currently available. FlashAttention does not work on this path. Sequences must divide by cp_size * 2.
It requires DeepSpeed 0.18.1+, and Accelerate 1.2.0+. This uses sp_size The following are some examples of how to use sp_backend="deepspeed" It is compatible with FlashAttention-2. The attention count is the limit. num_heads >= sp_size.
TRL offers specific advice. Ring Attention is suitable for sequences of 1M+ tokens with a limited topology. Ulysses is suitable for NVLink and InfiniBand networks, as well as sequences of up to 500k tokens.
TRL’s Ring Attention benchmark optimized Qwen3-8B on 1, 2, 4 and 8 H100 graphics cards. When using 8 GPUs you can train contexts longer than 300k tokens.
LLaMA: Megatron standard engine with LLaMA factory
LLaMA-Factory’s distributed training docs The documents cover DDP and DeepSpeed as well as FSDP2, Ray, and FSDP2 for both single and multi-node run. These documents also cover DeepSpeed AutoTP which is a combination of tensor and ZeRO parallelism.
Megatron’s core-training backend was its most important addition in 2025. mcore_adapter. It opens up an actual large-scale training path.
The FSDP+QLoRA route fine-tunes the 70B model using two 24 GB graphics cards. This is the least expensive documented path to 70B.
Interface is the friction point. LlamaBoard is not the place to configure distributed systems. Dev teams who used LLaMA-Factory to get its zero-code UI, lose this feature when scaling past one GPU.
Open gap: Unsloth
Unsloth’s multi-GPU documentation It states that Multi-GPU is possible through Accelerate, DeepSpeed and FSDP/DDP. The article also mentions that the setup process is manual and complex, while official support has yet to be announced.
This is the practical way to learn. Accelerate launch train.py The following are some examples of how to use torchrun --nproc_per_node N_GPUS train.py. Models too big for a single GPU device_map = "balanced" Splits the model between devices.
Unsloth’s PyPI listing Marks multi-GPU with major improvements in the works. The Studio changelog Description of automatic allocation to multiple GPUs for training and inference as early as March 2026.
The position is obvious when read together. Unsloth can support multiple GPUs. The matrix of parallelism that Axolotl, TRL and other documents describe is not available yet.
- Unsloth It also breaks when your model is not on its supported list, since gains come from architecture-specific kernels. It also breaks when your model is not in its supported list, since gains come from architecture-specific kernels.
- Axolotl The learning curve is broken into smaller segments. The FSDP2 and DeepSpeed constraints are configured. Those restrictions include GPU counts, sequence lengths, and attention head count.
- TRL It breaks defaults. You get the correct primitives and not those that have been tuned. The memory optimizations and parallelism plan are provided by you.
- LLaMA-Factory Breaks at the UI border. The abstraction of the UI is good up to a node, but thin beyond that.
- You can choose between LoRA and QLoRA architecture when buying a single-consumer GPU. Unsloth. It is justified by the context length alone.
- Two to eight GPUs with long context or RLHF pipelines. Axolotl. It is best to use FSDP2 and sequence parallelism.
- Customized training loops and novel algorithms for post-training, as well as tight hugging face coupling. TRL. The other wraps the previous layer.
- Broadest model coverage, non-engineer operators, fastest first run: LLaMA-Factory. When you are scaling, switch to CLI.
- The choices available are not limited. LLaMA Factory is able to use Unsloth for the backend. TRL offers an Unsloth Integration. Axolotl refers to TRL trainers as internal.
- Unsloth has the fastest single GPU speed, and context size; however its multi-GPU weakness is still documented.
- Axolotl provides the highest parallelism matrix, including FSDP2, DeepSpeed and TP.
- Ring Attention now available, as well as ALST/Ulysses split sequence.
- LLaMA factory trades off depth and breadth. 100+ models with zero code UI. Megatron backend. Distributed setup only via CLI.
- The scale of context for sequence parallelism is near linear, but the throughput performance drops dramatically after four GPUs.
Asif Razzaq serves as the CEO at Marktechpost Media Inc. As an entrepreneur, Asif has a passion for harnessing Artificial Intelligence to benefit society. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. Over 2 million views per month are a testament to the platform’s popularity.

