Diffusion models run the transformer 50-100 times per image. Two distillation techniques cut this down, but they work differently, cost differently to train, and break differently. Here’s how to choose.
Neither technique requires a special dataset. The teacher model generates the training signal. You can use the original training data, or even random noise as input.
A good example is FLUX.2 klein 4B from Black Forest Labs. It is a 4B parameter rectified flow transformer that already ships as a distilled model: 4 inference steps, sub-second generation, ~13GB VRAM. According to the official model table, it uses both step distillation (reduced to 4 inference steps) and guidance distillation (CFG baked into a single pass). At inference time you set guidance_scale=1.0 because the guidance effect is already distilled into the model. This is exactly what combining both techniques looks like in practice.
For background on where GPU memory goes during inference, see Where Does GPU Memory Actually Go?
Step Distillation: Fewer Steps
What: Train a student where 1 step = 2 teacher steps. Repeat: 50 → 25 → 12 → 8 → 4.
Why it works: Each round only matches 2 consecutive steps, not the full 50-step trajectory. The student is initialized from the teacher’s weights, so it converges fast.
The sweet spot: 4-8 steps. Below 4, faces get soft and textures lose crispness. At 1 step, you need adversarial losses (like SDXL Turbo) to recover sharpness. This reduces output diversity.
Training cost: Multiple rounds, each requiring a dataset + GPU hours. But you pay once; the distilled model serves forever.
Pitfalls:
- Each halving round accumulates error, so quality degrades progressively
- v-prediction parameterization is needed (standard ε-prediction breaks at high noise levels with few steps)
- Reducing steps weakens classifier-free guidance because the effect relies on repeated small adjustments
Guidance Distillation: Cheaper Steps
What: Train a student to produce the guided output in 1 pass instead of 2.
Why 2 passes exist: CFG subtracts the unconditional output from the conditional output to isolate the prompt’s contribution, then amplifies it 7×. Both passes take the current noisy image as input, so neither can be precomputed.
Why it works: The student learns to directly predict the amplified result, conditioned on the guidance scale. One forward pass replaces two.
Training cost: Single round, simpler task than step distillation. The student can still vary guidance strength at inference time.
Pitfalls:
- Only helps if you use CFG (most text-to-image models do)
- Must be applied before step distillation. Otherwise, the guidance effect is already weakened by fewer steps
- May not perfectly capture guidance at all noise levels, causing subtle prompt adherence issues
Decision Guide
| Scenario | Recommendation | Speedup |
|---|---|---|
| Ship fast, minimal training budget | Guidance distillation | 2× |
| Maximum throughput on limited hardware | Step distillation to 4-8 steps | 6-12× |
| Real-time generation (< 100ms) | Both (guidance first, then steps) + FP8 | ~24× |
| Can’t retrain at all | Caching (TeaCache, DBCache) | ~2× |
The Third Option: No Training Required
If retraining isn’t an option, caching skips transformer passes at runtime by detecting when consecutive steps produce nearly identical outputs. Zero training cost, ~2× speedup.
Caching is orthogonal to distillation. You can stack all three.
Always validate with a golden set
Distillation and quantization both trade quality for speed. The tradeoff is different for every model, every dataset, and every use case. You must measure it before deploying.
Build a validation set of 200-500 prompts that represent your production traffic. Include edge cases: long prompts, rare styles, text rendering, faces, fine details. Generate images with both the teacher (full model, 50 steps, CFG) and the student (distilled/quantized). Compare using:
- FID (Frechet Inception Distance): measures distribution-level quality. Lower is better. Expect 1-5 point increase after distillation.
- CLIP score: measures prompt-image alignment. If this drops significantly, your guidance distillation may be too aggressive.
- Human evaluation: no metric replaces looking at the outputs. Have 2-3 people rate 100 pairs blind. Focus on faces, text, and fine textures.
- LLM-as-a-judge: use a vision-language model (like GPT-4o or Gemini) to score image quality, prompt adherence, and artifact detection at scale. This automates what human eval does manually and scales to your full validation set. Note: this is a costly option. Each image evaluation requires a VLM API call, so running it on 500 images can cost $5-15 per evaluation round. Use it for final validation, not iterative tuning.
Run this validation after every change: after quantization, after each distillation round, after combining techniques. The quality loss compounds. A 2% drop from FP8 plus a 5% drop from step distillation plus a 3% drop from guidance distillation can add up to a noticeable degradation that no single step revealed.
References
- Salimans, Ho. “Progressive Distillation for Fast Sampling of Diffusion Models”, ICLR 2022.
- Meng et al. “On Distillation of Guided Diffusion Models”, CVPR 2023.
- Ho, Salimans. “Classifier-Free Diffusion Guidance”, NeurIPS 2021.
- Dieleman. “The paradox of diffusion distillation”, 2024.
- Sauer et al. “Adversarial Diffusion Distillation”, 2023.
- Black Forest Labs. “FLUX.2: Frontier Visual Intelligence”, 2025.