H3-Tiny: Structured Pruning of MiniMax H3
TL;DR H3-Tiny compresses the 33B MiniMax H3 Ref2VA transformer to 17B without any training, and to 14B with a brief recovery training distilled from the original 33B, while keeping its synchronized audio.
Nearly 40% of the original parameters are per-block AdaLN projections of a fixed time embedding; we factor them to rank 8 without data. We then remove 8 of 50 blocks chosen by audio-aware block influence and narrow the feed-forward layers. A few-step version of the 17B samples in as few as 4 steps, 6 by default.
Which blocks can go
How much each of the 50 blocks changes its input, separately for video and audio tokens. A few blocks barely do anything; this is what the pruning below exploits.
50 blocks; 13.0B of it in AdaLN projections.
Rank-8 AdaLN, 42 blocks. Also as a few-step model: 6 steps by default, as few as 4.
FFN narrowed to 9792 channels, then distilled from the 33B.
Method Overview
Each stage measures error on the inputs the layer really receives, and weighs audio separately so the short audio sequence is not drowned out by video.
Low-rank AdaLN
Every block's modulation table is a linear map of a fixed time embedding. We sample that embedding over the video and audio schedules, whiten it, and keep the best rank-8 map for those inputs. No data is needed.
Block removal
We rank blocks by influence on generated video, generated audio, references and text, protect the twelve most important for audio, and keep the drop set with the lowest audio error on held-out clips.
Narrower feed-forward
We keep the SwiGLU channels with the most output energy across both modalities and refit the down projection by ridge regression anchored at the original weights, with audio weighted twice.
Recovery. The 14B learns to match the frozen 33B teacher's velocity on noised clips at every point of the 50-step trajectory, audio weighted twice, so it keeps the ordinary sampler. The few-step 17B adds a further few-step distillation; details will follow with the paper. Training code will be released with the paper.
Checkpoints
All run in the official diffusers MiniMax H3 pipelines with the H3-Tiny transformer: Ref2VA, or FL2VA and T2VA for the FL2VA checkpoint.
| Checkpoint | Parameters | Blocks | FFN width | Training | Steps |
|---|---|---|---|---|---|
| MiniMax H3 Ref2VA (teacher) | 33.1B | 50 | 14336 | – | 50 |
| H3-Tiny-17B-Pruned-Ref2VA | 17.1B | 42 | 14336 | none | 50 |
| H3-Tiny-17B-Pruned-FL2VA FL2VA · T2VA | 17.1B | 42 | 14336 | none | 50 |
| H3-Tiny-17B-FewStep-Ref2VA-v0.1 | 17.1B | 42 | 14336 | few-step distillation | 6 (4 or more) |
| H3-Tiny-14B-OKD-Ref2VA-v0.1 | 14.0B | 42 | 9792 | distillation (OKD) | 50 |
Results
Same prompt, references and seed for every model. Full-step results are shown next to the original 33B and HyperFlow H3 Flash at 50 steps; few-step results next to HyperFlow H3 Flash at the same number of steps, with the original 33B at 50 steps on the left for reference. Turn the sound on.
The FL2VA / T2VA transformer of MiniMax H3, pruned with the same recipe (calibrated on FL2VA and T2VA clips) against the original at 50 steps. FL2VA takes a first and a last frame; T2VA only the text. Search picks the same eight blocks as for Ref2VA, and no training follows.
Downscaled previews. All reference images are our own renders.
Citation
The paper is in preparation; until then please cite the repository.
@misc{h3tiny2026,
title = {H3-Tiny: Structured Pruning of MiniMax H3},
author = {Fu, Haoshuai and Chen, Bin},
year = {2026},
howpublished = {\url{https://github.com/fuyi25/H3-Tiny}},
note = {arXiv coming soon}
}