H3-Tiny

H3-Tiny: Structured Pruning of MiniMax H3

Haoshuai Fu · Bin Chen
Video and audio generation · 33B → 17B → 14B
arXiv coming soon Code Models Results

TL;DR H3-Tiny compresses the 33B MiniMax H3 Ref2VA transformer to 17B without any training, and to 14B with a brief recovery training distilled from the original 33B, while keeping its synchronized audio.

Nearly 40% of the original parameters are per-block AdaLN projections of a fixed time embedding; we factor them to rank 8 without data. We then remove 8 of 50 blocks chosen by audio-aware block influence and narrow the feed-forward layers. A few-step version of the 17B samples in as few as 4 steps, 6 by default.

Which blocks can go

How much each of the 50 blocks changes its input, separately for video and audio tokens. A few blocks barely do anything; this is what the pruning below exploits.

video tokens audio tokens removed in 17B protected for audio
Block influence, 1 − cos(input, output), 90th percentile over calibration clips and timesteps, square-root scale. Blocks 6–11 barely touch either modality. We remove blocks 6–11, 13 and 15 and keep the twelve blocks the audio depends on most. The same search on H3's separate FL2VA / T2VA transformer removes exactly the same eight blocks.
Official MiniMax H3
33.1B

50 blocks; 13.0B of it in AdaLN projections.

H3-Tiny 17B · no training
17.1B

Rank-8 AdaLN, 42 blocks. Also as a few-step model: 6 steps by default, as few as 4.

H3-Tiny 14B v0.1 · recovered
14.0B

FFN narrowed to 9792 channels, then distilled from the 33B.

Method Overview

Each stage measures error on the inputs the layer really receives, and weighs audio separately so the short audio sequence is not drowned out by video.

STAGE 1

Low-rank AdaLN

Every block's modulation table is a linear map of a fixed time embedding. We sample that embedding over the video and audio schedules, whiten it, and keep the best rank-8 map for those inputs. No data is needed.

13.0B → 37M parameters · error 1–3e-5
STAGE 2

Block removal

We rank blocks by influence on generated video, generated audio, references and text, protect the twelve most important for audio, and keep the drop set with the lowest audio error on held-out clips.

50 → 42 blocks · 24 + 8 clips
STAGE 3

Narrower feed-forward

We keep the SwiGLU channels with the most output energy across both modalities and refit the down projection by ridge regression anchored at the original weights, with audio weighted twice.

14336 → 9792 channels · 17.1B → 14.0B

Recovery. The 14B learns to match the frozen 33B teacher's velocity on noised clips at every point of the 50-step trajectory, audio weighted twice, so it keeps the ordinary sampler. The few-step 17B adds a further few-step distillation; details will follow with the paper. Training code will be released with the paper.

Checkpoints

All run in the official diffusers MiniMax H3 pipelines with the H3-Tiny transformer: Ref2VA, or FL2VA and T2VA for the FL2VA checkpoint.

CheckpointParametersBlocksFFN widthTrainingSteps
MiniMax H3 Ref2VA (teacher)33.1B5014336–50
H3-Tiny-17B-Pruned-Ref2VA17.1B4214336none50
H3-Tiny-17B-Pruned-FL2VA FL2VA · T2VA17.1B4214336none50
H3-Tiny-17B-FewStep-Ref2VA-v0.117.1B4214336few-step distillation6 (4 or more)
H3-Tiny-14B-OKD-Ref2VA-v0.114.0B429792distillation (OKD)50

Results

Same prompt, references and seed for every model. Full-step results are shown next to the original 33B and HyperFlow H3 Flash at 50 steps; few-step results next to HyperFlow H3 Flash at the same number of steps, with the original 33B at 50 steps on the left for reference. Turn the sound on.

Downscaled previews. All reference images are our own renders.

Citation

The paper is in preparation; until then please cite the repository.

@misc{h3tiny2026,
  title        = {H3-Tiny: Structured Pruning of MiniMax H3},
  author       = {Fu, Haoshuai and Chen, Bin},
  year         = {2026},
  howpublished = {\url{https://github.com/fuyi25/H3-Tiny}},
  note         = {arXiv coming soon}
}