vllm.model_executor.kernels.mhc
¶
Modules:
-
aiter– -
cpu– -
flashinfer_mega–MHCFusedPostPreOp on FlashInfer's fused mHC kernel (
flashinfer.mega_mhc). -
tilelang– -
tilelang_kernels– -
torch– -
triton– -
warmup–
Functions:
-
hc_head_fused_cpu–CPU-ported HC head reduction (see
test_hc_head_cpuin -
mega_mhc_supported–Whether FlashInfer has a fused mHC program for this shape and device.
-
mhc_fused_post_pre_aiter–Fused mHC post + next mHC pre on ROCm via AITER.
-
mhc_fused_post_pre_delayed_rms_norm_aiter–mhc_pre_delayed_aiterwith the RMSNorm of the collapse folded in: -
mhc_fused_post_pre_flashinfer_mega–MHCFusedPostPreOp (with its RMSNorm) as one FlashInfer launch.
-
mhc_fused_post_pre_tilelang–See
_mhc_fused_post_pre_tilelang_impl. Never writes to its inputs. -
mhc_post_cpu–CPU-ported mHC post block (see
mhc_post_torchfor the eager reference). -
mhc_pre_aiter–Forward pass for mHC pre block.
-
mhc_pre_cpu–CPU-ported mHC pre block (see
mhc_pre_torchfor the eager reference). -
mhc_pre_delayed_aiter–MHC pre with the pre-mix carried in from the previous sublayer.
-
mhc_pre_delayed_torch–Reference for mHC pre using coefficients from the previous sublayer.
-
mhc_pre_mix_triton–Pre-mix gate for the delayed mHC pre, from AITER's split-k GEMM output.
-
mhc_pre_tilelang–Forward pass for mHC pre block.
-
mhc_pre_torch–Forward pass for mHC pre block.
hc_head_fused_cpu(hidden_states, hc_fn, hc_scale, hc_base, rms_norm_eps, hc_eps)
¶
CPU-ported HC head reduction (see test_hc_head_cpu in
tests/kernels/mhc/test_mhc_kernels.py for the eager reference this is tested
against).
The ported kernel's C++ signature takes (hc_eps, norm_eps) -- the
opposite order from this wrapper's (rms_norm_eps, hc_eps), which matches
the real call site in models/deepseek_v4/cpu/model.py.
Source code in vllm/model_executor/kernels/mhc/cpu.py
mega_mhc_supported(hidden_size, hc_mult)
¶
Whether FlashInfer has a fused mHC program for this shape and device.
Source code in vllm/model_executor/kernels/mhc/flashinfer_mega.py
mhc_fused_post_pre_aiter(x, residual, post_layer_mix, comb_res_mix, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1, tile_n=1, norm_weight=None, norm_eps=0.0)
¶
Fused mHC post + next mHC pre on ROCm via AITER.
Returns residual_cur, post_mix_cur, comb_mix_cur, layer_input_cur.
Source code in vllm/model_executor/kernels/mhc/aiter.py
mhc_fused_post_pre_delayed_rms_norm_aiter(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, pre_mix, sublayer_out, post_layer_mix, comb_res_mix, norm_weight, norm_eps, residual_out)
¶
mhc_pre_delayed_aiter with the RMSNorm of the collapse folded in:
layer_input comes back normalised with norm_weight / norm_eps.
Source code in vllm/model_executor/kernels/mhc/aiter.py
mhc_fused_post_pre_flashinfer_mega(x, residual, post_layer_mix, comb_res_mix, fn, fn_split, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, norm_weight, norm_eps)
¶
MHCFusedPostPreOp (with its RMSNorm) as one FlashInfer launch.
Same contract as mhc_fused_post_pre_tilelang, plus fn_split, the
BF16 pieces of fn (split_mhc_fn).
Source code in vllm/model_executor/kernels/mhc/flashinfer_mega.py
144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 | |
mhc_fused_post_pre_tilelang(x, residual, post_layer_mix, comb_res_mix, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1, tile_n=1, norm_weight=None, norm_eps=1e-06)
¶
See _mhc_fused_post_pre_tilelang_impl. Never writes to its inputs.
Source code in vllm/model_executor/kernels/mhc/tilelang.py
mhc_post_cpu(x, residual, post_layer_mix, comb_res_mix)
¶
CPU-ported mHC post block (see mhc_post_torch for the eager reference).
Source code in vllm/model_executor/kernels/mhc/cpu.py
mhc_pre_aiter(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1, norm_weight=None, norm_eps=0.0)
¶
Forward pass for mHC pre block.
Parameters:
-
(residual¶Tensor) –shape (..., hc_mult, hidden_size), dtype torch.bfloat16
-
(fn¶Tensor) –shape (hc_mult3, hc_mult * hidden_size), dtype torch.float32
-
(hc_scale¶Tensor) –shape (3,), dtype torch.float32
-
(hc_base¶Tensor) –shape (hc_mult3,), dtype torch.float32
-
(rms_eps¶float) –RMS normalization epsilon
-
(hc_pre_eps¶float) –pre-mix epsilon
-
(hc_sinkhorn_eps¶float) –sinkhorn epsilon
-
(hc_post_mult_value¶float) –post-mix multiplier value
-
(sinkhorn_repeat¶int) –number of sinkhorn iterations
-
(n_splits¶int, default:1) –split-k factor;
-
(norm_weight¶Tensor | None, default:None) –optional RMSNorm weight fused into the pre kernel
-
(norm_eps¶float, default:0.0) –epsilon for the fused RMSNorm when norm_weight is set
Returns:
-
post_mix(Tensor) –shape (..., hc_mult), dtype torch.float32
-
comb_mix(Tensor) –shape (..., hc_mult, hc_mult), dtype torch.float32
-
layer_input(Tensor) –shape (..., hidden_size), dtype torch.bfloat16
Source code in vllm/model_executor/kernels/mhc/aiter.py
mhc_pre_cpu(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1, norm_weight=None, norm_eps=0.0)
¶
CPU-ported mHC pre block (see mhc_pre_torch for the eager reference).
The ported kernel (hc_pre_fused_cpu) only exposes one merged hc_eps
(used for both hc_pre_eps/hc_sinkhorn_eps) and hardcodes the post-mix
multiplier to 2.0 -- true of every real call site in
models/deepseek_v4/cpu/model.py, so this is not a capability loss here.
Source code in vllm/model_executor/kernels/mhc/cpu.py
mhc_pre_delayed_aiter(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, pre_mix=None, sublayer_out=None, post_layer_mix=None, comb_res_mix=None, residual_out=None)
¶
MHC pre with the pre-mix carried in from the previous sublayer.
Matches mhc_pre_delayed_torch: the stream collapse uses pre_mix
rather than the gate computed here, and that gate is returned as the
pre-mix for the next sublayer seam.
Parameters:
-
(residual¶Tensor) –shape (..., hc_mult, hidden_size), dtype torch.bfloat16
-
(fn¶Tensor) –shape (hc_mult3, hc_mult * hidden_size), dtype torch.float32
-
(hc_scale¶Tensor) –shape (3,), dtype torch.float32
-
(hc_base¶Tensor) –shape (hc_mult3,), dtype torch.float32
-
(rms_eps¶float) –RMS normalization epsilon
-
(hc_pre_eps¶float) –pre-mix epsilon
-
(hc_sinkhorn_eps¶float) –sinkhorn epsilon
-
(hc_post_mult_value¶float) –post-mix multiplier value
-
(sinkhorn_repeat¶int) –number of sinkhorn iterations
-
(pre_mix¶Tensor | None, default:None) –shape (..., hc_mult) from the previous sublayer, or None at model entry to select residual stream zero.
-
(sublayer_out¶Tensor | None, default:None) –attention or FFN output, shape (..., hidden_size). When given with the two mixes below, the preceding post block is applied here so AITER can fold it into the pre projection.
-
(post_layer_mix¶Tensor | None, default:None) –shape (..., hc_mult, 1), post gate for that block.
-
(comb_res_mix¶Tensor | None, default:None) –shape (..., hc_mult, hc_mult), residual comb for it.
-
(residual_out¶Tensor | None, default:None) –shape (..., hc_mult, hidden_size), written with the post block's new residual. Required exactly when the post is folded in.
Returns:
-
post_mix(Tensor) –shape (..., hc_mult, 1), dtype torch.float32
-
comb_mix(Tensor) –shape (..., hc_mult, hc_mult), dtype torch.float32
-
layer_input(Tensor) –shape (..., hidden_size), dtype torch.bfloat16
-
next_pre_mix(Tensor) –shape (..., hc_mult), dtype torch.float32
Source code in vllm/model_executor/kernels/mhc/aiter.py
mhc_pre_delayed_torch(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, pre_mix=None, x=None)
¶
Reference for mHC pre using coefficients from the previous sublayer.
Source code in vllm/model_executor/kernels/mhc/torch.py
mhc_pre_mix_triton(gemm_out, sqrsum, hc_scale, hc_base, hc_mult, hc_hidden_size, rms_eps, hc_pre_eps)
¶
Pre-mix gate for the delayed mHC pre, from AITER's split-k GEMM output.
AITER's mhc_pre_big_fuse consumes the unreduced [splitk, tokens,
hc_mult3] GEMM output and the matching row square-sums, but only returns
the post and comb gates. The delayed formulation also needs the pre gate,
to carry into the next sublayer seam. It is the same slice of the same
numbers, so recover it here rather than repeating the projection.
Source code in vllm/model_executor/kernels/mhc/triton.py
mhc_pre_tilelang(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1, norm_weight=None, norm_eps=1e-06)
¶
Forward pass for mHC pre block.
Parameters:
-
(residual¶Tensor) –shape (..., hc_mult, hidden_size), dtype torch.bfloat16
-
(fn¶Tensor) –shape (hc_mult3, hc_mult * hidden_size), dtype torch.float32
-
(hc_scale¶Tensor) –shape (3,), dtype torch.float32
-
(hc_base¶Tensor) –shape (hc_mult3,), dtype torch.float32
-
(rms_eps¶float) –RMS normalization epsilon
-
(hc_pre_eps¶float) –pre-mix epsilon
-
(hc_sinkhorn_eps¶float) –sinkhorn epsilon
-
(hc_post_mult_value¶float) –post-mix multiplier value
-
(sinkhorn_repeat¶int) –number of sinkhorn iterations
-
(n_splits¶int, default:1) –retained for the shared MHC operator API; the active GEMM backend selects its split factor internally.
-
(norm_weight¶Tensor | None, default:None) –optional RMSNorm weight, shape (hidden_size,), dtype torch.bfloat16. When provided, RMSNorm is fused into the layer_input write path of the big_fuse kernel.
-
(norm_eps¶float, default:1e-06) –epsilon for the fused RMSNorm; only consulted when norm_weight is given.
Returns:
-
post_mix(Tensor) –shape (..., hc_mult), dtype torch.float32
-
comb_mix(Tensor) –shape (..., hc_mult, hc_mult), dtype torch.float32
-
layer_input(Tensor) –shape (..., hidden_size), dtype torch.bfloat16
Source code in vllm/model_executor/kernels/mhc/tilelang.py
524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 | |
mhc_pre_torch(residual, fn, hc_scale, hc_base, rms_eps, hc_pre_eps, hc_sinkhorn_eps, hc_post_mult_value, sinkhorn_repeat, n_splits=1)
¶
Forward pass for mHC pre block.
Parameters:
-
(residual¶Tensor) –shape (..., hc_mult, hidden_size), dtype torch.bfloat16
-
(fn¶Tensor) –shape (hc_mult3, hc_mult * hidden_size), dtype torch.float32
-
(hc_scale¶Tensor) –shape (3,), dtype torch.float32
-
(hc_base¶Tensor) –shape (hc_mult3,), dtype torch.float32
-
(rms_eps¶float) –RMS normalization epsilon
-
(hc_pre_eps¶float) –pre-mix epsilon
-
(hc_sinkhorn_eps¶float) –sinkhorn epsilon
-
(hc_post_mult_value¶float) –post-mix multiplier value
-
(sinkhorn_repeat¶int) –number of sinkhorn iterations
-
(n_splits¶int, default:1) –split-k factor;
Returns:
-
post_mix(Tensor) –shape (..., hc_mult), dtype torch.float32
-
comb_mix(Tensor) –shape (..., hc_mult, hc_mult), dtype torch.float32
-
layer_input(Tensor) –shape (..., hidden_size), dtype torch.bfloat16