vllm.models.minimax_m3.nvidia.specialized_decode.kernels.post.kernel
¶
CuTe DSL kernels of the post family (the attention -> MoE boundary of a sparse layer).
Every arithmetic op is spelled as an explicit PTX instruction (ftz / rn / approx variants, no contraction), so the outputs do not depend on compiler contraction. Cross-rank protocols: the prepushed two-parity exchange with the 0x80000000 empty sentinel, tagged rx router partials, tagged route lines, PDL trigger points.
Kernels:
post_route_kernel(allreduce_norm stage): one cluster of CL CTAs per row. Row warps land the W ranks' bf16 partials from this rank's exchange (poll + re-arm), add the residual, Gemma RMSNorm via a cluster st.async exchange of the row's sum of squares (and the |normed| argmax candidate), then the mode-Q per-token NVFP4 tail (and optional MXFP8). One extra routing warp (crank 0) polls the o_proj router CTAs' tagged partial logits, scales them by 1/rms and publishes the MiniMax2 top-4 as tagged route lines (optionally the fp32 logits).mxfp8_rows_kernel: MXFP8 (UE8M0, 128x4-swizzled scales) of bf16 rows.
All buffers are passed as raw 64-bit device addresses (the host checks shapes, dtypes and alignment).
Functions:
-
better–(h1, l1) > (h0, l0) lexicographically, uint32 -> Boolean.
-
bf16_words_to_f32–4 words (8 bf16) -> 8 fp32 (low half first).
-
bf16r–__bfloat162float(__float2bfloat16_rn(a))
-
bf16x2–__floats2bfloat162_rn(a, b) as a word (a in the low half).
-
ctz–__ffs(x) - 1 for x != 0.
-
e4m3x4–Four fp32 to E4M3 (RN, satfinite), v0 in the low byte (two __nv_fp8x2_e4m3 halves).
-
fimm–An fp32 constant with exactly these bits (kept out of the optimizer's constant folding).
-
gemma_w–(1 + w) of the gamma pair q (fp32, ftz add).
-
hi_f–The high bf16 of a word as fp32.
-
ld_nc_v4–Read-only 16 bytes (ld.global.nc), ordered where written.
-
ld_v4–16 bytes (zero when pred == 0), ordered where written (the post's residual may alias its residual_out).
-
lo_f–The low bf16 of a word as fp32.
-
mbar_wait_cluster0–Spin until phase 0 of the CTA's barrier completes (acquire, cluster scope).
-
mxfp8_8–lb::mxfp8_8: 4 lanes per 32-column block, warp-synchronous (stores predicated on active).
-
route_pub–route_pub: top-4 of the hoisted logits, then lane l stores words 2 (l & 1), +1 of replica l >> 1.
-
rx_ld–rx_ld: the lane's two 16-byte loads (experts 4 lane .. +3) of every rx block (unconditional).
-
rx_poll_sum–rx_poll_sum: poll the lane's rx words (exact tag match, globaltimer deadline), then S_e in rank order.
-
rx_take–rx_take: accept blocks whose four words carry tag (returns the new pending mask).
-
tanh_engine–Tanh as an explicit instruction sequence, so the routing keys are reproducible bit for bit (cute.math.tanh
-
twiddle–Order-preserving uint32 key of an fp32 (as Int32 bits; compare with ugt).
-
ue8m0_rp–__nv_cvt_float_to_e8m0(x, __NV_SATFINITE, cudaRoundPosInf)
-
ue8m0_to_f32–float(__nv_fp8_e8m0): 2^(c - 127); c = 0 -> 2^-127 (0x00400000), c = 255 -> NaN.
-
ugt–A > b as uint32 -> Int32 0 / 1.
-
x4_quant8–Shared tail of the NVFP4 16-column block quantizers: codes of xn * inv, sf byte from even lanes (stores
_b(cond)
¶
_cex(a, b)
¶
Compare-exchange on (hi, lo, sg) triples: a gets the larger.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
_poll_exchange(world)
¶
The post's exchange poll: re-load every rank's 16 bytes at p0 + r * 12288 that still holds the empty sentinel in any word, until none does (bounded spin, then trap). Returns the 4 * world words.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
better(h1, l1, h0, l0)
¶
(h1, l1) > (h0, l0) lexicographically, uint32 -> Boolean.
bf16_words_to_f32(w)
¶
4 words (8 bf16) -> 8 fp32 (low half first).
bf16r(a, *, loc=None, ip=None)
¶
__bfloat162float(__float2bfloat16_rn(a))
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
bf16x2(a, b, *, loc=None, ip=None)
¶
__floats2bfloat162_rn(a, b) as a word (a in the low half).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ctz(x, *, loc=None, ip=None)
¶
__ffs(x) - 1 for x != 0.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None)
¶
Four fp32 to E4M3 (RN, satfinite), v0 in the low byte (two __nv_fp8x2_e4m3 halves).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
fimm(bits, *, loc=None, ip=None)
¶
An fp32 constant with exactly these bits (kept out of the optimizer's constant folding).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
gemma_w(gw, q)
¶
(1 + w) of the gamma pair q (fp32, ftz add).
hi_f(w, *, loc=None, ip=None)
¶
The high bf16 of a word as fp32.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ld_nc_v4(addr, *, loc=None, ip=None)
¶
Read-only 16 bytes (ld.global.nc), ordered where written.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ld_v4(addr, pred, *, loc=None, ip=None)
¶
16 bytes (zero when pred == 0), ordered where written (the post's residual may alias its residual_out).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
lo_f(w, *, loc=None, ip=None)
¶
The low bf16 of a word as fp32.
mbar_wait_cluster0(bar, *, loc=None, ip=None)
¶
Spin until phase 0 of the CTA's barrier completes (acquire, cluster scope).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
mxfp8_8(s, active, t, c0, out_q, out_sf, tid)
¶
lb::mxfp8_8: 4 lanes per 32-column block, warp-synchronous (stores predicated on active).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
route_pub(route, repl, rmaxt, t, lane, tag, inv, S, bb)
¶
route_pub: top-4 of the hoisted logits, then lane l stores words 2 (l & 1), +1 of replica l >> 1.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 | |
rx_ld(row, lane, nb, buf)
¶
rx_ld: the lane's two 16-byte loads (experts 4 lane .. +3) of every rx block (unconditional).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
rx_poll_sum(row, lane, tag, deadline, nb, rx2, S)
¶
rx_poll_sum: poll the lane's rx words (exact tag match, globaltimer deadline), then S_e in rank order. Returns (late, stale).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 | |
rx_take(tag, nb, buf, pending, acc)
¶
rx_take: accept blocks whose four words carry tag (returns the new pending mask).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
tanh_engine(x, *, loc=None, ip=None)
¶
Tanh as an explicit instruction sequence, so the routing keys are reproducible bit for bit (cute.math.tanh lowers to a sequence with different bits).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
twiddle(x, *, loc=None, ip=None)
¶
Order-preserving uint32 key of an fp32 (as Int32 bits; compare with ugt).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ue8m0_rp(x, *, loc=None, ip=None)
¶
__nv_cvt_float_to_e8m0(x, __NV_SATFINITE, cudaRoundPosInf)
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ue8m0_to_f32(c, *, loc=None, ip=None)
¶
float(__nv_fp8_e8m0): 2^(c - 127); c = 0 -> 2^-127 (0x00400000), c = 255 -> NaN.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
ugt(a, b, *, loc=None, ip=None)
¶
A > b as uint32 -> Int32 0 / 1.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
x4_quant8(xn, active, t, c0, inv_of, xq, xsf, tid, sfb)
¶
Shared tail of the NVFP4 16-column block quantizers: codes of xn * inv, sf byte from even lanes (stores predicated on active).