vllm.v1.attention.ops.mxfp4_mla.read
¶
Triton read-side unpack for the mxfp4_mla KV cache.
A row is packed data then scales, row_bytes(d) == d // 2 + d // 32.
Unpacks a gathered tile to bf16 once and hands it to the existing tl.dot
calls. On gfx950 tl.dot_scaled lowers to the same convert plus a bf16
MFMA, so it would only convert twice.
Functions:
-
load_mxfp4_rows–Gather BLOCK_K packed rows as a bf16 [BLOCK_K, LATENT] tile, using the
-
unpack_mxfp4_tile–Packed MXFP4 tile -> bf16
[BLOCK_K, LATENT]in latent order. -
unpack_mxfp4_tile_hw–Packed MXFP4 -> bf16 [BLOCK_K, LATENT], using the hardware converter.
_apply_e8m0_scale(values, encoded_scales)
¶
Multiply fp32 values by 2**(X - 127) via an integer exponent add.
Exact, because scaling by a power of two only shifts the exponent. Zero must be special-cased: adding to a zero's exponent field manufactures a denormal instead of leaving it zero.
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
_cvt_pair(packed_i32, ASM)
¶
One int32 word -> one int32 holding two packed bf16 (a nibble pair).
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
_load_mxfp4_rows(cache_ptr, slots, valid, row_pitch, BLOCK_K, LATENT, HW_UNPACK)
¶
Gather BLOCK_K packed rows as a bf16 [BLOCK_K, LATENT] tile.
cache_ptr is the uint8 cache, row_pitch its row size in bytes.
Invalid slots come back as zeros. Each row, with no padding::
| packed E2M1 codes | E8M0 scales |
|<----- LATENT // 2 ------>|<- LATENT // GROUP ->|
|<------- row_pitch == row_bytes(LATENT) -------->|
Code byte b holds values 2b (low nibble) and 2b + 1 (high
nibble); scale byte g covers values GROUP * g to
GROUP * g + GROUP - 1.
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
_split_pair(pair)
¶
int32 of two packed bf16 -> (low fp32, high fp32).
A bf16 occupies the top 16 bits of the matching fp32, so the low bf16 is recovered by shifting it up and the high one by masking in place.
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
load_mxfp4_rows(cache_ptr, slots, valid, row_pitch, BLOCK_K, LATENT)
¶
Gather BLOCK_K packed rows as a bf16 [BLOCK_K, LATENT] tile, using the gfx950 hardware unpack where available.
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
unpack_mxfp4_tile(packed, encoded_scales, BLOCK_K, LATENT, GROUP)
¶
Packed MXFP4 tile -> bf16 [BLOCK_K, LATENT] in latent order.
The low nibble holds the lower element index, so the two nibble streams are interleaved back into value order rather than concatenated.
Source code in vllm/v1/attention/ops/mxfp4_mla/read.py
unpack_mxfp4_tile_hw(packed_i32, encoded_scales, BLOCK_K, LATENT, GROUP)
¶
Packed MXFP4 -> bf16 [BLOCK_K, LATENT], using the hardware converter.
Four instructions cover all eight nibbles of each int32 word, in natural
order: value j of word w is latent dim 8w + j.