vllm.v1.attention.ops.rocm_aiter_mla_sparse
¶
Functions:
-
build_prefill_topk_ragged_indices–Map prefill top-k rows to a ragged stream of compressed-cache slots.
-
fp8_mqa_logits_torch–Compute FP8 MQA logits for a single sequence without KV paging.
-
indexer_k_cache_is_shuffled–Value layout of the ROCm FP8 indexer-K cache (ratio 1 / 2): 16x16 SHUFFLE
-
rocm_fp8_mqa_logits–Compute FP8 MQA logits for a single sequence without KV paging.
-
rocm_fp8_paged_mqa_logits–Compute FP8 MQA logits using paged KV-cache.
-
rocm_fp8_paged_mqa_logits_triton–Triton paged MQA-logits for decode and MTP; matches the torch ref but
-
rocm_inv_rope_einsum–Inverse-RoPE + WO_A bmm path used on ROCm.
-
rocm_inverse_rope_mxfp8_rows–Inverse-RoPE bf16 attention rows and MXFP8-quantize them for wo_a.
-
rocm_inverse_rope_rows_–Inverse-RoPE attention output rows in place.
-
rocm_mxfp8_wo_a_bmm–Grouped MXFP8 wo_a:
out[t, g, :] = a[t, g, :] @ W[g].T, bf16 out. -
rocm_sparse_attn_decode–Run sparse MLA decode into
output. -
rocm_sparse_attn_decode_bf16–Run split-K sparse attention over decode rows using an unquantized KV cache.
-
rocm_sparse_decode_bf16_num_splits–Number or kv splits in splitK for the sparse bf16 decode, or 1 for single-pass.
_aiter_wo_a_bmm(a, a_scale, weight_preshuffled, weight_scale, n_groups, rank)
¶
wo_a on AITER's tuned batched GEMMs and the preshuffled weight; [T, G * R].
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_apply_candidate_mask_strided(logits, row_ks, row_ke, candidate_blocks, block_size, row_repeat=1)
¶
ROCm decode variant of apply_candidate_mask.
Same masking semantics over [0, end), but the grid is sized by a fixed
program count rather than by the logits width. Only worth using where the
width is the max_model_len workspace and the live context is far
shorter, i.e. the paged decode path below; the prefill chunks pass
chunk-sized logits and stay on the shared kernel.
On gfx942 and gfx950 the candidate flags are built by the masking programs themselves, each over only its own live tiles, so the whole mask is a single launch and the flags past each row's end are never cleared.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 | |
_can_use_aiter_pa_prefill_sparse(q, kv)
¶
Cheap gate. The AITER import runs only after this passes.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_decode_num_splits(num_queries, heads_blocks, avg_main_len=0.0, avg_extra_len=0.0, block_k=32)
¶
Pick a flash-decode split count to keep the GPU busy across batch sizes.
Decode launches only num_queries * heads_blocks workgroups otherwise,
which severely under-fills the device for the low-concurrency regime that
dominates latency. Splitting the KV sequence adds parallelism.
We model the relative partial-kernel latency for a given split count s
as waves * (1/s + mu) where waves = ceil(base * s / CU) and mu
is a small per-wave overhead penalty:
waves / scaptures the partial compute: each wave walks roughlytotal_tokens / stokens and there arewavesof them, so dividing bysmakes more splits cheaper until they spill into extra waves.mu * wavescharges per-wave launch/tail overhead so we do not over-split into many mostly-idle waves (e.g. batch 224 on 256 CUs is best left at 1 split rather than 8 splits across 7 waves).
The minimiser naturally prefers split counts that pack the device into full
waves (base * s near a multiple of CU) and falls back to 1 split
once the batch already fills the device. Ties favour the smaller split
count (less reduce work).
Finally we "snap down" the chosen split count to the smallest value that yields the same wave count and the same per-workgroup BLOCK_K iteration count. Because latency tracks iteration count (not raw token count), extra splits that do not lower the iteration count add only reduce/HBM overhead for no parallelism gain (e.g. batch 24: s8 and s10 both walk 4 extra iters in one wave, so s8 is strictly better). Snapping needs the average segment lengths, which the caller derives sync-free from the ragged index sizes.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 | |
_decode_partial_iters(avg_main_len, avg_extra_len, splits, block_k)
¶
BLOCK_K iterations one partial workgroup walks for splits splits.
Each split processes ceil(seg_len / splits) tokens of a segment, walked
BLOCK_K at a time, and the main/extra segments are handled separately.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_fused_inverse_rope_gptj(o, positions, cos_sin_cache, rope_head_dim, out=None)
¶
bf16 inverse GPT-J RoPE via a single fused Triton kernel.
out may alias o: the rotation is a per-row bijection whose kernel
reads both lanes of a pair before storing either.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_get_cached_wo_a_bf16(wo_a, n_local_groups, o_lora_rank, hidden_dim)
¶
Dequantize wo_a to bf16 once and cache it on the module.
wo_a weights are static, so the fp8 -> fp32 -> (* block scale) -> bf16
dequant only needs to run once. Recomputing it every decode step shows up
in the profile as the largest copy/mul kernels (direct_copy float ~55us
and MulFunctor float ~31us per two layers). SGLang / ATOM keep wo_a in
bf16 and feed a plain bf16 GEMM; this mirrors that.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_indexer_k_is_c4a_block_flat(compress_ratio)
¶
V4.0 C4A is block-flat (NORMAL). Ratio 1 and 2 are 16×16 SHUFFLE.
_inverse_rope_gptj_kernel(o_ptr, out_ptr, pos_ptr, cos_sin_ptr, s_t, s_h, os_t, os_h, cs_stride, NOPE, HALF, BLOCK_NOPE, BLOCK_HALF)
¶
Fused inverse GPT-J RoPE on the trailing rope_dim of each (token, head).
Mirrors DeepseekV4ScalingRotaryEmbedding.forward_native(inverse=True)
for the GPT-J (non-neox) layout, writing bf16 directly. Replaces the
clone + index_select + repeat_interleave + neg + stack + cat + cast chain
(~10 small kernels) with a single launch.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_max_decode_logits_rows(num_batched_tokens)
¶
Upper bound on decode rows the paged-MQA logits buffer can ever hold.
rocm_fp8_paged_mqa_logits sizes its workspace as
(batch_size * next_n, max_model_len). batch_size is bounded by
max_num_seqs and next_n by 1 + num_speculative_tokens, which is
far tighter than max_num_batched_tokens -- 192 vs 16384 for a typical
32-seq DSpark-5 deployment. The loose bound is harmless at short contexts
but scales with max_model_len, so at the model's full context it asks
for tens of TiB and the engine cannot start. Take whichever valid bound is
smaller; the workspace is locked after profiling, so it must not be under-
estimated.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_mxfp8_quantize_rows(x, ROWS, COLS)
¶
MXFP8-quantize x [ROWS, COLS] in registers, one scale per 32 lanes.
Returns the rescaled fp32 values (to be cast to e4m3 on store) and the [ROWS, COLS // 32] biased E8M0 exponents.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_mxfp8_scale_bits(amax)
¶
Biased E8M0 exponent that puts amax at the top of the e4m3 range.
Same rounding as mxfp8_e4m3_quantize, so the output is bit-identical to
quantizing the tensor there.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_mxfp8_wo_a_bmm_config(num_tokens, n_groups)
¶
(BLOCK_M, BLOCK_N, BLOCK_K, num_warps, num_stages) for gfx950.
Tuned under HIP graphs with a cold weight at G = 4 and 2, over every decode shape of conc 1-128 x 0-5 spec tokens plus prefill chunks up to 8K tokens. The best tile tracks the total work T * G, so the tiers are keyed on it.
This will be replaced after new GEMM kernel from AITER with proper 32x32 scale shape GEMM fp8 enabled.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_rocm_sparse_attn_decode_ragged_bf16_triton(q, kv, indices, indptr, scale, attn_sink, nope_head_dim, rope_head_dim, num_splits, out=None, is_mxfp4=False)
¶
Split-K decode over an bf16 ragged KV cache.
Partitions each query's selected tokens across different workgroups and combines the partials through reduction.
Parameters:
-
(q¶Tensor) –Queries laid out as
[sq, h, d]. -
(kv¶Tensor) –Unquantized KV rows laid out as
[skv, d]. -
(indices¶Tensor) –Flattened per-query KV slots.
-
(indptr¶Tensor) –Segment offsets into
indices,[sq + 1]. -
(scale¶float) –Softmax scale.
-
(attn_sink¶Tensor | None) –Optional per-head sink logits.
-
(nope_head_dim¶int) –NoPE width of
d. -
(rope_head_dim¶int) –RoPE width of
d. -
(num_splits¶int) –Number of KV splits per query.
-
(out¶Tensor | None, default:None) –Optional destination with
dtrailing elements. -
(is_mxfp4¶bool, default:False) –kvholds packedmxfp4_mlarows instead of bf16.
Returns:
-
Tensor–The attention output,
outwhen provided.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 | |
_rocm_sparse_attn_decode_ragged_triton(q, main_cache, main_indices, main_indptr, scale, attn_sink, nope_head_dim, rope_head_dim, extra_cache=None, extra_indices=None, extra_indptr=None, out=None, extra_cache_nan_free=False, adaptive_splits=False, inv_rope_positions=None, inv_rope_cos_sin_cache=None, out_mxfp8=None)
¶
Split-K sparse decode; returns the attention output.
With out_mxfp8 = (data, scale) the reduce writes MXFP8 instead of
bf16: data is [b, h * d] e4m3 and scale [b, h * d // 32] E8M0, and
data viewed as [b, h, d] is returned.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 4656 4657 4658 4659 4660 4661 4662 4663 4664 4665 4666 4667 4668 4669 4670 4671 4672 4673 4674 4675 4676 4677 4678 4679 4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4695 4696 4697 4698 4699 4700 4701 4702 4703 4704 4705 4706 4707 4708 4709 4710 4711 4712 4713 4714 4715 4716 4717 4718 4719 4720 4721 4722 4723 4724 4725 4726 4727 4728 4729 4730 4731 4732 4733 4734 4735 4736 4737 4738 4739 4740 4741 4742 4743 4744 4745 4746 4747 4748 4749 4750 4751 4752 4753 | |
_rocm_sparse_attn_prefill_aiter(pa_prefill_sparse, q, kv, indices, indptr, scale, attn_sink, output)
¶
gfx942 path. One KV pool; the extend sources stay unset.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
_unshuffle_indexer_k_values(values, block_size, dim)
¶
[..., block_size * dim] FP8 bytes in 16x16 SHUFFLE tile order
([blk/16, dim/16, 16, 16]) -> flat rows ([blk, dim]).
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
build_prefill_topk_ragged_indices(topk_indices, token_to_req_indices, query_start_loc, seq_lens, is_valid_token, block_table, block_size, compress_ratio, num_compressed, token_offset, num_rows=-1)
¶
Map prefill top-k rows to a ragged stream of compressed-cache slots.
topk_indices holds local compressed positions for the prefill tokens,
which sit at token_offset in the batch; token_to_req_indices,
query_start_loc, seq_lens and block_table are batch-wide.
block_size is the compressed cache's, i.e. already divided by the ratio.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
fp8_mqa_logits_torch(q, kv, weights, cu_seqlen_ks, cu_seqlen_ke)
¶
Compute FP8 MQA logits for a single sequence without KV paging.
Parameters:
-
(q¶Tensor) –Query tensor of shape [M, H, D]. Casted to
torch.float8_e4m3fnby caller. -
(kv¶tuple[Tensor, Tensor]) –Tuple
(k_fp8, k_scales)wherek_fp8has shape [N, D] with dtypetorch.float8_e4m3fnandk_scaleshas shape [N] (or [N, 1]) with dtypetorch.float32. -
(weights¶Tensor) –weights of shape [M, H], dtype
torch.float32. -
(cu_seqlen_ks¶Tensor) –Start indices (inclusive) for valid K per query position, shape [M], dtype int32.
-
(cu_seqlen_ke¶Tensor) –End indices (exclusive) for valid K per query position, shape [M], dtype int32.
Returns:
-
Tensor–Logits tensor of shape [M, N], dtype
torch.float32.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
indexer_k_cache_is_shuffled(block_size)
¶
Value layout of the ROCm FP8 indexer-K cache (ratio 1 / 2): 16x16 SHUFFLE tiles when a block holds more than one token, flat rows otherwise.
The one rule shared by the insert, the prefill gather and the paged-logits readers (and by fused writers, through their cache object), so a writer and its readers cannot disagree on the layout.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_fp8_mqa_logits(q, kv, weights, cu_seqlen_ks, cu_seqlen_ke)
¶
Compute FP8 MQA logits for a single sequence without KV paging.
Parameters:
-
(q¶Tensor) –Query tensor of shape [M, H, D]. Casted to
torch.float8_e4m3fnby caller. -
(kv¶tuple[Tensor, Tensor]) –Tuple
(k_fp8, k_scales)wherek_fp8has shape [N, D] with dtypetorch.float8_e4m3fnandk_scaleshas shape [N] (or [N, 1]) with dtypetorch.float32. -
(weights¶Tensor) –weights of shape [M, H], dtype
torch.float32. -
(cu_seqlen_ks¶Tensor) –Start indices (inclusive) for valid K per query position, shape [M], dtype int32.
-
(cu_seqlen_ke¶Tensor) –End indices (exclusive) for valid K per query position, shape [M], dtype int32.
Returns:
-
Tensor–Logits tensor of shape [M, N], dtype
torch.float32.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_fp8_paged_mqa_logits(q_fp8, kv_cache_fp8, weights, context_lens, block_tables, schedule_metadata, max_model_len, *, compress_ratio=1, cache_layout=None)
¶
Compute FP8 MQA logits using paged KV-cache.
Parameters:
-
(q_fp8¶Tensor) –Query tensor of shape [B, next_n, H, D]. Casted to
torch.float8_e4m3fnby caller. -
(kv_cache_fp8¶Tensor) –Paged KV-cache in packed FP8+scale layout with shape [num_blocks, block_size, 1, D+4], dtype
torch.uint8. -
(weights¶Tensor) –Tensor of shape [B * next_n, H], dtype
torch.float32. -
(context_lens¶Tensor) –Tensor of shape [B] or [B, next_n], dtype int32; effective context length for each batch element, or for each of its Q rows.
-
(block_tables¶Tensor) –Tensor of shape [B, max_blocks], dtype int32; maps logical block indices to physical blocks in the paged cache.
-
(schedule_metadata¶Tensor) –Returned by
get_paged_mqa_logits_metadata; used to distribute work across SMs. -
(max_model_len¶int) –Maximum sequence length used to size the logits output.
-
(compress_ratio¶int, default:1) –C4A (4) takes block-flat Triton; 1 and 2 stay on AITER.
-
(cache_layout¶str | None, default:None) –Explicit cache layout override; "NORMAL" (block-flat) also takes the Triton path.
Returns:
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 | |
rocm_fp8_paged_mqa_logits_triton(q_fp8, kv_cache_fp8, weights, context_lens, block_tables, max_model_len)
¶
Triton paged MQA-logits for decode and MTP; matches the torch ref but has no host sync, so it is safe to capture under a full CUDA graph.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_inv_rope_einsum(rotary_emb, o, positions, rope_head_dim, n_local_groups, o_lora_rank, wo_a, inverse_rope=True)
¶
Inverse-RoPE + WO_A bmm path used on ROCm.
Fuses the inverse GPT-J RoPE into one Triton kernel and caches the bf16
wo_a weight so the per-step dequant disappears. Callers whose attention
already rotated every row pass inverse_rope=False; that is a property
of the attention backend, not of the batch, so it stays constant across
steps and is safe to read from compiled code.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_inverse_rope_mxfp8_rows(o, positions, cos_sin_cache, rope_head_dim, out_data, out_scale)
¶
Inverse-RoPE bf16 attention rows and MXFP8-quantize them for wo_a.
The counterpart of rocm_inverse_rope_rows_ for layers whose attention
output is MXFP8: rows the decode reduce did not emit (prefill) go through
here. o is [T, H, D]; out_data [T, H * D] e4m3 and out_scale
[T, H * D // 32] E8M0, the layout the reduce epilogue writes.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_inverse_rope_rows_(o, positions, cos_sin_cache, rope_head_dim)
¶
Inverse-RoPE attention output rows in place.
For rows no attention kernel rotated in its epilogue. Call it from the eager attention segment: which rows still owe a rotation depends on the prefill/decode split, and the o_proj that used to do this runs inside the compiled region, where a batch-dependent Python value would be frozen at trace time.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_mxfp8_wo_a_bmm(a, a_scale, wo_a, n_groups, o_lora_rank)
¶
Grouped MXFP8 wo_a: out[t, g, :] = a[t, g, :] @ W[g].T, bf16 out.
a is the [T, G * K] e4m3 attention output and a_scale its
[T, G * K // 32] E8M0 scales, as the sparse decode reduce writes them.
The weight is the checkpoint's MXFP8 wo_a as loaded, [G * R, K] with
either [G * R // 32, K // 32] block scales or [G * R, K // 32] per-row
scales, so there is no dequantized copy to keep. A weight_preshuffled
the linear kernel set (possibly the weight itself) runs AITER's GEMMs.
Returns [T, G * R].
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
rocm_sparse_attn_decode(q, kv_cache, swa_k_cache, swa_only, topk_indices, topk_lens, swa_indices, swa_lens, swa_ragged_indices, swa_ragged_indptr, topk_ragged_indices, topk_ragged_indptr, attn_sink, scale, head_dim, nope_head_dim, rope_head_dim, output, extra_cache_nan_free=False, adaptive_splits=False, inv_rope_positions=None, inv_rope_cos_sin_cache=None, output_mxfp8=None)
¶
Run sparse MLA decode into output.
Passing inv_rope_positions folds the inverse RoPE into the reduce
epilogue. Returns how many leading rows of output came back rotated,
so a caller mixing in a decode path that does not fuse still knows what it
owes the standalone pass. Read it from the eager attention segment only.
output_mxfp8 = (data, scale) replaces output: the reduce also
MXFP8-quantizes the rotated rows for the FP8 wo_a (see
_rocm_sparse_attn_decode_ragged_triton). It needs gfx950 and the fused
inverse RoPE, and always covers every row.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
4997 4998 4999 5000 5001 5002 5003 5004 5005 5006 5007 5008 5009 5010 5011 5012 5013 5014 5015 5016 5017 5018 5019 5020 5021 5022 5023 5024 5025 5026 5027 5028 5029 5030 5031 5032 5033 5034 5035 5036 5037 5038 5039 5040 5041 5042 5043 5044 5045 5046 5047 5048 5049 5050 5051 5052 5053 5054 5055 5056 5057 5058 5059 5060 5061 5062 5063 5064 5065 5066 5067 5068 5069 5070 5071 5072 5073 5074 5075 5076 5077 5078 5079 5080 5081 5082 5083 5084 5085 5086 5087 5088 5089 5090 5091 5092 5093 5094 5095 5096 | |
rocm_sparse_attn_decode_bf16(q, kv, scale, head_dim, nope_head_dim, rope_head_dim, attn_sink, output, ragged_indices, ragged_indptr, num_splits, kv_cache_dtype='auto')
¶
Run split-K sparse attention over decode rows using an unquantized KV cache.
Parameters:
-
(q¶Tensor) –Decode queries laid out as
[sq, h, d]. -
(kv¶Tensor) –KV cache laid out as
[skv, 1, d]. -
(scale¶float) –Softmax scale.
-
(head_dim¶int) –Post-absorption head width.
-
(nope_head_dim¶int) –NoPE width of
head_dim. -
(rope_head_dim¶int) –RoPE width of
head_dim. -
(attn_sink¶Tensor | None) –Optional per-head sink logits.
-
(output¶Tensor) –Destination, written in place.
-
(ragged_indices¶Tensor) –Flattened per-query KV slots.
-
(ragged_indptr¶Tensor) –Segment offsets into
ragged_indices,[sq + 1]. -
(num_splits¶int) –KV splits per query, from
rocm_sparse_decode_bf16_num_splits. -
(kv_cache_dtype¶str, default:'auto') –"mxfp4_mla"readskvas packed rows.
Source code in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
4928 4929 4930 4931 4932 4933 4934 4935 4936 4937 4938 4939 4940 4941 4942 4943 4944 4945 4946 4947 4948 4949 4950 4951 4952 4953 4954 4955 4956 4957 4958 4959 4960 4961 4962 4963 4964 4965 4966 4967 4968 4969 4970 4971 4972 4973 4974 4975 4976 4977 4978 4979 4980 4981 4982 4983 4984 4985 4986 4987 4988 4989 4990 4991 4992 4993 4994 | |
rocm_sparse_decode_bf16_num_splits(num_queries, num_heads, sparse_len)
¶
Number or kv splits in splitK for the sparse bf16 decode, or 1 for single-pass.
Parameters:
-
(num_queries¶int) –Decode rows in the batch.
-
(num_heads¶int) –Query heads per row.
-
(sparse_len¶int) –Longest selected KV run any decode row can walk.
Returns:
-
int–The split count, or 1 when the caller should use the single-pass kernel.