vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.row_shard
¶
The DeepSeek-V4.1 indexer's prefill rows split over the TP ranks, on SM80 and SM90.
Under tensor parallelism every rank holds the whole indexer: it scores each prefill row of a step against the row's compressed keys and keeps the row's top-k. The rows are independent, so the ranks can split them instead: each rank scores a contiguous share, balanced by keys, and the ranks exchange the int32 top-k rows. vLLM's shared indexer no longer splits its rows (prefill is token-sharded in the sparse MLA instead, which DeepSeek-V4.1's attention layers do not use), so the SM80 and SM90 indexers keep the split here; their metadata builders decide when a step splits.
RowShardedPrefillMetadatais the indexer's prefill metadata with the ranks' shares (row_shard_sizes), whichbalanced_prefill_row_shardcomputes;PrefillRowShardnarrows a prefill chunk, or a whole step's chunks, to one rank's rows, and gathers the ranks' top-k rows back into step order;prefill_row_sharding_supportedsays where the split may run, andreserve_exchange_memoryreserves the exchange's transient memory in the memory profile, whose dummy runs never reach it.
Classes:
-
PrefillRowShard–One TP rank's contiguous share of a step's prefill rows.
-
RowShardedPrefillMetadata–The indexer's prefill metadata with the TP ranks' contiguous shares of
Functions:
-
balanced_prefill_row_shard–The
tp_sizeranks' contiguous shares of a step's prefill rows, -
prefill_row_sharding_supported–Whether a step's prefill rows may split over the TP ranks: the exchange
-
reserve_exchange_memory–Allocate the exchange's worst-case transient memory for the memory
PrefillRowShard
¶
One TP rank's contiguous share of a step's prefill rows.
The rank scores only its share, and the group then exchanges the final int32 top-k rows instead of the logits.
Methods:
-
exchange_topk–Gather every rank's top-k rows back into step order.
-
from_metadata–This rank's share of the step whose prefill metadata is
-
narrow–This rank's row window of
chunkand the window's causal key -
narrow_prefill–A copy of
prefillwhose chunks hold only this rank's rows. A
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
exchange_topk(topk_buffer, width)
¶
Gather every rank's top-k rows back into step order.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
from_metadata(prefill, num_decode_tokens)
classmethod
¶
This rank's share of the step whose prefill metadata is
prefill; None when the step does not split.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
narrow(chunk)
¶
This rank's row window of chunk and the window's causal key
bounds; None when the chunk holds none of its rows.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
narrow_prefill(prefill)
¶
A copy of prefill whose chunks hold only this rank's rows. A
chunk without any keeps an empty row window rather than going away:
it still gathers its keys, which a continuation chunk reuses.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
RowShardedPrefillMetadata
dataclass
¶
Bases: DeepseekV32IndexerPrefillMetadata
The indexer's prefill metadata with the TP ranks' contiguous shares of its rows; None when every rank scores every row.
Methods:
-
of–prefill, as built by the stock builder, with the shares
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
of(prefill, sizes)
classmethod
¶
prefill, as built by the stock builder, with the shares
sizes.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
balanced_prefill_row_shard(seq_lens_cpu, query_lens_cpu, compress_ratio, tp_size)
¶
The tp_size ranks' contiguous shares of a step's prefill rows,
balanced by their compressed keys, from the prefill requests' sequence
lengths and new tokens; None with fewer rows than ranks.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
prefill_row_sharding_supported(vllm_config, dcp_world_size, use_pcp, tp_size)
¶
Whether a step's prefill rows may split over the TP ranks: the exchange runs on PyNCCL outside FULL CUDA graphs, without context parallelism.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/row_shard.py
reserve_exchange_memory(topk_indices_buffer, device)
¶
Allocate the exchange's worst-case transient memory for the memory profile: this rank's staged rows and the gathered tensor, each bounded by the top-k buffer.