vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.columns
¶
The SM80 and SM90 TP all-gathers along the last dimension: logits and Engram.
Some of a decode step's tensors are split by columns across the TP ranks and all-gathered side by side, every rank's columns of a row next to each other:
- the vocabulary-parallel logits of the target and the DSpark draft model: each rank's LM head computes every row over its slice of the vocabulary (16,160 BF16 values at TP8) and sampling needs all of it;
- the two Engram layers' embedded hash heads (split by heads) and their
wkvoutput (split by output columns).
vLLM gathers them with NCCL's ring into rank-major blocks and then copies the
blocks side by side: at 96 rows on 8x A100 about 170 + 60 us for a step's
logits and 35 + 70 us (with their copies) for an Engram layer. Here they go
through the Lamport all-gather of all_gather.py laid out side by side,
which writes the gathered rows in place. The kernel uses SM80 instructions
only, so H100 (SM90) takes it too.
init_column_gathers builds one AmpereAllGather per row size in use
(a mailbox each), collectively, where supports_column_gathers holds.
all_gather_columns picks the one for a tensor's row size and falls back to
the TP group's all-gather for shapes it does not take (more than
MAX_ROWS rows, another row size, or before init_column_gathers).
Every rank makes the same calls, so all take the same path.
Classes:
-
AmpereLogitsProcessor–LogitsProcessorgathering the vocabulary shards with
Functions:
-
all_gather_columns–The TP all-gather of a 2-D
xalong its last dimension, `[rows, -
column_gathers_ready–Whether
init_column_gathershas built any gather: the target model -
init_column_gathers–Build the side-by-side all-gathers for rows of these sizes, up to
-
supports_column_gathers–Whether the model gathers its logits and Engram columns with the
AmpereLogitsProcessor
¶
Bases: LogitsProcessor
LogitsProcessor gathering the vocabulary shards with
all_gather_columns.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/columns.py
_gather_logits(logits)
¶
The logits over the whole vocabulary, on every rank.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/columns.py
all_gather_columns(x)
¶
The TP all-gather of a 2-D x along its last dimension, [rows,
world * columns].
Parameters:
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/columns.py
column_gathers_ready()
¶
Whether init_column_gathers has built any gather: the target model
builds them before its DSpark draft model, whose head then shares the
logits gather (a row size without a gather keeps the TP group's).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/columns.py
init_column_gathers(row_bytes)
¶
Build the side-by-side all-gathers for rows of these sizes, up to
MAX_ROWS rows a call; sizes that are not whole 16-byte fragments
keep the TP group's all-gather. Collective over the TP group: every rank
calls it with the same sizes.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/columns.py
supports_column_gathers(vllm_config)
¶
Whether the model gathers its logits and Engram columns with the Lamport kernel: on SM80 and SM90 at TP 2, 4 or 8 where vLLM's custom all-reduce runs, so the GPUs are fully connected with peer access. Collective decisions follow from it, so it holds on every rank or none.