vllm.models.minimax_m3.nvidia.specialized_decode.drafter
¶
The DSpark drafter's sequential Markov sampling in K + 1 launches.
The stock loop (DSparkSpeculator._sample_markov_chain) runs, for each of
the K draft positions, about ten kernels: the Markov embedding lookup, the
[rows, r] x [r, V] bias GEMM, the bias add, two index copies, the Gumbel
block argmax (which also stores the draft logits cache column), the argmax
over the blocks, and two token copies, most of the time in small kernels
around one stream of W2 (V x r, BF16).
Here one kernel per position does all of it, for every request at once:
- its prologue reads the previous position's token from a packed argmax key
per request (position 0 reads the anchor token instead); program 0 also
stores that token into
draft_tokensand re-arms the key buffer the next position reduces into; - it gathers the Markov embedding
W1[prev]and, tile by tile over its vocabulary slice, computes the biasW1[prev] @ W2^T(fp32 accumulate, rounded to the dtype the stock GEMM returns), adds the base logits (rounded to the dtype the stock add returns), stores the draft logits cache column, and runs the Gumbel-max sample (the stock helpers, so the noise is bit-identical) or the plain argmax at temperature 0; - it folds its (max, leftmost index) per request into the key with one
atomic max: the key orders (value, -index), so the reduction returns the
leftmost global maximum, which is what the stock block argmax plus
torch.argmaxreturn.
A last one-program launch stores position K-1's token.
Only the bias GEMM's accumulation order differs from the stock path (it is
rounded to the same dtype), so the drafts are those of the stock sampler up to
that rounding. The base draft logits (draft LM head) stay stock, gathered
through peer memory (peer_logits.py) instead of NCCL when supported.
The sampler engages only on configurations it reproduces exactly (otherwise
the stock loop runs): probabilistic drafting with a draft logits cache, no
reduced draft vocabulary, watermarker, acceptance estimator or confidence
head, FP32 Gumbel sampling, an unquantized Markov head, at most
MARKOV_MAX_REQUESTS requests.
Classes:
-
FusedMarkovSampler–sample_sequentialof a MiniMax-M3 DSpark draft model under the -
GemmaRMSNormRows–forwardof the draft model's hidden-size Gemma RMSNorms -
ShardedContextFc–combine_hidden_stateswith the context projectionfcsplit over
Functions:
-
attach_drafter–Give a MiniMax-M3 DSpark draft model the fused Markov sampling when the
-
markov_keys–The step kernels' reduction buffer ([3, MARKOV_MAX_REQUESTS] int64 keys).
-
markov_sample–Sample every draft position; see the module docstring.
-
warm_up_drafters–Collective (every rank, at start-up): warm up every attached sampler.
FusedMarkovSampler
¶
sample_sequential of a MiniMax-M3 DSpark draft model under the
specialized decode (see DSparkSpeculator._sample_sequential).
Methods:
-
warm_up–Collective, at start-up (
warm_up_drafters): the draft LM head's
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 | |
_peer_logits(n_spec)
¶
The LM head through peer memory, or None for the NCCL gather.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
_supported(speculator)
¶
Static checks, the same on every call (capture and replay).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
warm_up(n_spec)
¶
Collective, at start-up (warm_up_drafters): the draft LM head's
peer logits, allocated and compiled before memory profiling.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
GemmaRMSNormRows
¶
forward of the draft model's hidden-size Gemma RMSNorms
(x * rsqrt(mean(x^2) + eps) * (1 + w), optionally fused with the
residual add, in place like the FlashInfer path): one program per row.
It replaces the FlashInfer CuTe-DSL fused add + norm for the 28-56 rows of a draft block (10 calls per draft step), which is slower at these rows.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
ShardedContextFc
¶
combine_hidden_states with the context projection fc split over
the TP ranks.
fc ([hidden, 5 x hidden], BF16, unquantized) is replicated: every rank
streams all of it (377 MB) for the 8 rows of a verify step. Here
each rank multiplies its 1/TP slice of the input features (strided views,
no copies) into FP32, the FP32 partials are all-reduced and rounded to the
model dtype once, as a split-K GEMM would. Larger (prefill) batches keep
the replicated GEMM.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
_argmax_key(value, index)
¶
(value, -index) packed into an int64 ordered like the pair: an atomic max over keys picks the leftmost maximum.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
attach_drafter(model, vllm_config)
¶
Give a MiniMax-M3 DSpark draft model the fused Markov sampling when the specialized decode is on (the caller checks the switch; the stock draft model is left as it is otherwise).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
markov_keys(device)
¶
The step kernels' reduction buffer ([3, MARKOV_MAX_REQUESTS] int64 keys).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
markov_sample(base_logits, anchor_ids, w1, w2, bias_dtype, logits_cache, row_mapping, idx_mapping, sample_pos, temperature, seeds, out_tokens, keys, num_programs, block_v=None, num_warps=None, num_stages=None, swap=None)
¶
Sample every draft position; see the module docstring.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/drafter.py
warm_up_drafters(n_spec)
¶
Collective (every rank, at start-up): warm up every attached sampler.