vllm.models.mistral.nvidia.ops.ssmax
¶
SSMax query scaling q *= s * log(pos + 1) + b, alone or fused with RoPE.
bf16 only. Every kernel reproduces the unfused rounding: the scale is computed in bf16 (product and sum each rounded), and RoPE output is rounded to bf16 before it is scaled.
Functions:
-
fused_rope_ssmax_scaling_–In place: RoPE on
q[..., nope_dim:]andk, then SSMax onq. -
fused_ssmax_scaling–Return
F.pad(q, (0, out_head_dim - D)) * (log_positions * s + b).
_round_bf16(x)
¶
Round fp32 to the nearest-even bf16 value, kept in fp32.
Done on the bits: Triton may narrow fp32 math followed by a bf16 cast into native bf16 instructions, which do not match torch on ties.
Source code in vllm/models/mistral/nvidia/ops/ssmax.py
_ssmax_scale(log_pos_ptr, s_ptr, b_ptr, token, head, mask)
¶
bf16 log_pos * s + b (each op rounded), returned in fp32.
Inputs may be fp32 or bf16 (SSMax parameters load in the model dtype).
Source code in vllm/models/mistral/nvidia/ops/ssmax.py
fused_rope_ssmax_scaling_(q, k, positions, cos_sin_cache, log_positions, s, b, nope_dim, is_neox_style)
¶
In place: RoPE on q[..., nope_dim:] and k, then SSMax on q.
q is [T, H, nope_dim + rope_dim], k is [T, 1, rope_dim]
(the shared MLA rope key); token strides are free.
Source code in vllm/models/mistral/nvidia/ops/ssmax.py
fused_ssmax_scaling(q, log_positions, s, b, out_head_dim=None)
¶
Return F.pad(q, (0, out_head_dim - D)) * (log_positions * s + b).
q is [T, H, D] with contiguous head dims; the padding (NoPE layers'
zero rope dims) is fused into the same launch.