vllm.model_executor.layers.quantization.online.lm_head
¶
Online lm_head quantization: weight-only methods for the model's own
lm_head, selected by a quantization_config.targets pattern naming it, and
private quantized copies of an already-loaded lm_head for speculative-decoding
drafters (draft_lm_head_quantization).
Classes:
-
OnlineLMHeadMethodBase–Weight-only lm_head quantized at load time and run on the platform's
Functions:
-
quantized_lm_head_copy–A ParallelLMHead holding this rank's shard of
lm_head, quantized. -
refresh_quantized_lm_heads–Re-derive the quantized lm_head copies in
modelfrom their source heads. -
restore_quantized_lm_heads–Undo
share_source_lm_heads, then re-derive the copies inmodel. -
share_source_lm_heads–Replace each quantized copy in
modelwith its source head.
OnlineLMHeadMethodBase
¶
Bases: OnlineLinearBase
Weight-only lm_head quantized at load time and run on the platform's
mixed-precision linear kernel (choose_mp_linear_kernel).
A subclass sets the weight format and implements quantize. An untied
head loads its fp16/bf16 weight like any online linear layer and replaces
it with the quantized copy. A tied head shares the input embeddings'
weight, which stays unchanged: the quantized copy is added next to it.
Methods:
-
quantize–Quantize
weight[N, K] into theMPLinearKernellayout.
Attributes:
-
group_size(int) –Input channels per scale; -1 for one scale per output channel.
-
k_align(int) –Required divisor of the hidden size (input columns).
-
n_align(int) –Required divisor of the vocab shard (output rows).
-
scheme(str) –The weight name in
QUANT_KEY_NAMES, for logs and errors.
Source code in vllm/model_executor/layers/quantization/online/lm_head.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | |
group_size
class-attribute
¶
Input channels per scale; -1 for one scale per output channel.
k_align = 1
class-attribute
¶
Required divisor of the hidden size (input columns).
n_align = 1
class-attribute
¶
Required divisor of the vocab shard (output rows).
scheme
class-attribute
¶
The weight name in QUANT_KEY_NAMES, for logs and errors.
quantize(weight)
abstractmethod
¶
Quantize weight [N, K] into the MPLinearKernel layout.
Returns the weight packed along K, the scales [N, K // group_size] ([N, 1] per channel) and the zero points packed along N, or None without zero points.
Source code in vllm/model_executor/layers/quantization/online/lm_head.py
quantized_lm_head_copy(lm_head, quantization)
¶
A ParallelLMHead holding this rank's shard of lm_head, quantized.
"fp8" uses per-row weight and per-token activation scales on the platform's
FP8 linear kernel; "nvfp4" is weight-only on the
platform's W4A16 kernel. lm_head is left
unchanged and no bf16 copy of its weight is made. The copy is derived state:
refresh_quantized_lm_heads re-derives it after lm_head changes.
Source code in vllm/model_executor/layers/quantization/online/lm_head.py
refresh_quantized_lm_heads(model)
¶
Re-derive the quantized lm_head copies in model from their source heads.
Copies in place, so CUDA graphs keep pointing at valid tensors.
Source code in vllm/model_executor/layers/quantization/online/lm_head.py
restore_quantized_lm_heads(model, refresh=True)
¶
Undo share_source_lm_heads, then re-derive the copies in model.
Parameters:
-
(model¶Module) –The model passed to
share_source_lm_heads, if any. -
(refresh¶bool, default:True) –Whether to re-derive the copies; False when the source heads may be mid-reload.
Source code in vllm/model_executor/layers/quantization/online/lm_head.py
share_source_lm_heads(model)
¶
Replace each quantized copy in model with its source head.
model then loads weights like a drafter sharing the unquantized head,
until restore_quantized_lm_heads swaps the copies back in.