vllm.model_executor.kernels.linear.nvfp4
¶
Modules:
Classes:
-
NvFp4LinearKernel–Base class for NVFP4 quantized linear kernels.
-
NvFp4LinearLayerConfig–Configuration for an NVFP4 linear layer.
NvFp4LinearKernel
¶
Bases: ABC
Base class for NVFP4 quantized linear kernels.
Each subclass implements a specific GEMM backend (CUTLASS, Marlin, etc).
The kernel selection mechanism iterates over registered subclasses in
priority order,calling is_supported and can_implement to find the best
match for the current hardware.
Methods:
-
apply_weights–Run the quantized GEMM.
-
can_implement–Return whether this kernel can handle config.
-
executed_activation_key–The activation quantization the kernel runs for a layer requesting
-
input_quant_key–Return the input quantization key supported by this kernel. If the kernel
-
is_supported–Return whether this kernel can run on the current platform.
-
prepare_input–Return 2D packed NVFP4 data with swizzled 128x4 block scales.
-
process_weights_after_loading–Transform weights into the format required by this kernel.
Source code in vllm/model_executor/kernels/linear/nvfp4/base.py
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
apply_weights(layer, x, bias=None)
abstractmethod
¶
Run the quantized GEMM.
can_implement(config)
abstractmethod
classmethod
¶
Return whether this kernel can handle config.
executed_activation_key(activation_quant_key)
classmethod
¶
The activation quantization the kernel runs for a layer requesting
activation_quant_key. Weight-only kernels return None; kernel
selection then warns about, or under VLLM_STRICT_QUANT_SCHEME forbids,
the fallback.
Source code in vllm/model_executor/kernels/linear/nvfp4/base.py
input_quant_key()
¶
Return the input quantization key supported by this kernel. If the kernel does not support input quantization outside of the kernel, return None.
is_supported(compute_capability=None)
abstractmethod
classmethod
¶
Return whether this kernel can run on the current platform.
prepare_input(layer, x)
¶
Return 2D packed NVFP4 data with swizzled 128x4 block scales.
K is zero-padded to match the padded weight. Prequantized inputs are validated and padded without modifying the caller's activation.
Source code in vllm/model_executor/kernels/linear/nvfp4/base.py
process_weights_after_loading(layer)
abstractmethod
¶
Transform weights into the format required by this kernel.
Called once after checkpoint weights have been loaded onto the device. Implementations should repack / swizzle / pad weights and scales in-place on layer.
Source code in vllm/model_executor/kernels/linear/nvfp4/base.py
NvFp4LinearLayerConfig
dataclass
¶
Configuration for an NVFP4 linear layer.
All NVFP4 layers share the same structure: packed uint8 weights (2 FP4 values per
byte), FP8-E4M3 per-block weight scales (group size 16), and scalar global
scales for both weights and activations. act_dtype is the activation
dtype of weight-only layers, when the caller knows it.