vllm.utils.flashinfer
¶
Compatibility wrapper for FlashInfer API changes.
Users of vLLM should always import only these wrappers.
Functions:
-
can_use_trtllm_attention–Check if the current configuration supports TRTLLM attention.
-
has_flashinfer–Return
Trueif flashinfer-python package is available. -
has_flashinfer_autotune_v2–Return
Trueif FlashInfer ships the managed-cache autotuner -
has_flashinfer_b12x_gemm–Return True if FlashInfer b12x FP4 GEMM backend is available (SM120+).
-
has_flashinfer_b12x_moe–Return
Trueif FlashInfer CuteDSL SM12x fused MoE is available. -
has_flashinfer_bf16_fp4–Return
Trueif FlashInfer's CuTe-DSL W4A16 GEMM is available. -
has_flashinfer_bf16_gemm–Return whether FlashInfer exposes the BF16 dense GEMM API.
-
has_flashinfer_comm–Return
Trueif FlashInfer comm module is available. -
has_flashinfer_concat_mla_kv_quant_fp8–Return whether FlashInfer exposes the MLA K/V pack + fp8 cast API.
-
has_flashinfer_cutedsl_grouped_gemm_nt_masked–Return
Trueif FlashInfer CUTLASS fused MoE is available. -
has_flashinfer_cutedsl_moe_nvfp4–Return
Trueif FlashInfer cute_dsl_fused_moe_nvfp4 is available. -
has_flashinfer_cutlass_fused_moe–Return
Trueif FlashInfer CUTLASS fused MoE is available. -
has_flashinfer_fp8_blockscale_gemm–Return
Trueif FlashInfer block-scale FP8 GEMM is available. -
has_flashinfer_fused_kda_decode–Return whether FlashInfer fused KDA decode is available.
-
has_flashinfer_jit_cache_wheels–Return whether FlashInfer publishes flashinfer-jit-cache for the installed
-
has_flashinfer_localized_moe–Return
Trueif cute_dsl_fused_moe_nvfp4 accepts thelocalized_*args. -
has_flashinfer_moe–Return
Trueif FlashInfer MoE module is available. -
has_flashinfer_nvlink_one_sided–Return whether FlashInfer exposes the NVLink MoE EP wrapper.
-
has_flashinfer_nvlink_two_sided–Return whether FlashInfer exposes the NVLink two-sided MoE EP wrapper.
-
has_flashinfer_recurrent_kda–Return whether FlashInfer recurrent KDA prefill is available.
-
has_nvidia_artifactory–Return
Trueif NVIDIA's artifactory is accessible. -
is_flashinfer_autotuning–Whether a FlashInfer autotune(True) context is active.
-
is_flashinfer_bf16_gemm_supported–Return whether an exact FlashInfer BF16 backend is available.
-
is_flashinfer_cudnn_fp8_prefill_attn_supported–Check if FP8 ViT attention is supported on this platform.
-
is_flashinfer_cutedsl_bf16_gemm_supported–Return whether the CuTeDSL BF16 dense GEMM backend is available.
-
is_flashinfer_fp8_blockscale_gemm_supported–Return
Trueif FlashInfer block-scale FP8 GEMM is supported. -
supports_trtllm_attention–Return whether TRTLLM attention is available on the current platform
-
use_trtllm_attention–Return
Trueif TRTLLM attention is used. -
warn_if_flashinfer_kernels_missing–Warn on Hopper and newer GPUs when FlashInfer's precompiled kernels are
_flashinfer_concat_mla_k(k, k_nope, k_pe)
¶
Custom op wrapper for flashinfer's concat_mla_k.
This is an in-place operation that concatenates k_nope and k_pe into k.
The kernel is optimized for DeepSeek V3 dimensions: - num_heads=128 - nope_dim=128 - rope_dim=64
Key optimizations: - Warp-based processing with software pipelining - Vectorized memory access (int2 for nope, int for rope) - L2 prefetching for next row while processing current - Register reuse for rope values across all heads
Parameters:
-
(k¶Tensor) –Output tensor, shape [num_tokens, num_heads, nope_dim + rope_dim]. Modified in-place.
-
(k_nope¶Tensor) –The nope part of k, shape [num_tokens, num_heads, nope_dim].
-
(k_pe¶Tensor) –The rope part of k (shared), shape [num_tokens, 1, rope_dim]. This is broadcast to all heads.
Source code in vllm/utils/flashinfer.py
_flashinfer_nvcc_path()
¶
Return the nvcc FlashInfer's JIT would run, or None if it is missing.
Mirrors flashinfer.jit.cpp_ext.get_cuda_path() without importing
FlashInfer, whose import initializes CUDA.
Source code in vllm/utils/flashinfer.py
_get_submodule(module_name)
¶
Safely import a submodule and return it, or None if not available.
_lazy_import_wrapper(module_name, attr_name, fallback_fn=_missing)
¶
Create a lazy import wrapper for a specific function.
Source code in vllm/utils/flashinfer.py
_missing(*_, **__)
¶
Placeholder for unavailable FlashInfer backend.
Source code in vllm/utils/flashinfer.py
can_use_trtllm_attention(num_qo_heads, num_kv_heads, is_prefill=False)
¶
Check if the current configuration supports TRTLLM attention.
Source code in vllm/utils/flashinfer.py
force_use_trtllm_attention()
¶
This function should only be called during initialization stage when vllm config
is set.
Return None if --attention-config.use_trtllm_attention is not set,
return True if TRTLLM attention is forced to be used,
return False if TRTLLM attention is forced to be not used.
Source code in vllm/utils/flashinfer.py
has_flashinfer()
cached
¶
Return True if flashinfer-python package is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_autotune_v2()
cached
¶
Return True if FlashInfer ships the managed-cache autotuner
(flashinfer.autotune_v2, flashinfer-ai/flashinfer#3861).
Source code in vllm/utils/flashinfer.py
has_flashinfer_b12x_gemm()
cached
¶
Return True if FlashInfer b12x FP4 GEMM backend is available (SM120+).
Source code in vllm/utils/flashinfer.py
has_flashinfer_b12x_moe()
cached
¶
Return True if FlashInfer CuteDSL SM12x fused MoE is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_bf16_fp4()
cached
¶
Return True if FlashInfer's CuTe-DSL W4A16 GEMM is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_bf16_gemm()
cached
¶
Return whether FlashInfer exposes the BF16 dense GEMM API.
Source code in vllm/utils/flashinfer.py
has_flashinfer_comm()
cached
¶
Return True if FlashInfer comm module is available.
has_flashinfer_concat_mla_kv_quant_fp8()
cached
¶
Return whether FlashInfer exposes the MLA K/V pack + fp8 cast API.
Source code in vllm/utils/flashinfer.py
has_flashinfer_cubin()
cached
¶
Return True if flashinfer-cubin package is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_cutedsl()
cached
¶
Return True if FlashInfer cutedsl module is available.
has_flashinfer_cutedsl_grouped_gemm_nt_masked()
cached
¶
Return True if FlashInfer CUTLASS fused MoE is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_cutedsl_moe_nvfp4()
cached
¶
Return True if FlashInfer cute_dsl_fused_moe_nvfp4 is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_cutlass_fused_moe()
cached
¶
Return True if FlashInfer CUTLASS fused MoE is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_fp8_blockscale_gemm()
cached
¶
Return True if FlashInfer block-scale FP8 GEMM is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_fused_kda_decode()
cached
¶
Return whether FlashInfer fused KDA decode is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_jit_cache_wheels()
¶
Return whether FlashInfer publishes flashinfer-jit-cache for the installed PyTorch's CUDA version, which it does from CUDA 12.9.
Source code in vllm/utils/flashinfer.py
has_flashinfer_localized_moe()
cached
¶
Return True if cute_dsl_fused_moe_nvfp4 accepts the localized_* args.
Locality-domain execution landed in flashinfer-ai/flashinfer#5276 and was
adapted to torch.cuda.execute_on_streams in #6488. Neither is in a
FlashInfer 0.7.x release, so probe the signature rather than the version.
Source code in vllm/utils/flashinfer.py
has_flashinfer_moe()
cached
¶
Return True if FlashInfer MoE module is available.
has_flashinfer_msa()
cached
¶
Return True if FlashInfer MiniMax Sparse Attention (MSA) ops are
available. Which architectures they serve depends on the build.
Source code in vllm/utils/flashinfer.py
has_flashinfer_msa_packed_kv()
cached
¶
Return True if the FlashInfer MSA attention kernels accept K/V
views split from vLLM's packed-content KV cache, advertised by the
SUPPORTS_PACKED_KV flag on flashinfer.msa_ops.
Source code in vllm/utils/flashinfer.py
has_flashinfer_msa_per_token_topk()
cached
¶
Return True if flashinfer.msa_ops.msa_topk_select accepts a
per-token num_valid_pages tensor. Detected from the parameter's type
annotation, since flashinfer exposes no capability flag for it.
Source code in vllm/utils/flashinfer.py
has_flashinfer_msa_sm90()
cached
¶
Return True if the FlashInfer MSA attention kernels serve SM90
(Hopper). Detected from the out parameter both attention ops gained
with their SM90 backend, since flashinfer exposes no capability flag for
it.
Source code in vllm/utils/flashinfer.py
has_flashinfer_nvlink_one_sided()
cached
¶
Return whether FlashInfer exposes the NVLink MoE EP wrapper.
Source code in vllm/utils/flashinfer.py
has_flashinfer_nvlink_two_sided()
cached
¶
Return whether FlashInfer exposes the NVLink two-sided MoE EP wrapper.
Source code in vllm/utils/flashinfer.py
has_flashinfer_packed_fused_kda_decode()
cached
¶
Return whether FlashInfer's packed fused KDA decode API is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_prims_ts_sparse_mla()
cached
¶
Return True if FlashInfer ships the task-scheduled (prims_ts) sparse
MLA decode wrapper with the plan API vLLM calls. Its quantized formats are
probed separately by prims_ts_sparse_mla_format_error once a device is
available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_recurrent_kda()
cached
¶
Return whether FlashInfer recurrent KDA prefill is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_sm90_nope_mla()
cached
¶
FlashInfer SM90 NoPE MLA (FP8 KV with in-kernel dequant, kpe=0).
Feature-detected via the ckv_scale_arr run() kwarg introduced with
the SM90 NoPE support (FlashInfer >= 0.6.18), so dev builds carry the
gate without a version parse.
Source code in vllm/utils/flashinfer.py
has_flashinfer_sparse_mla_sm120()
cached
¶
Return True if FlashInfer sparse MLA decode support is available.
Source code in vllm/utils/flashinfer.py
has_flashinfer_sparse_mla_sm120_config(num_q_heads, top_k)
cached
¶
Return whether FlashInfer ships an SM120 DSV4 decode specialization.
The public sparse MLA API predates some DSV4 shapes, so checking only that the callable exists can select a package that later aborts or rejects a valid vLLM configuration. Inspect FlashInfer's dispatch table until it exposes a public capability query.
Source code in vllm/utils/flashinfer.py
has_flashinfer_topk_varlen_gvr2()
cached
¶
Return True if FlashInfer ships the gvr_2 varlen top-k backend
(self-sampling GVR V2, flashinfer.top_k_varlen(backend="gvr_2")).
Feature-detected on the production entry run_varlen (FlashInfer >=
0.7.2: per-row kv_lens, a capture-stable max_seq_len and the
hint-free top_k) and the band-aware warmup_varlen (row_stride)
that the dispatcher calls.
Source code in vllm/utils/flashinfer.py
has_flashinfer_trtllm_fused_moe()
cached
¶
Return True if FlashInfer TRTLLM fused MoE is available.
Source code in vllm/utils/flashinfer.py
has_nvidia_artifactory()
cached
¶
Return True if NVIDIA's artifactory is accessible.
This checks connectivity to the kernel inference library artifactory which is required for downloading certain cubin kernels like TRTLLM FHMA.
Source code in vllm/utils/flashinfer.py
is_flashinfer_autotuning()
¶
is_flashinfer_bf16_gemm_supported(backend, compute_capability=None)
cached
¶
Return whether an exact FlashInfer BF16 backend is available.
Source code in vllm/utils/flashinfer.py
is_flashinfer_cudnn_fp8_prefill_attn_supported()
cached
¶
Check if FP8 ViT attention is supported on this platform.
Requires Blackwell (SM 100) or newer, the FlashInfer cuDNN backend, and cuDNN >= 9.17.1.
cuDNN's FP8 SDPA forward path with bf16/fp16 output (used by
MMEncoderAttention._forward_flashinfer) gates internally on
prop.major >= 10; on Hopper it raises a misleading
cudnnGraphNotSupportedError: ... cuDNN version 9.13.0 and newer
even when the installed cuDNN is new enough. See PR #38065 for the
original Blackwell-only design intent.
Source code in vllm/utils/flashinfer.py
is_flashinfer_cutedsl_bf16_gemm_supported()
cached
¶
Return whether the CuTeDSL BF16 dense GEMM backend is available.
Source code in vllm/utils/flashinfer.py
is_flashinfer_fp8_blockscale_gemm_supported()
cached
¶
Return True if FlashInfer block-scale FP8 GEMM is supported.
Source code in vllm/utils/flashinfer.py
pin_host_range_buf(length)
¶
Pin flashinfer's cached host-side arange buffer covering length.
flashinfer's decode plan() builds its host qo_indptr from the cached
_get_range_buf entry and copies it to the GPU with non_blocking=True on
every call; the cache entry is unpinned, so the copy silently blocks the
host. Replace the entry with a pinned copy so the copies stay async.
Source code in vllm/utils/flashinfer.py
prims_ts_sparse_mla_format_error(kv_cache_format)
cached
¶
Why the installed FlashInfer cannot plan prims_ts sparse MLA (D576, 128
E4M3 query heads, LSE) over kv_cache_format on this GPU, else None.
Asks the public workspace-size query, which resolves the plan without allocating or compiling (FlashInfer >= the nightly with PR 7410 for "nvfp4.fp8rope").
Source code in vllm/utils/flashinfer.py
supports_trtllm_attention(is_prefill=False)
cached
¶
Return whether TRTLLM attention is available on the current platform for the given attention phase.
SM90 (Hopper) and SM12x support the XQA decode kernel but not TRTLLM prefill. SM100+ supports TRTLLM for both phases. All others are unsupported.
Source code in vllm/utils/flashinfer.py
use_trtllm_attention(num_qo_heads, num_kv_heads, num_tokens, max_seq_len, dcp_world_size, kv_cache_dtype, q_dtype, is_prefill, force_use_trtllm=None, has_sinks=False, has_spec=False)
¶
Return True if TRTLLM attention is used.
Source code in vllm/utils/flashinfer.py
917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 | |
warn_if_flashinfer_kernels_missing()
¶
Warn on Hopper and newer GPUs when FlashInfer's precompiled kernels are missing or were installed for another FlashInfer version.