Skip to content

vllm.model_executor.layers.mamba.kda_checkpoint

Classes:

Functions:

FlashKDAPrefillCheckpointExporter dataclass

Bases: MambaPrefillCheckpointExporter

Store KDA (FlashKDA or Triton) recurrent and convolution checkpoint states.

Source code in vllm/model_executor/layers/mamba/kda_checkpoint.py
class FlashKDAPrefillCheckpointExporter(MambaPrefillCheckpointExporter):
    """Store KDA (FlashKDA or Triton) recurrent and convolution checkpoint states."""

get_triton_kda_checkpoint_buffer(checkpoint, spec, num_seqs)

The checkpoint_state and checkpoint_offsets to pass the Triton chunk kernel, or Nones when the batch has no checkpoint.

Source code in vllm/model_executor/layers/mamba/kda_checkpoint.py
def get_triton_kda_checkpoint_buffer(
    checkpoint: MambaPrefillCheckpointMetadata | None,
    spec: TritonKDACheckpointSpec | None,
    num_seqs: int,
) -> tuple[torch.Tensor | None, torch.Tensor | None]:
    """The `checkpoint_state` and `checkpoint_offsets` to pass the Triton chunk
    kernel, or `None`s when the batch has no checkpoint."""
    if checkpoint is None:
        return None, None
    assert spec is not None
    (checkpoint_state,) = current_workspace_manager().get_simultaneous(spec)
    return checkpoint_state[:num_seqs], checkpoint.checkpoint_offsets

reserve_triton_kda_checkpoint_buffer(vllm_config, num_heads, head_dim, dtype)

Reserve the [max_num_seqs, H, V, K] workspace buffer the Triton chunk kernel stores checkpoint states into; profiling skips attention, so the layer reserves it before the KV cache is sized.

Source code in vllm/model_executor/layers/mamba/kda_checkpoint.py
def reserve_triton_kda_checkpoint_buffer(
    vllm_config: VllmConfig, num_heads: int, head_dim: int, dtype: torch.dtype
) -> TritonKDACheckpointSpec:
    """Reserve the `[max_num_seqs, H, V, K]` workspace buffer the Triton chunk
    kernel stores checkpoint states into; profiling skips attention, so the
    layer reserves it before the KV cache is sized."""
    max_seqs = vllm_config.scheduler_config.max_num_seqs
    spec = ((max_seqs, num_heads, head_dim, head_dim), dtype)
    if is_workspace_manager_initialized():
        current_workspace_manager().get_simultaneous(spec)
    return spec