Skip to content

vllm.distributed.pp_transport

Alternatives to NCCL for sending a PP stage's intermediate tensors to the next stage, selected by VLLM_PP_TRANSPORT.

Modules:

  • copy_engine –

    PP transport on CUDA copy engines, synchronized on the GPU (one NVLink

  • rocm_sdma –

    The flag ops and peer copy of copy_engine.CopyEngineTransport on ROCm

Classes:

  • PPTransport –

    One rank's side of its PP stage boundaries.

Functions:

PPTransport

Bases: ABC

One rank's side of its PP stage boundaries.

Methods:

  • recv –

    Receive this step's inputs from the previous stage.

  • send –

    Send this step's outputs to the next stage. The caller may

Source code in vllm/distributed/pp_transport/__init__.py
class PPTransport(ABC):
    """One rank's side of its PP stage boundaries."""

    @abstractmethod
    def send(self, tensors: dict, pp_group: "GroupCoordinator") -> None:
        """Send this step's outputs to the next stage. The caller may
        overwrite `tensors` once this returns."""

    @abstractmethod
    def recv(self, pp_group: "GroupCoordinator") -> "IntermediateTensors":
        """Receive this step's inputs from the previous stage."""

recv(pp_group) abstractmethod

Receive this step's inputs from the previous stage.

Source code in vllm/distributed/pp_transport/__init__.py
@abstractmethod
def recv(self, pp_group: "GroupCoordinator") -> "IntermediateTensors":
    """Receive this step's inputs from the previous stage."""

send(tensors, pp_group) abstractmethod

Send this step's outputs to the next stage. The caller may overwrite tensors once this returns.

Source code in vllm/distributed/pp_transport/__init__.py
@abstractmethod
def send(self, tensors: dict, pp_group: "GroupCoordinator") -> None:
    """Send this step's outputs to the next stage. The caller may
    overwrite `tensors` once this returns."""

check_intermediate_tensors(tensors, expected)

Raise unless tensors match the receiving stage's persistent intermediate tensors in keys, dtypes and per-token shapes, which every PP transport requires. Otherwise the copy into them would silently drop, cast or broadcast values.

Source code in vllm/distributed/pp_transport/__init__.py
def check_intermediate_tensors(
    tensors: Mapping[str, torch.Tensor], expected: Mapping[str, RowSpec]
) -> None:
    """Raise unless `tensors` match the receiving stage's persistent intermediate
    tensors in keys, dtypes and per-token shapes, which every PP transport
    requires. Otherwise the copy into them would silently drop, cast or
    broadcast values."""
    actual = row_specs(tensors)
    if actual != expected:
        raise RuntimeError(
            f"PP intermediate tensors {actual} do not match the receiving "
            f"stage's {dict(expected)}. Declare them in the model's "
            "make_empty_intermediate_tensors."
        )

create_pp_transport(vllm_config, persistent)

Collective over the PP group. Returns None (send over NCCL) without PP, under "auto" on ROCm, or under "auto" when copy engines are not usable on some rank; every rank makes the same choice.

Parameters:

  • vllm_config

    (VllmConfig) –

    The engine config.

  • persistent

    (IntermediateTensors | None) –

    The model runner's persistent intermediate tensors (non-first ranks), which received tensors are copied into.

Source code in vllm/distributed/pp_transport/__init__.py
def create_pp_transport(
    vllm_config: "VllmConfig", persistent: "IntermediateTensors | None"
) -> PPTransport | None:
    """Collective over the PP group. Returns None (send over NCCL) without PP,
    under "auto" on ROCm, or under "auto" when copy engines are not usable on
    some rank; every rank makes the same choice.

    Args:
        vllm_config: The engine config.
        persistent: The model runner's persistent intermediate tensors (non-first
            ranks), which received tensors are copied into.

    """
    from vllm.distributed.parallel_state import get_pp_group

    pp_group = get_pp_group()
    if pp_group.world_size == 1:
        return None
    mode = envs.VLLM_PP_TRANSPORT
    assert mode in ("auto", "copy_engine")
    from vllm.distributed.pp_transport.copy_engine import (
        CopyEngineTransport,
        CopyEngineUnavailable,
    )
    from vllm.platforms import current_platform

    if mode == "auto" and current_platform.is_rocm():
        logger.info_once(
            "PP transport: NCCL (SDMA copy engines on ROCm are opt-in: "
            "VLLM_PP_TRANSPORT=copy_engine)"
        )
        return None

    try:
        transport = CopyEngineTransport(
            vllm_config, pp_group, None if persistent is None else persistent.tensors
        )
    except CopyEngineUnavailable as e:
        if mode == "copy_engine":
            raise
        logger.info_once("PP transport: NCCL (%s)", e)
        return None
    logger.info_once("PP transport: copy engines")
    return transport