Skip to content

vllm.distributed.eplb.policy.abstract

Classes:

AbstractEplbPolicy

Bases: ABC

Methods:

Source code in vllm/distributed/eplb/policy/abstract.py
class AbstractEplbPolicy(ABC):
    @abstractmethod
    def rebalance_experts(
        self,
        stats: EplbStats,
        old_global_expert_indices: torch.Tensor | None = None,
    ) -> torch.Tensor:
        """Entry point for expert-parallelism load balancer.

        Args:
            stats: Load/topology snapshot with a CPU load tensor.
                Built-in policies use `[layers, num_logical_experts]` loads.
                Policy-specific subclasses may provide additional inputs.
            old_global_expert_indices: [layers, num_replicas], the old global
                expert indices. Used to avoid unnecessary weight copying
                for experts moving within one rank.

        Returns:
            physical_to_logical_map: [layers, num_replicas], the expert
                index of each replica, on CPU.

        """
        raise NotImplementedError

rebalance_experts(stats, old_global_expert_indices=None) abstractmethod

Entry point for expert-parallelism load balancer.

Parameters:

  • stats

    (EplbStats) –

    Load/topology snapshot with a CPU load tensor. Built-in policies use [layers, num_logical_experts] loads. Policy-specific subclasses may provide additional inputs.

  • old_global_expert_indices

    (Tensor | None, default: None ) –

    [layers, num_replicas], the old global expert indices. Used to avoid unnecessary weight copying for experts moving within one rank.

Returns:

  • physical_to_logical_map ( Tensor ) –

    [layers, num_replicas], the expert index of each replica, on CPU.

Source code in vllm/distributed/eplb/policy/abstract.py
@abstractmethod
def rebalance_experts(
    self,
    stats: EplbStats,
    old_global_expert_indices: torch.Tensor | None = None,
) -> torch.Tensor:
    """Entry point for expert-parallelism load balancer.

    Args:
        stats: Load/topology snapshot with a CPU load tensor.
            Built-in policies use `[layers, num_logical_experts]` loads.
            Policy-specific subclasses may provide additional inputs.
        old_global_expert_indices: [layers, num_replicas], the old global
            expert indices. Used to avoid unnecessary weight copying
            for experts moving within one rank.

    Returns:
        physical_to_logical_map: [layers, num_replicas], the expert
            index of each replica, on CPU.

    """
    raise NotImplementedError

EplbStats dataclass

Load and topology snapshot for one EPLB planning call.

Policies may accept subclasses with additional planning inputs.

Attributes:

Source code in vllm/distributed/eplb/policy/abstract.py
@dataclass(slots=True)
class EplbStats:
    """Load and topology snapshot for one EPLB planning call.

    Policies may accept subclasses with additional planning inputs."""

    global_expert_load_window: torch.Tensor
    """
    Expert load window, converted to CPU before policy invocation.
    Built-in policies use shape (num_moe_layers, num_logical_experts).
    """
    num_replicas: int
    """
    Number of physical experts.
    """
    num_groups: int
    """
    Number of expert groups.
    """
    num_nodes: int
    """
    Number of nodes.
    """
    num_gpus: int
    """
    Number of GPUs.
    """

global_expert_load_window instance-attribute

Expert load window, converted to CPU before policy invocation. Built-in policies use shape (num_moe_layers, num_logical_experts).

num_gpus instance-attribute

Number of GPUs.

num_groups instance-attribute

Number of expert groups.

num_nodes instance-attribute

Number of nodes.

num_replicas instance-attribute

Number of physical experts.