vllm.distributed.eplb.policy.abstract
¶
Classes:
-
AbstractEplbPolicy– -
EplbStats–Load and topology snapshot for one EPLB planning call.
AbstractEplbPolicy
¶
Bases: ABC
Methods:
-
rebalance_experts–Entry point for expert-parallelism load balancer.
Source code in vllm/distributed/eplb/policy/abstract.py
rebalance_experts(stats, old_global_expert_indices=None)
abstractmethod
¶
Entry point for expert-parallelism load balancer.
Parameters:
-
(stats¶EplbStats) –Load/topology snapshot with a CPU load tensor. Built-in policies use
[layers, num_logical_experts]loads. Policy-specific subclasses may provide additional inputs. -
(old_global_expert_indices¶Tensor | None, default:None) –[layers, num_replicas], the old global expert indices. Used to avoid unnecessary weight copying for experts moving within one rank.
Returns:
-
physical_to_logical_map(Tensor) –[layers, num_replicas], the expert index of each replica, on CPU.
Source code in vllm/distributed/eplb/policy/abstract.py
EplbStats
dataclass
¶
Load and topology snapshot for one EPLB planning call.
Policies may accept subclasses with additional planning inputs.
Attributes:
-
global_expert_load_window(Tensor) –Expert load window, converted to CPU before policy invocation.
-
num_gpus(int) –Number of GPUs.
-
num_groups(int) –Number of expert groups.
-
num_nodes(int) –Number of nodes.
-
num_replicas(int) –Number of physical experts.
Source code in vllm/distributed/eplb/policy/abstract.py
global_expert_load_window
instance-attribute
¶
Expert load window, converted to CPU before policy invocation. Built-in policies use shape (num_moe_layers, num_logical_experts).
num_gpus
instance-attribute
¶
Number of GPUs.
num_groups
instance-attribute
¶
Number of expert groups.
num_nodes
instance-attribute
¶
Number of nodes.
num_replicas
instance-attribute
¶
Number of physical experts.