Skip to content

vllm.model_executor.model_loader.weight_cache.daemon

Weight cache daemon for fast engine restarts.

One daemon process per GPU holds the post-quantized, TP-sharded weights of its rank in GPU memory and serves CUDA IPC handles to vLLM engines over a Unix domain socket. Restarting engines map the weights via zero-copy IPC instead of reloading from disk.

Launch one daemon per TP rank with a single command:

vllm preload \
    --model /path/to/model --tensor-parallel-size 4

Engines then load from the daemons with:

vllm serve /path/to/model --tensor-parallel-size 4 \
    --load-format ipc_cache

Tensor, expert, data and pipeline parallelism are supported.

For multi-node tensor parallelism, run one launcher per node with a shared rendezvous so the global TP group forms across nodes (CUDA IPC handles are node-local, so each node serves only its local GPUs' shards). Reuse the same --nnodes/--node-rank/--master-addr flags you pass the engine, plus a --weight-cache-master-port distinct from the engine's --master-port:

# node 0 (8 local GPUs)
vllm preload \
    --model /path/to/model --tensor-parallel-size 16 \
    --nnodes 2 --node-rank 0 --master-addr 10.0.0.1 \
    --weight-cache-master-port 29600
# node 1 (8 local GPUs)
vllm preload \
    --model /path/to/model --tensor-parallel-size 16 \
    --nnodes 2 --node-rank 1 --master-addr 10.0.0.1 \
    --weight-cache-master-port 29600

The global TP rank of local GPU i on node r is r * (tp_size // nnodes) + i, matching vLLM's contiguous per-node rank assignment, so each engine worker maps its shard from the daemon on its own node. With pipeline parallelism each node additionally runs one daemon per PP stage; the global rank is pp_rank * tp_size + tp_rank.

For data parallelism (e.g. a TP1 x DP16 x EP decode fleet) run one launcher per node with the engine's DP placement flags. Local GPU i serves DP rank start_rank + i // tp_size and TP rank i % tp_size, and all dp_size * tp_size daemons form one world group on --data-parallel-address/--weight-cache-master-port so the expert shards are laid out exactly as in the engine:

# node r (4 local GPUs)
vllm preload \
    --model /path/to/model --tensor-parallel-size 1 --enable-expert-parallel \
    --data-parallel-size 16 --data-parallel-size-local 4 \
    --data-parallel-start-rank 4r --data-parallel-address 10.0.0.1 \
    --weight-cache-master-port 29600

Data parallelism also combines with multi-node tensor parallelism: pass both flag sets (--nnodes/--node-rank/--master-addr and the DP flags with a reachable --data-parallel-address). Each node then serves a contiguous block of the dp_size * tp_size global ranks, e.g. TP8 x DP2 on 4 nodes:

# node r (4 local GPUs)
vllm preload \
    --model /path/to/model --tensor-parallel-size 8 --enable-expert-parallel \
    --nnodes 4 --node-rank r --master-addr 10.0.0.1 \
    --data-parallel-size 2 --data-parallel-address 10.0.0.1 \
    --weight-cache-master-port 29600

With MTP, EAGLE or EAGLE3 speculative decoding the launcher additionally starts a draft daemon group that caches the draft model. It uses its own cache key, Unix sockets (*_draft.sock) and rendezvous port (--weight-cache-draft-master-port, default --weight-cache-master-port + 1), so each process serves exactly one model role. Other draft types are not cached and keep loading from disk in the engine. With pipeline parallelism only the draft daemons of the last stage serve the draft model.

Seeding a replica from another

A second replica's daemons can seed their shards from a loaded replica with --weight-cache-seed; see docs/features/preload.md.

Classes:

  • WeightCacheDaemon –

    Per-GPU process that loads one TP/PP shard and serves CUDA IPC handles.

Functions:

WeightCacheDaemon

Per-GPU process that loads one TP/PP shard and serves CUDA IPC handles.

Methods:

  • get_model –

    Load the daemon's model, composed from the configured loader.

  • release_distributed –

    Release process groups and BLAS workspaces after loading and warmup.

  • serve_forever –

    Serve requests until terminated.

  • warmup –

    Run each warmup cache's pass (adopting remote caches via D2D).

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
class WeightCacheDaemon:
    """Per-GPU process that loads one TP/PP shard and serves CUDA IPC handles."""

    def __init__(
        self,
        vllm_config: VllmConfig,
        global_rank: int,
        local_rank: int,
        distributed_init_method: str,
        socket_dir: str | None = None,
        is_draft: bool = False,
        dp_rank: int = 0,
        pp_rank: int = 0,
        seed_config: SeedConfig | None = None,
        device_index: int | None = None,
        warmup_cache_stores: Iterable[str] = (),
    ):
        parallel_config = vllm_config.parallel_config
        if parallel_config.enable_eplb:
            raise ValueError("The weight cache daemon does not support EPLB.")
        # Seed peers are served on their own thread; this serializes their
        # reads of the state with the local socket's requests. Reentrant
        # because the local "seed" command re-enters it through
        # WeightShardSeed.resolve.
        self._state_lock = threading.RLock()
        self.seed_config = seed_config or SeedConfig()
        warmup_cache_stores = tuple(warmup_cache_stores)
        self.warmup_caches: list[WarmupCache] = [
            FlashInferWarmupCache(warmup_cache_stores),
            CompileArtifactWarmupCache(warmup_cache_stores),
        ]
        self.weight_seed = WeightShardSeed(self)
        self.d2d = SeedTransport(
            [*self.warmup_caches, self.weight_seed], self.seed_config
        )
        self.tp_size = parallel_config.tensor_parallel_size
        self.pp_size = parallel_config.pipeline_parallel_size
        self.dp_size = parallel_config.data_parallel_size
        if self.dp_size > 1:
            # The engine sets these on each DP rank's workers; the MoE/EP
            # layers read them to place their expert shards.
            if parallel_config.nnodes > 1:
                world_size = self.dp_size * self.tp_size
                local_world = world_size // parallel_config.nnodes
                start_dp_rank = parallel_config.node_rank * local_world // self.tp_size
            else:
                start_dp_rank = parallel_config.data_parallel_rank
            parallel_config = replace(
                parallel_config,
                data_parallel_rank=dp_rank,
                data_parallel_rank_local=dp_rank - start_dp_rank,
            )
            vllm_config = replace(vllm_config, parallel_config=parallel_config)
        self.vllm_config = vllm_config
        # A draft daemon builds the draft with the target's VllmConfig, like
        # the engine does; only the ModelConfig differs.
        if is_draft:
            spec = vllm_config.speculative_config
            assert spec is not None and spec.draft_model_config is not None
            self.model_config = spec.draft_model_config
        else:
            self.model_config = vllm_config.model_config
        self.global_rank = global_rank
        rank_in_dp = global_rank - dp_rank * self.pp_size * self.tp_size
        # Set like the engine's workers do: per-rank state such as the
        # torch.compile cache is keyed by it, and a compiled artifact shared
        # across ranks would run every rank's kernels on rank 0's device.
        vllm_config.parallel_config.rank = rank_in_dp
        self.tp_rank = rank_in_dp % self.tp_size
        self.pp_rank = rank_in_dp // self.tp_size
        self.dp_rank = dp_rank
        self.world_size = self.dp_size * self.pp_size * self.tp_size
        self.local_rank = local_rank
        # A peer's CUDA IPC handle names its device by index, so a same-host
        # mirror keeps every device visible and places its own ranks on the
        # GPUs the source does not use; the launcher infers the placement
        # from the source's GPU UUIDs and passes it explicitly.
        self.device_index = local_rank if device_index is None else device_index
        self.distributed_init_method = distributed_init_method
        self.socket_dir = socket_dir
        self.is_draft = is_draft
        self.role = format_daemon_role(is_draft)
        self.serves_model = daemon_serves_model(is_draft, self.pp_rank, self.pp_size)
        self.model: torch.nn.Module | None = None
        # Engine watch connections, held open for the daemon's lifetime.
        self._watchers: list[socket.socket] = []
        # Fingerprint before loading: process_weights_after_loading may
        # mutate hf_config.quantization_config. Built by the shared helper
        # so the peer loader requests exactly the shard fingerprinted here.
        self.cache_key = build_weight_cache_key(
            vllm_config, self.model_config, is_draft=is_draft
        )
        if is_draft:
            # The engine builds the draft as the only pipeline stage.
            self.cache_key = replace(self.cache_key, pp_size=1, pp_rank=0)

    def warmup(self) -> None:
        """Run each warmup cache's pass (adopting remote caches via D2D)."""
        for cache in self.warmup_caches:
            cache.warmup(self, self.d2d)

    def load_model(self) -> None:
        torch.accelerator.set_device_index(self.device_index)
        # Outside the config context so the daemon keeps its own rendezvous
        # instead of the engine's; model parallel groups (TP/DP/EP) are then
        # carved out of this world group like in the engine.
        init_distributed_environment(
            world_size=self.world_size,
            rank=self.global_rank,
            distributed_init_method=self.distributed_init_method,
            local_rank=self.device_index,
            backend=current_platform.dist_backend,
        )
        if not self.serves_model:
            # Model parallel groups are created collectively by every rank.
            with set_current_vllm_config(self.vllm_config):
                ensure_model_parallel_initialized(self.tp_size, self.pp_size)
            return
        for cache in self.warmup_caches:
            cache.adopt_before_load(self, self.d2d)
        with set_current_vllm_config(self.vllm_config):
            ensure_model_parallel_initialized(self.tp_size, self.pp_size)
            with single_pp_stage() if self.is_draft else contextlib.nullcontext():
                self.model = self.get_model()
        # In-place reloads use configs already normalized by model construction.
        self.loaded_rope_parameters_hash = hash_rope_parameters(self.model_config)
        # Loading and post-processing leave freed transients in the caching
        # allocator; return them so engines sharing the GPU can use them.
        gc.collect()
        torch.accelerator.empty_cache()
        logger.info(
            "Weight cache %s daemon rank %d loaded model (%s GiB allocated, "
            "%s GiB reserved)",
            self.role,
            self.global_rank,
            format_gib(torch.accelerator.memory_allocated()),
            format_gib(torch.accelerator.memory_reserved()),
        )

    def release_distributed(self) -> None:
        """Release process groups and BLAS workspaces after loading and warmup.

        Serving only exports CUDA IPC handles, so the communicators (NCCL,
        custom all-reduce and FlashInfer workspaces) and cached BLAS scratch
        would otherwise pin GPU memory for the daemon's lifetime.
        """
        free_before, _ = torch.accelerator.get_memory_info()
        # No rank may tear down a communicator while a peer is still in a
        # warmup collective on it.
        get_world_group().barrier()
        # Created lazily by warmup's all-reduce fusion and owned by no group.
        destroy_fi_ar_workspace()
        destroy_model_parallel()
        destroy_distributed_environment()
        gc.collect()
        if current_platform.is_cuda():
            # ATen caches these separately from the model's exported storages.
            clear_workspaces = getattr(torch._C, "_cuda_clearCublasWorkspaces", None)
            if clear_workspaces is not None:
                clear_workspaces()
            else:
                logger.debug("PyTorch does not expose BLAS workspace cleanup")
        torch.accelerator.empty_cache()
        free_after, _ = torch.accelerator.get_memory_info()
        logger.info(
            "Weight cache %s daemon rank %d released startup resources (%s GiB freed)",
            self.role,
            self.global_rank,
            format_gib(free_after - free_before),
        )

    def get_model(self) -> torch.nn.Module:
        """Load the daemon's model, composed from the configured loader.

        Runs the quantization check after model creation but before the slow
        weight load, so an unsupported method fails fast. Online quantization
        always fails the check, so load_model's finalize step for it is
        unnecessary here.
        """
        vllm_config = self.vllm_config
        model_config = self.model_config
        load_config = vllm_config.load_config
        loader = get_model_loader(load_config)
        device_config = vllm_config.device_config
        target_device = torch.device(
            device_config.device if load_config.device is None else load_config.device
        )
        # Attention backends may take workspace at construction, like in the
        # engine's worker init.
        init_workspace_manager(target_device)
        if isinstance(loader, PeerModelLoader):
            # The daemon is both ends of the weights kind: it fetches this
            # rank's shard from its source over its own D2D endpoint, and
            # the loader installs the payload into a model.
            payload = self.d2d.fetch(self.weight_seed, self)
            # The weights kind answers a payload or an error, never a miss.
            assert payload is not None
            return loader.load_model(
                vllm_config, model_config, payload=payload, transport=self.d2d
            )
        with set_default_torch_dtype(model_config.dtype):
            with target_device:
                model = loader.create_model(vllm_config, model_config)
            check_ipc_quant_support(model)
            loader.load_weights(model, model_config)
            process_weights_after_loading(model, model_config, target_device)
        return model.eval()

    def serve_forever(self, ready_callback: Callable[[], None] | None = None) -> None:
        """Serve requests until terminated.

        The socket is only bound once the model is fully cached, so clients
        get a connection error (and fall back to disk) until the daemon is
        ready.

        Args:
            ready_callback: Invoked once the socket is bound and listening,
                so the launcher can report overall readiness.

        """

        def _shutdown(signum, frame):
            raise SystemExit(128 + signum)

        signal.signal(signal.SIGINT, _shutdown)
        signal.signal(signal.SIGTERM, _shutdown)
        socket_path = self._socket_path
        ensure_private_socket_dir(
            os.path.dirname(socket_path), strict_perms=self.socket_dir is None
        )
        # Hold an exclusive per-GPU lock for the daemon's lifetime so a second
        # daemon cannot remove this daemon's live socket and hijack the path.
        lock_fd = self._acquire_gpu_lock(socket_path)
        if os.path.exists(socket_path):
            os.unlink(socket_path)
        server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        server.bind(socket_path)
        os.chmod(socket_path, 0o600)
        server.listen()
        self.d2d.start_listener()
        logger.info(
            "Weight cache %s daemon rank %d serving on %s",
            self.role,
            self.global_rank,
            socket_path,
        )
        if ready_callback is not None:
            ready_callback()

        try:
            while True:
                conn, _ = server.accept()
                self._serve_connection(conn)
        finally:
            release_error = self._shutdown(server, socket_path, lock_fd)
            if release_error:
                raise release_error[0]

    def _serve_connection(self, conn: socket.socket) -> None:
        with conn:
            try:
                verify_peer_is_owner(conn)
                self._handle_connection(conn)
            except (ConnectionError, EOFError):
                logger.warning("Client disconnected mid-request")
            except Exception as e:
                # Report the error back instead of just closing the
                # socket, but don't let it take the daemon down.
                logger.exception("Error handling weight cache client; continuing")
                with contextlib.suppress(OSError):
                    send_msg(conn, {"status": "error", "message": str(e)})

    def _release_weights_on_shutdown(self, release_error: list[Exception]) -> None:
        try:
            self.weight_seed.release()
        except Exception as e:
            release_error.append(e)
            logger.exception("Error releasing weight cache during shutdown")

    def _drain_shutdown_clients(
        self, server: socket.socket, release_thread: threading.Thread
    ) -> None:
        shutdown_clients = []
        # Keep accepting local lease acknowledgements until the source
        # registrations and weights are safe to tear down.
        while release_thread.is_alive():
            try:
                conn, _ = server.accept()
            except TimeoutError:
                continue
            conn.settimeout(5.0)
            client_thread = threading.Thread(
                target=self._serve_connection,
                args=(conn,),
                name="vllm-weight-cache-shutdown-client",
                daemon=True,
            )
            shutdown_clients.append(client_thread)
            client_thread.start()
        release_thread.join()
        for client_thread in shutdown_clients:
            client_thread.join(timeout=5.0)

    def _shutdown(
        self, server: socket.socket, socket_path: str, lock_fd: int
    ) -> list[Exception]:
        release_error: list[Exception] = []
        release_thread = threading.Thread(
            target=self._release_weights_on_shutdown,
            args=(release_error,),
            name="vllm-weight-cache-shutdown-release",
        )
        release_thread.start()
        server.settimeout(0.5)
        try:
            self._drain_shutdown_clients(server, release_thread)
        finally:
            server.close()
            for watcher in self._watchers:
                watcher.close()
            self.d2d.close()
            if os.path.exists(socket_path):
                os.unlink(socket_path)
            os.close(lock_fd)
        return release_error

    def _acquire_gpu_lock(self, socket_path: str) -> int:
        """Take an exclusive lock guarding this GPU's socket path.

        The lock is advisory and released automatically when the daemon exits
        (or crashes), so a stale socket is only ever removed by whoever owns
        the lock. A running daemon holding it makes a second daemon fail fast
        instead of clobbering the live socket.
        """
        lock_fd = os.open(f"{socket_path}.lock", os.O_CREAT | os.O_RDWR, 0o600)
        try:
            fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
        except OSError as e:
            os.close(lock_fd)
            raise WeightCacheUnavailableError(
                f"Another weight cache daemon already owns {socket_path}"
            ) from e
        return lock_fd

    @property
    def _socket_path(self) -> str:
        return get_socket_path(
            get_current_device_uuid(),
            self.socket_dir,
            is_draft=self.is_draft,
        )

    @property
    def has_weights(self) -> bool:
        return self.model is not None

    def _export_state(self) -> tuple[dict[str, TensorEntry], dict[str, str], dict]:
        """Export this rank's tensors, aliases and flags for one client."""
        assert self.model is not None
        entries, aliases = export_entries(self.model)
        return entries, aliases, export_model_attrs(self.model)

    def _seed_state(
        self,
    ) -> tuple[dict[str, torch.Tensor], dict[str, TensorKind], dict[str, str], dict]:
        """This rank's canonical tensors, their kinds, aliases and flags."""
        assert self.model is not None
        tensors, kinds, aliases = collect_tensors(self.model)
        return tensors, kinds, aliases, export_model_attrs(self.model)

    def _handle_connection(self, conn: socket.socket) -> None:
        request = recv_msg(conn)
        cmd = request.get("cmd")
        if cmd == "get_state":
            with self._state_lock:
                self._handle_get_state(conn, request)
        elif cmd == "get_memory":
            with self._state_lock:
                send_msg(
                    conn,
                    {
                        "status": "ok",
                        "memory_bytes": exported_weight_bytes(self.model),
                        # The launcher maps source GPUs to local device indices
                        # when it places a same-host mirror.
                        "gpu_uuid": get_current_device_uuid(),
                    },
                )
        elif cmd == "seed":
            # A same-host peer's D2D request; the pickle plane can carry the
            # peer_ipc seed's CUDA IPC handles, which JSON cannot.
            send_msg(conn, self.d2d.handle_request(request))
        elif cmd == "release":
            release_conn = conn.dup()
            threading.Thread(
                target=self._handle_release,
                args=(release_conn,),
                name="vllm-weight-cache-release",
                daemon=True,
            ).start()
        elif cmd == "watch":
            with self._state_lock:
                self._handle_watch(conn)
        else:
            send_msg(conn, {"status": "error", "message": f"Unknown command {cmd!r}"})

    def _handle_get_state(self, conn: socket.socket, request: dict) -> None:
        client_key = request.get("cache_key")
        if not isinstance(client_key, WeightCacheKey):
            send_msg(conn, {"status": "error", "message": "Missing cache_key"})
            return
        mismatched = self.cache_key.mismatched_fields(client_key)
        if (
            request.get("is_reload", False)
            and "rope_parameters_hash" in mismatched
            and client_key.rope_parameters_hash == self.loaded_rope_parameters_hash
        ):
            mismatched.remove("rope_parameters_hash")
        if mismatched:
            logger.warning("WeightCacheKey mismatch on fields: %s", mismatched)
            send_msg(conn, {"status": "mismatch", "fields": mismatched})
            return
        if not self.has_weights:
            send_msg(conn, {"status": "error", "message": "Weights were released"})
            return
        entries, aliases, attrs = self._export_state()
        send_msg(
            conn,
            {
                "status": "ok",
                "entries": entries,
                "aliases": aliases,
                "attrs": attrs,
                "gpu_uuid": get_current_device_uuid(),
            },
        )
        logger.info_once(
            "Weight cache %s daemon rank %d sent %d tensors (+%d aliases) to engine",
            self.role,
            self.global_rank,
            len(entries),
            len(aliases),
        )

    def _handle_watch(self, conn: socket.socket) -> None:
        """Hold the connection open until this daemon exits.

        Zero-copy engines keep serving from the mapped weights after the
        daemon exits; the closed connection tells them their daemon is gone.
        """
        self._prune_watchers()
        # serve_forever closes conn after the request; keep a duplicate.
        self._watchers.append(conn.dup())
        send_msg(conn, {"status": "ok"})

    def _prune_watchers(self) -> None:
        """Close the watch connections of engines that have exited."""
        alive = []
        for watcher in self._watchers:
            # Engines never write after the request, so a readable watcher
            # has been closed by its engine.
            readable, _, _ = select.select([watcher], [], [], 0)
            if readable:
                watcher.close()
            else:
                alive.append(watcher)
        self._watchers = alive

    def _handle_release(self, conn: socket.socket) -> None:
        with conn:
            try:
                # Unregister from the NIC before the memory returns to the driver.
                self.weight_seed.release()
                with self._state_lock:
                    self.model = None
                    torch.accelerator.empty_cache()
                logger.info(
                    "Weight cache %s daemon rank %d released cached weights",
                    self.role,
                    self.global_rank,
                )
                send_msg(conn, {"status": "ok"})
            except Exception as e:
                logger.exception("Error releasing weight cache")
                with contextlib.suppress(OSError):
                    send_msg(conn, {"status": "error", "message": str(e)})

_acquire_gpu_lock(socket_path)

Take an exclusive lock guarding this GPU's socket path.

The lock is advisory and released automatically when the daemon exits (or crashes), so a stale socket is only ever removed by whoever owns the lock. A running daemon holding it makes a second daemon fail fast instead of clobbering the live socket.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _acquire_gpu_lock(self, socket_path: str) -> int:
    """Take an exclusive lock guarding this GPU's socket path.

    The lock is advisory and released automatically when the daemon exits
    (or crashes), so a stale socket is only ever removed by whoever owns
    the lock. A running daemon holding it makes a second daemon fail fast
    instead of clobbering the live socket.
    """
    lock_fd = os.open(f"{socket_path}.lock", os.O_CREAT | os.O_RDWR, 0o600)
    try:
        fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError as e:
        os.close(lock_fd)
        raise WeightCacheUnavailableError(
            f"Another weight cache daemon already owns {socket_path}"
        ) from e
    return lock_fd

_export_state()

Export this rank's tensors, aliases and flags for one client.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _export_state(self) -> tuple[dict[str, TensorEntry], dict[str, str], dict]:
    """Export this rank's tensors, aliases and flags for one client."""
    assert self.model is not None
    entries, aliases = export_entries(self.model)
    return entries, aliases, export_model_attrs(self.model)

_handle_watch(conn)

Hold the connection open until this daemon exits.

Zero-copy engines keep serving from the mapped weights after the daemon exits; the closed connection tells them their daemon is gone.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _handle_watch(self, conn: socket.socket) -> None:
    """Hold the connection open until this daemon exits.

    Zero-copy engines keep serving from the mapped weights after the
    daemon exits; the closed connection tells them their daemon is gone.
    """
    self._prune_watchers()
    # serve_forever closes conn after the request; keep a duplicate.
    self._watchers.append(conn.dup())
    send_msg(conn, {"status": "ok"})

_prune_watchers()

Close the watch connections of engines that have exited.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _prune_watchers(self) -> None:
    """Close the watch connections of engines that have exited."""
    alive = []
    for watcher in self._watchers:
        # Engines never write after the request, so a readable watcher
        # has been closed by its engine.
        readable, _, _ = select.select([watcher], [], [], 0)
        if readable:
            watcher.close()
        else:
            alive.append(watcher)
    self._watchers = alive

_seed_state()

This rank's canonical tensors, their kinds, aliases and flags.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _seed_state(
    self,
) -> tuple[dict[str, torch.Tensor], dict[str, TensorKind], dict[str, str], dict]:
    """This rank's canonical tensors, their kinds, aliases and flags."""
    assert self.model is not None
    tensors, kinds, aliases = collect_tensors(self.model)
    return tensors, kinds, aliases, export_model_attrs(self.model)

get_model()

Load the daemon's model, composed from the configured loader.

Runs the quantization check after model creation but before the slow weight load, so an unsupported method fails fast. Online quantization always fails the check, so load_model's finalize step for it is unnecessary here.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def get_model(self) -> torch.nn.Module:
    """Load the daemon's model, composed from the configured loader.

    Runs the quantization check after model creation but before the slow
    weight load, so an unsupported method fails fast. Online quantization
    always fails the check, so load_model's finalize step for it is
    unnecessary here.
    """
    vllm_config = self.vllm_config
    model_config = self.model_config
    load_config = vllm_config.load_config
    loader = get_model_loader(load_config)
    device_config = vllm_config.device_config
    target_device = torch.device(
        device_config.device if load_config.device is None else load_config.device
    )
    # Attention backends may take workspace at construction, like in the
    # engine's worker init.
    init_workspace_manager(target_device)
    if isinstance(loader, PeerModelLoader):
        # The daemon is both ends of the weights kind: it fetches this
        # rank's shard from its source over its own D2D endpoint, and
        # the loader installs the payload into a model.
        payload = self.d2d.fetch(self.weight_seed, self)
        # The weights kind answers a payload or an error, never a miss.
        assert payload is not None
        return loader.load_model(
            vllm_config, model_config, payload=payload, transport=self.d2d
        )
    with set_default_torch_dtype(model_config.dtype):
        with target_device:
            model = loader.create_model(vllm_config, model_config)
        check_ipc_quant_support(model)
        loader.load_weights(model, model_config)
        process_weights_after_loading(model, model_config, target_device)
    return model.eval()

release_distributed()

Release process groups and BLAS workspaces after loading and warmup.

Serving only exports CUDA IPC handles, so the communicators (NCCL, custom all-reduce and FlashInfer workspaces) and cached BLAS scratch would otherwise pin GPU memory for the daemon's lifetime.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def release_distributed(self) -> None:
    """Release process groups and BLAS workspaces after loading and warmup.

    Serving only exports CUDA IPC handles, so the communicators (NCCL,
    custom all-reduce and FlashInfer workspaces) and cached BLAS scratch
    would otherwise pin GPU memory for the daemon's lifetime.
    """
    free_before, _ = torch.accelerator.get_memory_info()
    # No rank may tear down a communicator while a peer is still in a
    # warmup collective on it.
    get_world_group().barrier()
    # Created lazily by warmup's all-reduce fusion and owned by no group.
    destroy_fi_ar_workspace()
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    if current_platform.is_cuda():
        # ATen caches these separately from the model's exported storages.
        clear_workspaces = getattr(torch._C, "_cuda_clearCublasWorkspaces", None)
        if clear_workspaces is not None:
            clear_workspaces()
        else:
            logger.debug("PyTorch does not expose BLAS workspace cleanup")
    torch.accelerator.empty_cache()
    free_after, _ = torch.accelerator.get_memory_info()
    logger.info(
        "Weight cache %s daemon rank %d released startup resources (%s GiB freed)",
        self.role,
        self.global_rank,
        format_gib(free_after - free_before),
    )

serve_forever(ready_callback=None)

Serve requests until terminated.

The socket is only bound once the model is fully cached, so clients get a connection error (and fall back to disk) until the daemon is ready.

Parameters:

  • ready_callback

    (Callable[[], None] | None, default: None ) –

    Invoked once the socket is bound and listening, so the launcher can report overall readiness.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def serve_forever(self, ready_callback: Callable[[], None] | None = None) -> None:
    """Serve requests until terminated.

    The socket is only bound once the model is fully cached, so clients
    get a connection error (and fall back to disk) until the daemon is
    ready.

    Args:
        ready_callback: Invoked once the socket is bound and listening,
            so the launcher can report overall readiness.

    """

    def _shutdown(signum, frame):
        raise SystemExit(128 + signum)

    signal.signal(signal.SIGINT, _shutdown)
    signal.signal(signal.SIGTERM, _shutdown)
    socket_path = self._socket_path
    ensure_private_socket_dir(
        os.path.dirname(socket_path), strict_perms=self.socket_dir is None
    )
    # Hold an exclusive per-GPU lock for the daemon's lifetime so a second
    # daemon cannot remove this daemon's live socket and hijack the path.
    lock_fd = self._acquire_gpu_lock(socket_path)
    if os.path.exists(socket_path):
        os.unlink(socket_path)
    server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    server.bind(socket_path)
    os.chmod(socket_path, 0o600)
    server.listen()
    self.d2d.start_listener()
    logger.info(
        "Weight cache %s daemon rank %d serving on %s",
        self.role,
        self.global_rank,
        socket_path,
    )
    if ready_callback is not None:
        ready_callback()

    try:
        while True:
            conn, _ = server.accept()
            self._serve_connection(conn)
    finally:
        release_error = self._shutdown(server, socket_path, lock_fd)
        if release_error:
            raise release_error[0]

warmup()

Run each warmup cache's pass (adopting remote caches via D2D).

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def warmup(self) -> None:
    """Run each warmup cache's pass (adopting remote caches via D2D)."""
    for cache in self.warmup_caches:
        cache.warmup(self, self.d2d)

_reject_unsupported_parallelism(parallel_config)

Reject placements the daemon cannot map.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def _reject_unsupported_parallelism(parallel_config: ParallelConfig) -> None:
    """Reject placements the daemon cannot map."""
    dp_size = parallel_config.data_parallel_size
    tp_size = parallel_config.tensor_parallel_size
    pp_size = parallel_config.pipeline_parallel_size
    if parallel_config.nnodes > 1:
        world_size = dp_size * pp_size * tp_size
        if world_size % parallel_config.nnodes != 0:
            raise ValueError(
                f"--nnodes ({parallel_config.nnodes}) must evenly divide the "
                f"daemon world size ({world_size} = data-parallel-size "
                f"{dp_size} x pipeline-parallel-size {pp_size} x "
                f"tensor-parallel-size {tp_size})"
            )
        return
    if dp_size == 1:
        return
    end_rank = (
        parallel_config.data_parallel_rank + parallel_config.data_parallel_size_local
    )
    if end_rank > dp_size:
        raise ValueError(
            "--data-parallel-start-rank + --data-parallel-size-local "
            f"({end_rank}) exceeds --data-parallel-size ({dp_size})"
        )

daemon_serves_model(is_draft, pp_rank, pp_size)

Whether the daemon at this placement loads and serves a model.

The engine runs the draft model only on the last pipeline stage. Draft daemons on other stages join the group's rendezvous and then exit.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def daemon_serves_model(is_draft: bool, pp_rank: int, pp_size: int) -> bool:
    """Whether the daemon at this placement loads and serves a model.

    The engine runs the draft model only on the last pipeline stage. Draft
    daemons on other stages join the group's rendezvous and then exit.
    """
    return not is_draft or pp_rank == pp_size - 1

get_draft_daemon_config(vllm_config)

VllmConfig for the draft daemon group

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def get_draft_daemon_config(vllm_config: VllmConfig) -> VllmConfig | None:
    """VllmConfig for the draft daemon group"""
    if not is_draft_model_cacheable(vllm_config.speculative_config):
        return None
    assert vllm_config.speculative_config is not None
    return vllm_config.speculative_config.apply_draft_overrides(vllm_config)

plan_local_ranks(parallel_config)

(local_rank, dp_rank, pp_rank, tp_rank) for every local daemon.

Global ranks enumerate DP, PP, then TP: global = dp_rank * pp_size * tp_size + pp_rank * tp_size + tp_rank. With --nnodes the engine hands each node a contiguous block of global ranks, so node r serves node_rank * local + i; without it a launcher's block starts at --data-parallel-start-rank * tp_size.

Source code in vllm/model_executor/model_loader/weight_cache/daemon.py
def plan_local_ranks(
    parallel_config: ParallelConfig,
) -> list[tuple[int, int, int, int]]:
    """`(local_rank, dp_rank, pp_rank, tp_rank)` for every local daemon.

    Global ranks enumerate DP, PP, then TP: `global = dp_rank * pp_size *
    tp_size + pp_rank * tp_size + tp_rank`.
    With `--nnodes` the engine hands each node a contiguous block of global
    ranks, so node `r` serves `node_rank * local + i`; without it a
    launcher's block starts at `--data-parallel-start-rank * tp_size`.
    """
    tp_size = parallel_config.tensor_parallel_size
    pp_size = parallel_config.pipeline_parallel_size
    world_size = parallel_config.data_parallel_size * pp_size * tp_size
    if parallel_config.nnodes > 1:
        local_world_size = world_size // parallel_config.nnodes
        base = parallel_config.node_rank * local_world_size
    else:
        local_world_size = parallel_config.data_parallel_size_local * pp_size * tp_size
        base = parallel_config.data_parallel_rank * pp_size * tp_size
    placements = []
    for local_rank in range(local_world_size):
        global_rank = base + local_rank
        rank_in_dp = global_rank % (pp_size * tp_size)
        dp_rank = global_rank // (pp_size * tp_size)
        pp_rank, tp_rank = divmod(rank_in_dp, tp_size)
        placements.append((local_rank, dp_rank, pp_rank, tp_rank))
    return placements