Skip to content

FLy Verification

FLy is an approximate verification policy that can defer a native rejection at a high-entropy position when the following draft tokens remain aligned with native verification.

Warning

FLy is intentionally lossy. Unlike standard speculative decoding, it does not preserve the target distribution and may degrade model outputs.

Usage

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-8B",
    speculative_config={
        "method": "draft_model",
        "model": "Qwen/Qwen3-0.6B",
        "num_speculative_tokens": 8,
        "rejection_sample_method": "fly",
        "fly_window_size": 6,
        "fly_entropy_threshold": 0.3,
        "fly_entropy_top_k": 3,
    },
)

FLy requires at least two speculative tokens. fly_window_size is the number of subsequent native acceptance decisions checked; it must be smaller than num_speculative_tokens and defaults to min(6, num_speculative_tokens - 1).

The entropy gate takes the fly_entropy_top_k largest processed target scores (three by default), renormalizes them into a distribution over those candidates, and divides its entropy by log(fly_entropy_top_k). The gate value is therefore in [0, 1], from 0 when the target is certain to 1 when the top-k candidates are equally likely, and fly_entropy_threshold keeps the same meaning for every fly_entropy_top_k. fly_entropy_top_k must be positive and is capped at the vocabulary size; with a top-k of 1 the gate value is always 0, so FLy defers nothing unless the threshold is 0.

FLy supports greedy draft sampling with target-only acceptance and probabilistic draft sampling with standard p/q acceptance. Token-Level Intersection can be used for cross-vocabulary draft models with greedy draft sampling, but is incompatible with use_local_argmax_reduction.

FLy has been validated with the MTP, EAGLE, DFlash, and DSpark proposers.

Measuring FLy Cost with Synthetic Acceptance

rejection_sample_method="synthetic" fixes the acceptance length, which is how benchmarks pin speculative decoding to a measured acceptance length. Setting fly_window_size together with synthetic also runs FLy's per-step entropy computation during verification but never defers a rejection, so the acceptance length stays exactly the synthetic one. A benchmark pinned to FLy's measured acceptance length then also pays FLy's verification cost:

speculative_config={
    "method": "dspark",
    "num_speculative_tokens": 8,
    "rejection_sample_method": "synthetic",
    "synthetic_acceptance_length": 4.8,
    "fly_window_size": 2,
    "fly_entropy_threshold": 0.3,
    "fly_entropy_top_k": 3,
}

The same fly_window_size < num_speculative_tokens constraint applies. This mode is only supported by ModelRunner V2.

Validated Configurations

The following evaluation was run with the draft_model proposer of the now-removed V1 model runner; current validation is summarized below. Configurations not listed here are not necessarily unsupported, but have not yet been experimentally validated.

FLy was faster than Standard SD in all 40 primary configurations. Across all four target/draft pairs, the 29 configurations with ratio-based quality measurements had a minimum quality retention of 97.9%.

Accelerator Model configuration Workloads Sampling Measured speedup
AMD MI355X Llama-3.1-70B-Instruct / Llama-3.1-8B-Instruct 10-benchmark suite Temperature 0, greedy draft 1.092x–1.335x vs Standard SD
AMD MI355X DeepSeek-R1-Distill-Llama-70B / DeepSeek-R1-Distill-Llama-8B 10-benchmark suite Temperature 0, greedy draft 1.147x–1.467x vs Standard SD
AMD MI355X Llama-3.1-405B-Instruct-FP8-KV / Llama-3.1-8B-Instruct 10-benchmark suite Temperature 0, greedy draft 1.189x–1.938x vs Standard SD
AMD MI355X Qwen3-235B-A22B-Thinking-2507-FP8 / Qwen3-8B 10-benchmark suite Temperature 0, greedy draft 1.067x–1.478x vs Standard SD
NVIDIA B300 DeepSeek-R1-Distill-Llama-70B / DeepSeek-R1-Distill-Llama-8B MATH-500, HumanEval, HLE Temperature 0, greedy draft 1.26x–1.43x vs Standard SD
NVIDIA B300 Qwen2.5-72B-Instruct / Qwen2.5-7B-Instruct GSM8K, MGSM, GPQA-Diamond, Spec-Bench Temperature 0, greedy draft 1.07x–1.32x vs Standard SD
AMD MI300X Kimi-K2.5 (1T) Random 8K input / 1K output DFlash comparison >1.09x vs DFlash

The 10-benchmark suite contains GSM8K, MGSM, MATH-500, AIME 2024, AIME 2025, GPQA-Diamond, HumanEval, MBPP, Spec-Bench, and HLE. Subsets of GSM8K and MATH-500 were also validated with temperature 1 and probabilistic draft sampling.

ModelRunner V2

MRV2 runs used greedy draft sampling, fly_window_size=2, fly_entropy_threshold=0.3, and fly_entropy_top_k=3. Decode throughput was measured with 1,024 output tokens per request at batch sizes 1, 4, 8, 16, 32, and 64, excluding prefill.

Accelerator Model Speculative method Draft tokens (K) Measured decode speedup
AMD MI355X DeepSeek-V4-Flash-0731 DSpark 7 1.103x–1.168x vs Standard SD
AMD MI355X Qwen3.8-27B MTP 3 1.055x–1.116x vs Standard SD
AMD MI355X GLM-5.2-FP8 MTP 5 1.110x–1.229x vs Standard SD
AMD MI355X Qwen3.5-122B-A10B-FP8 MTP 8 1.251x–1.310x vs Standard SD
NVIDIA B300 DeepSeek-V4-Flash-0731 DSpark 7 1.099x–1.253x vs Standard SD
NVIDIA B300 Qwen3.8-27B MTP 3 1.041x–1.095x vs Standard SD
NVIDIA B300 GLM-5.2-FP8 MTP 5 1.120x–1.207x vs Standard SD
NVIDIA B300 Qwen3.5-122B-A10B-FP8 MTP 8 1.202x–1.273x vs Standard SD

Single-pass quality evaluations compared Target-only, Standard SD, and FLy on GSM8K (100 examples), MATH-500 (100), AIME25 (30), and GPQA-Diamond (198). Relative to Target-only, the minimum measured accuracy retention was 96.43% on AMD and 96.30% on NVIDIA; both corresponded to one fewer correct answer on AIME25.