Skip to content

vllm.models.minimax_m3.nvidia.specialized_decode.kernels.front_kernel

CuTe DSL device code of stage front (the host side is front.py).

A operand in shared memory at bucket 8, early x polling, Lamport sentinel hand-off, one weight phase. Every float operation is spelled with an explicit rounding mode (add.rn, fma.rn, *.ftz, and FMA only where intended), so the outputs do not depend on compiler contraction.

Grid: 4 * (heads + meta) CTAs, cluster 4, 512 threads, one CTA per SM (~226 KB shared memory). Per head cluster:

  1. warp 15: TMEM allocation, weight tile TMA (12 tiles of 128 x 128 E4M3, 128-byte swizzle) and scale atoms, their scale copy into TMEM (A operand in TMEM at bucket 16);
  2. clusters c < rows, warps 0..14: boundary row c: Lamport push of the partial into every rank's exchange (unless the producer pushed), poll every rank's partial, residual add, Gemma RMSNorm over the cluster (DSMEM), MXFP8 of the normed row into this rank's private hand-off buffer (two parities, sentinel-armed);
  3. every CTA: re-arm the other parity, poll its K slice of every row's MXFP8 activation into shared memory, 48 tcgen05 block-scaled MMAs (M 128 weight rows, N 8/16 rows, K 32), K-split reduce over the cluster (st.async);
  4. warps 0..3: per-head Gemma RMSNorm + NeoX RoPE, FP8 Q / index-Q, FP8 K/V and index-K cache inserts;
  5. optionally one more cluster writes the step metadata of the layer.

Compile-time parameters: the bucket (MAXT 8 / 16), rows (0 = runtime), the TP world, the per-rank head geometry (nq, nkv, niq: TP4 16/1/1, TP8 8/1/1), the rotary dim, the page size and the exchange mode.

Functions:

  • asl –

    TMEM layout: bucket 8 reads A (weights) straight from shared memory; bucket 16 stages A into TMEM.

  • bf16_round –

    float(bf16(x)) (RN).

  • bf16x2 –

    bf16(lo) | bf16(hi) << 16 (RN).

  • e4m3x2 –

    E4M3 (RN, satfinite) of lo (byte 0) and hi (byte 1), zero-extended.

  • f1 –

    One-operand op: "rsqrt.approx.ftz", "rsqrt.approx", "rcp.approx.ftz", "abs".

  • fop –

    op.f32 a, b, e.g. op = "add.rn", "mul.rn.ftz", "div.full.ftz", "max", "min".

  • ldg_v4 –

    ld.global.v4.b32, zeros when pred is 0.

  • ldv_v4_pred –

    ld.volatile.global.v4 when pred, else four EMPTY words is not wanted: zeros (the caller masks the poll).

  • mbar_wait –

    Spin on mbarrier.try_wait.parity (sem: "" or ".acquire.cluster").

  • poll_empty –

    Re-poll a 16-byte exchange item (first load w0..w3) while pred and any word is EMPTY; traps after SPIN_CAP.

  • poll_x –

    The Lamport hand-off poll of one thread: up to four 16-byte activation items (bits 0..3 of pend) and one scale

  • ptx –

    A side-effecting PTX statement without operands.

  • ptx_p –

    A PTX statement executed when pred != 0.

  • ptx_rp –

    A PTX statement with one register operand ($0), executed when pred != 0.

  • sdesc –

    tcgen05 shared memory descriptor of a 16-byte aligned shared address.

  • stg_v4 –

    st.global.v4.b32 (explicit .global: a generic store to an NVLS multicast address faults).

  • tc_ld16 –

    Two tcgen05.ld 32x32b.x8 (columns c, c + 8) and one wait.

  • tc_ld8 –

    tcgen05.ld 32x32b.x8 and its wait in one statement (the values exist only after the wait).

  • tc_mma_ss –

    Block-scaled MMA, A and B from shared memory descriptors.

  • tc_mma_ts –

    Block-scaled MMA, A from TMEM, B from a shared memory descriptor.

  • ue8m0 –

    UE8M0 of x rounded up, satfinite (the low byte of cvt.rp.satfinite.ue8m0x2.f32 with a 0 partner).

  • ue8m0_value –

    The float value of a UE8M0 code (through bf16, as __nv_fp8_e8m0 -> float).

  • zero_byte_mask –

    0x80 in every byte of t that is zero, else 0 (exact per byte).

_boundary(MAXT, WORLD, PREPUSHED, t, crank, tid, lane, warp, bx, warp_sq, cluster_sq, xbar, x, x2, fc2, perm, scales, topk, scale_fp32, residual, norm_w, eps, out_residual, ws_uc, flags, xq, xsf, ctr, rank, npeers, single_round, peers)

Boundary row t (FlashInfer one-shot all-reduce + norm rounding), on warps 0..14; 192 threads do the math.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
def _boundary(
    MAXT,
    WORLD,
    PREPUSHED,
    t,
    crank,
    tid,
    lane,
    warp,
    bx,
    warp_sq,
    cluster_sq,
    xbar,
    x,
    x2,
    fc2,
    perm,
    scales,
    topk,
    scale_fp32,
    residual,
    norm_w,
    eps,
    out_residual,
    ws_uc,
    flags,
    xq,
    xsf,
    ctr,
    rank,
    npeers,
    single_round,
    peers,
):
    """Boundary row t (FlashInfer one-shot all-reduce + norm rounding), on warps 0..14; 192 threads do the math."""
    t0 = b2i(tid == I32(0))
    ptx_rp("mbarrier.init.shared::cta.b64 [$0], 1;", xbar, t0)
    ptx_p("fence.mbarrier_init.release.cluster;", t0)
    ptx("barrier.cluster.arrive.relaxed.aligned;")
    c0 = crank * I32(KS) + tid * I32(8)
    active = tid < I32(BTH)
    act = b2i(active)
    c0c = cutlass.select_(active, c0, I32(0))
    half_words = MAXT * WORLD * (H // 2)
    fl = t * I32(CL) + crank
    if not PREPUSHED:
        ptx("griddepcontrol.wait;")
    parity = ldv_b32(flags + I64(fl * I32(4))) & I32(1)
    call = ldg_b32(ctr + I64(bx * I32(4)), I32(1))
    xqo = xq + I64(call & I32(1)) * I64(XQ_BUF)
    xsfo = xsf + I64(call & I32(1)) * I64(XSF_BUF)
    row = I64(t) * I64(H) + I64(c0c)
    rraw = ldg_v4(residual + row * I64(2), act)
    wraw = ldg_v4(norm_w + I64(c0c) * I64(2), act)
    if not PREPUSHED:
        has_fin = fc2 != I64(0)
        has_x2 = x2 != I64(0)
        # finalize: x = sum_k scales[t, k] * fc2[perm[t, k]] (+ x2), fp32, rounded once
        f = [F32(0.0)] * 8
        for k in range(8):
            kin = active & has_fin & (I32(k) < topk)
            e_ = t * topk + I32(k)
            prow = ldg_b32(perm + I64(e_) * I64(4), b2i(kin))
            sc32 = ldg_b32(scales + I64(e_) * I64(4), b2i(kin & (scale_fp32 != I32(0))))
            sc16 = ldg_u16(scales + I64(e_) * I64(2), b2i(kin & (scale_fp32 == I32(0))))
            sc = cutlass.select_(
                scale_fp32 != I32(0), sc32.bitcast(F32), (sc16 << I32(16)).bitcast(F32)
            )
            valid = kin & (prow >= I32(0))
            rv = ldg_v4(fc2 + (I64(prow) * I64(H) + I64(c0c)) * I64(2), b2i(valid))
            for q in range(4):
                f[2 * q] = cutlass.select_(
                    valid, ffma(bf_lo(rv[q]), sc, f[2 * q]), f[2 * q]
                )
                f[2 * q + 1] = cutlass.select_(
                    valid, ffma(bf_hi(rv[q]), sc, f[2 * q + 1]), f[2 * q + 1]
                )
        xv2 = ldg_v4(x2 + row * I64(2), b2i(active & has_x2))
        fin_w = []
        for q in range(4):
            a0 = cutlass.select_(
                has_x2, fop("add.rn", bf_lo(xv2[q]), f[2 * q]), f[2 * q]
            )
            a1 = cutlass.select_(
                has_x2, fop("add.rn", bf_hi(xv2[q]), f[2 * q + 1]), f[2 * q + 1]
            )
            fin_w.append(bf16x2(a0, a1))
        xv = ldg_v4(x + row * I64(2), b2i(active & (has_fin == cutlass.Boolean(False))))
        x_w = []
        for q in range(4):
            s0 = bf16x2(
                fop("add.rn", bf_lo(xv[q]), bf_lo(xv2[q])),
                fop("add.rn", bf_hi(xv[q]), bf_hi(xv2[q])),
            )
            x_w.append(cutlass.select_(has_x2, s0, xv[q]))
        raw = [cutlass.select_(has_fin, fin_w[q], x_w[q]) for q in range(4)]
        # canonicalise -0 halves to +0 (0x80000000 marks an empty exchange word)
        val = []
        for q in range(4):
            w = raw[q]
            w = cutlass.select_((w & I32(0xFFFF)) == I32(0x8000), w & I32(-65536), w)
            w = cutlass.select_(
                (w & I32(-65536)) == I32(-2147483648), w & I32(0xFFFF), w
            )
            val.append(w)
        slot = (
            I64(parity) * I64(half_words)
            + (I64(t) * I64(WORLD) + I64(rank)) * I64(H // 2)
            + I64(c0c // I32(2))
        )
        for r in range(PEER_TABLE):
            if r < WORLD:
                stg_v4(
                    peers[r] + slot * I64(4),
                    val[0],
                    val[1],
                    val[2],
                    val[3],
                    b2i(active & (I32(r) < npeers)),
                )
    # poll every rank's partial of this row
    p0 = ws_uc + (
        I64(parity) * I64(half_words)
        + I64(t) * I64(WORLD * (H // 2))
        + I64(c0c // I32(2))
    ) * I64(4)
    E = I32(EMPTY)
    first = [ldv_v4_pred(p0 + I64(r * (H // 2) * 4), act) for r in range(WORLD)]
    s = [F32(0.0)] * 8
    for r in range(WORLD):
        a = p0 + I64(r * (H // 2) * 4)
        ww = poll_empty(a, first[r][0], first[r][1], first[r][2], first[r][3], act)
        stg_v4(a, E, E, E, E, act)
        for q in range(4):
            if r == 0:
                s[2 * q] = bf_lo(ww[q])
                s[2 * q + 1] = bf_hi(ww[q])
            else:
                s[2 * q] = fop("add.rn.ftz", s[2 * q], bf_lo(ww[q]))
                s[2 * q + 1] = fop("add.rn.ftz", s[2 * q + 1], bf_hi(ww[q]))
    sq = F32(0.0)
    upd = []
    for q in range(4):
        rx, ry = bf_lo(rraw[q]), bf_hi(rraw[q])
        u1 = bf16x2(
            fop("add.rn.ftz", s[2 * q], rx), fop("add.rn.ftz", s[2 * q + 1], ry)
        )
        u2 = bf16x2(
            fop("add.rn.ftz", bf16_round(s[2 * q]), rx),
            fop("add.rn.ftz", bf16_round(s[2 * q + 1]), ry),
        )
        u = cutlass.select_(single_round != I32(0), u1, u2)
        upd.append(u)
        s[2 * q] = bf_lo(u)
        s[2 * q + 1] = bf_hi(u)
        sq = ffma(s[2 * q + 1], s[2 * q + 1], ffma(s[2 * q], s[2 * q], sq, True), True)
    stg_v4(out_residual + row * I64(2), upd[0], upd[1], upd[2], upd[3], act)
    for off in range(5):
        sq = fop("add.rn.ftz", sq, shfl_xor(sq, I32(16 >> off)))
    sts_b32_pred(
        warp_sq + (warp & I32(7)) * I32(4),
        sq.bitcast(I32),
        b2i((lane == I32(0)) & active),
    )
    ptx("bar.sync 1, 480;")
    stg_b32(
        flags + I64(fl * I32(4)), parity ^ I32(1), t0
    )  # after the barrier: every boundary warp read the parity
    vv = cutlass.select_(
        tid < I32(BTH // 32), lds_f32(warp_sq + (tid & I32(7)) * I32(4)), F32(0.0)
    )
    for off in range(5):
        vv = fop("add.rn.ftz", vv, shfl_xor(vv, I32(16 >> off)))
    ptx("barrier.cluster.wait.aligned;")
    ptx_rrp(
        "mbarrier.arrive.expect_tx.shared::cta.b64 _, [$0], $1;", xbar, I32(CL * 4), t0
    )
    for c in range(CL):
        st_async(
            mapa(cluster_sq + crank * I32(4), I32(c)),
            vv.bitcast(I32),
            mapa(xbar, I32(c)),
            t0,
        )
    mbar_wait(xbar, I32(0), ".acquire.cluster")
    total = F32(0.0)
    for c in range(CL):
        total = fop("add.rn.ftz", total, lds_f32(cluster_sq + I32(4 * c)))
    inv = f1(
        "rsqrt.approx.ftz",
        fop("add.rn.ftz", fop("div.full.ftz", total, F32(float(H))), eps),
    )
    for q in range(4):
        wx = fop("add.rn.ftz", F32(1.0), bf_lo(wraw[q]))
        wy = fop("add.rn.ftz", F32(1.0), bf_hi(wraw[q]))
        n = bf16x2(
            fop("mul.rn.ftz", fop("mul.rn.ftz", s[2 * q], inv), wx),
            fop("mul.rn.ftz", fop("mul.rn.ftz", s[2 * q + 1], inv), wy),
        )
        s[2 * q] = bf_lo(n)
        s[2 * q + 1] = bf_hi(n)
    # MXFP8 of the normed row (trtllm mxfp8_quantize order)
    amax = F32(0.0)
    for z in range(8):
        amax = fop("max", amax, f1("abs", s[z]))
    amax = fop("max", amax, shfl_xor(amax, I32(1)))
    amax = fop("max", amax, shfl_xor(amax, I32(2)))
    code = ue8m0(fop("mul.rn", amax, f1("rcp.approx.ftz", F32(448.0))))
    out_scale = cutlass.select_(
        amax != F32(0.0), f1("rcp.approx.ftz", ue8m0_value(code)), F32(0.0)
    )
    packed = []
    for hh in range(2):
        lo = e4m3x2(
            fop("mul.rn", s[4 * hh], out_scale), fop("mul.rn", s[4 * hh + 1], out_scale)
        )
        hi = e4m3x2(
            fop("mul.rn", s[4 * hh + 2], out_scale),
            fop("mul.rn", s[4 * hh + 3], out_scale),
        )
        pk = lo | (hi << I32(16))
        pk = pk & (
            zero_byte_mask(pk ^ I32(-2139062144)) ^ I32(-1)
        )  # -0 (0x80) is the sentinel: store +0
        packed.append(pk)
    tile = c0c // I32(128)
    ch = (c0c % I32(128)) // I32(16)
    xo = xqo + I64(
        (tile * I32(MAX_ROWS) + t) * I32(128)
        + ((ch ^ (t & I32(7))) * I32(16))
        + (c0c % I32(16))
    )
    stg_v2(xo, packed[0], packed[1], act)
    stg_u8(
        xsfo
        + I64((c0c // I32(128)) * I32(XATOM) + t * I32(16) + (c0c // I32(32)) % I32(4)),
        code,
        b2i(active & ((tid & I32(3)) == I32(0))),
    )
    ptx("bar.sync 1, 480;")

asl(maxt)

TMEM layout: bucket 8 reads A (weights) straight from shared memory; bucket 16 stages A into TMEM.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
def asl(maxt: int) -> dict:
    """TMEM layout: bucket 8 reads A (weights) straight from shared memory; bucket 16 stages A into TMEM."""
    on = maxt <= 8
    return dict(
        on=on,
        cols=128 if on else 512,
        sfa=32 if on else 64,
        sfb=80 if on else 112,
        slots=maxt // 4 if on else 4,
        ta=128,
    )

bf16_round(x, *, loc=None, ip=None)

float(bf16(x)) (RN).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def bf16_round(x, *, loc=None, ip=None):
    """float(bf16(x)) (RN)."""
    return F32(
        _asm(
            _f32t(),
            [F32(x)],
            "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}",
            "=f,f",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

bf16x2(lo, hi, *, loc=None, ip=None)

bf16(lo) | bf16(hi) << 16 (RN).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def bf16x2(lo, hi, *, loc=None, ip=None):
    """bf16(lo) | bf16(hi) << 16 (RN)."""
    return I32(
        _asm(
            _i32t(),
            [F32(lo), F32(hi)],
            "cvt.rn.bf16x2.f32 $0, $2, $1;",
            "=r,f,f",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

e4m3x2(lo, hi, *, loc=None, ip=None)

E4M3 (RN, satfinite) of lo (byte 0) and hi (byte 1), zero-extended.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def e4m3x2(lo, hi, *, loc=None, ip=None):
    """E4M3 (RN, satfinite) of lo (byte 0) and hi (byte 1), zero-extended."""
    return I32(
        _asm(
            _i32t(),
            [F32(lo), F32(hi)],
            "{\n\t.reg .b16 h;\n\tcvt.rn.satfinite.e4m3x2.f32 h, $2, $1;\n\tcvt.u32.u16 $0, h;\n\t}",
            "=r,f,f",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

f1(op, a, *, loc=None, ip=None)

One-operand op: "rsqrt.approx.ftz", "rsqrt.approx", "rcp.approx.ftz", "abs".

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def f1(op, a, *, loc=None, ip=None):
    """One-operand op: "rsqrt.approx.ftz", "rsqrt.approx", "rcp.approx.ftz", "abs"."""
    return F32(
        _asm(_f32t(), [F32(a)], f"{op}.f32 $0, $1;", "=f,f", side=False, loc=loc, ip=ip)
    )

fop(op, a, b, *, loc=None, ip=None)

op.f32 a, b, e.g. op = "add.rn", "mul.rn.ftz", "div.full.ftz", "max", "min".

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def fop(op, a, b, *, loc=None, ip=None):
    """`op`.f32 a, b, e.g. op = "add.rn", "mul.rn.ftz", "div.full.ftz", "max", "min"."""
    return F32(
        _asm(
            _f32t(),
            [F32(a), F32(b)],
            f"{op}.f32 $0, $1, $2;",
            "=f,f,f",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

ldg_v4(addr, pred, *, loc=None, ip=None)

ld.global.v4.b32, zeros when pred is 0.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ldg_v4(addr, pred, *, loc=None, ip=None):
    """ld.global.v4.b32, zeros when pred is 0."""
    out = _asm(
        _struct(4),
        [I64(addr), I32(pred)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $5, 0;\n\tmov.b32 $0, 0;\n\tmov.b32 $1, 0;\n\tmov.b32 $2, 0;\n\t"
        "mov.b32 $3, 0;\n\t@p ld.global.v4.b32 {$0, $1, $2, $3}, [$4];\n\t}",
        "=r,=r,=r,=r,l,r",
        loc=loc,
        ip=ip,
    )
    return _outs(out, 4, I32, loc, ip)

ldv_v4_pred(addr, pred, *, loc=None, ip=None)

ld.volatile.global.v4 when pred, else four EMPTY words is not wanted: zeros (the caller masks the poll).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ldv_v4_pred(addr, pred, *, loc=None, ip=None):
    """ld.volatile.global.v4 when pred, else four EMPTY words is not wanted: zeros (the caller masks the poll)."""
    out = _asm(
        _struct(4),
        [I64(addr), I32(pred)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $5, 0;\n\tmov.b32 $0, 0;\n\tmov.b32 $1, 0;\n\tmov.b32 $2, 0;\n\t"
        "mov.b32 $3, 0;\n\t@p ld.volatile.global.v4.s32 {$0, $1, $2, $3}, [$4];\n\t}",
        "=r,=r,=r,=r,l,r",
        loc=loc,
        ip=ip,
    )
    return _outs(out, 4, I32, loc, ip)

mbar_wait(bar, parity, sem='', *, loc=None, ip=None)

Spin on mbarrier.try_wait.parity (sem: "" or ".acquire.cluster").

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def mbar_wait(bar, parity, sem="", *, loc=None, ip=None):
    """Spin on mbarrier.try_wait.parity (sem: "" or ".acquire.cluster")."""
    _asm(
        None,
        [I32(bar), I32(parity)],
        "{\n\t.reg .pred p;\n\tMBW:\n\t"
        f"mbarrier.try_wait.parity{sem}.shared::cta.b64 p, [$0], $1;\n\t@!p bra MBW;\n\t}}",
        "r,r",
        loc=loc,
        ip=ip,
    )

poll_empty(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None)

Re-poll a 16-byte exchange item (first load w0..w3) while pred and any word is EMPTY; traps after SPIN_CAP.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def poll_empty(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None):
    """Re-poll a 16-byte exchange item (first load w0..w3) while pred and any word is EMPTY; traps after SPIN_CAP."""
    out = _asm(
        _struct(4),
        [I64(addr), I32(w0), I32(w1), I32(w2), I32(w3), I32(pred)],
        "{\n\t.reg .pred p, q;\n\t.reg .b64 sp;\n\tmov.b32 $0, $5;\n\tmov.b32 $1, $6;\n\tmov.b32 $2, $7;\n\t"
        "mov.b32 $3, $8;\n\tsetp.ne.b32 q, $9, 0;\n\t@!q bra PED;\n\tmov.b64 sp, 0;\n\tPEL:\n\t"
        "setp.eq.b32 p, $0, 0x80000000;\n\tsetp.eq.or.b32 p, $1, 0x80000000, p;\n\t"
        "setp.eq.or.b32 p, $2, 0x80000000, p;\n\tsetp.eq.or.b32 p, $3, 0x80000000, p;\n\t@!p bra PED;\n\t"
        "ld.volatile.global.v4.s32 {$0, $1, $2, $3}, [$4];\n\tadd.s64 sp, sp, 1;\n\t"
        "setp.gt.s64 p, sp, 2147483648;\n\t@p trap;\n\tbra PEL;\n\tPED:\n\t}",
        "=r,=r,=r,=r,l,r,r,r,r,r",
        loc=loc,
        ip=ip,
    )
    return _outs(out, 4, I32, loc, ip)

poll_x(pend, g0, g1, g2, g3, gw, s0, s1, s2, s3, sw, *, loc=None, ip=None)

The Lamport hand-off poll of one thread: up to four 16-byte activation items (bits 0..3 of pend) and one scale word (bit 4), all loaded in one round per spin; an item is accepted once none of its bytes is the sentinel (0x80 for activations, 0xFF for scales) and is stored to shared memory.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def poll_x(pend, g0, g1, g2, g3, gw, s0, s1, s2, s3, sw, *, loc=None, ip=None):
    """The Lamport hand-off poll of one thread: up to four 16-byte activation items (bits 0..3 of pend) and one scale
    word (bit 4), all loaded in one round per spin; an item is accepted once none of its bytes is the sentinel (0x80
    for activations, 0xFF for scales) and is stored to shared memory."""
    loads = []
    checks = []
    for q in range(4):
        loads.append(
            f"and.b32 t, pend, {1 << q};\n\tsetp.ne.b32 p, t, 0;\n\t"
            f"@p ld.volatile.global.v4.s32 {{x{q}0, x{q}1, x{q}2, x{q}3}}, [${1 + q}];\n\t"
        )
        chk = f"and.b32 t, pend, {1 << q};\n\tsetp.eq.b32 p, t, 0;\n\t@p bra SK{q};\n\tmov.b32 acc, 0;\n\t"
        for j in range(4):
            chk += (
                f"xor.b32 t, x{q}{j}, 0x80808080;\n\tand.b32 z, t, 0x7F7F7F7F;\n\tadd.u32 z, z, 0x7F7F7F7F;\n\t"
                "or.b32 z, z, t;\n\tnot.b32 z, z;\n\tand.b32 z, z, 0x80808080;\n\tor.b32 acc, acc, z;\n\t"
            )
        chk += (
            f"setp.ne.b32 p, acc, 0;\n\t@p bra SK{q};\n\t"
            f"st.shared.v4.b32 [${6 + q}], {{x{q}0, x{q}1, x{q}2, x{q}3}};\n\tand.b32 pend, pend, {~(1 << q) & 0xFFFFFFFF};\n\t"
            f"SK{q}:\n\t"
        )
        checks.append(chk)
    text = (
        "{\n\t.reg .pred p;\n\t.reg .b32 pend, t, z, acc, w;\n\t.reg .b64 sp;\n\t"
        + "".join(f".reg .b32 x{q}0, x{q}1, x{q}2, x{q}3;\n\t" for q in range(4))
        + "mov.b32 pend, $0;\n\tmov.b64 sp, 0;\n\tPXL:\n\tsetp.eq.b32 p, pend, 0;\n\t@p bra PXD;\n\t"
        + "".join(loads)
        + "and.b32 t, pend, 16;\n\tsetp.ne.b32 p, t, 0;\n\t@p ld.volatile.global.u32 w, [$5];\n\t"
        + "".join(checks)
        + "and.b32 t, pend, 16;\n\tsetp.eq.b32 p, t, 0;\n\t@p bra SKW;\n\tnot.b32 t, w;\n\t"
        "and.b32 z, t, 0x7F7F7F7F;\n\tadd.u32 z, z, 0x7F7F7F7F;\n\tor.b32 z, z, t;\n\tnot.b32 z, z;\n\t"
        "and.b32 z, z, 0x80808080;\n\tsetp.ne.b32 p, z, 0;\n\t@p bra SKW;\n\tst.shared.b32 [$10], w;\n\t"
        "and.b32 pend, pend, 0xFFFFFFEF;\n\tSKW:\n\t"
        "add.s64 sp, sp, 1;\n\tsetp.gt.s64 p, sp, 2147483648;\n\t@p trap;\n\tbra PXL;\n\tPXD:\n\t}"
    )
    _asm(
        None,
        [
            I32(pend),
            I64(g0),
            I64(g1),
            I64(g2),
            I64(g3),
            I64(gw),
            I32(s0),
            I32(s1),
            I32(s2),
            I32(s3),
            I32(sw),
        ],
        text,
        "r,l,l,l,l,l,r,r,r,r,r",
        loc=loc,
        ip=ip,
    )

ptx(text, *, loc=None, ip=None)

A side-effecting PTX statement without operands.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ptx(text, *, loc=None, ip=None):
    """A side-effecting PTX statement without operands."""
    _asm(None, [], text, "", loc=loc, ip=ip)

ptx_p(text, pred, *, loc=None, ip=None)

A PTX statement executed when pred != 0.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ptx_p(text, pred, *, loc=None, ip=None):
    """A PTX statement executed when pred != 0."""
    _asm(
        None,
        [I32(pred)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $0, 0;\n\t@p " + text + "\n\t}",
        "r",
        loc=loc,
        ip=ip,
    )

ptx_rp(text, a, pred, *, loc=None, ip=None)

A PTX statement with one register operand ($0), executed when pred != 0.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ptx_rp(text, a, pred, *, loc=None, ip=None):
    """A PTX statement with one register operand ($0), executed when pred != 0."""
    _asm(
        None,
        [I32(a), I32(pred)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $1, 0;\n\t@p " + text + "\n\t}",
        "r,r",
        loc=loc,
        ip=ip,
    )

sdesc(saddr, lbo, sbo, layout)

tcgen05 shared memory descriptor of a 16-byte aligned shared address.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
def sdesc(saddr, lbo: int, sbo: int, layout: int):
    """tcgen05 shared memory descriptor of a 16-byte aligned shared address."""
    lo = I64((saddr >> I32(4)) & I32(0x3FFF))
    return lo | I64(
        ((lbo >> 4) & 0x3FFF) << 16
        | ((sbo >> 4) & 0x3FFF) << 32
        | (1 << 46)
        | (layout << 61)
    )

stg_v4(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None)

st.global.v4.b32 (explicit .global: a generic store to an NVLS multicast address faults).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def stg_v4(addr, w0, w1, w2, w3, pred, *, loc=None, ip=None):
    """st.global.v4.b32 (explicit .global: a generic store to an NVLS multicast address faults)."""
    _asm(
        None,
        [I64(addr), I32(w0), I32(w1), I32(w2), I32(w3), I32(pred)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $5, 0;\n\t@p st.global.v4.b32 [$0], {$1, $2, $3, $4};\n\t}",
        "l,r,r,r,r,r",
        loc=loc,
        ip=ip,
    )

tc_ld16(taddr, *, loc=None, ip=None)

Two tcgen05.ld 32x32b.x8 (columns c, c + 8) and one wait.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def tc_ld16(taddr, *, loc=None, ip=None):
    """Two tcgen05.ld 32x32b.x8 (columns c, c + 8) and one wait."""
    out = _asm(
        _struct(16),
        [I32(taddr)],
        "{\n\t.reg .b32 t;\n\tadd.u32 t, $16, 8;\n\t"
        "tcgen05.ld.sync.aligned.32x32b.x8.b32 {$0, $1, $2, $3, $4, $5, $6, $7}, [$16];\n\t"
        "tcgen05.ld.sync.aligned.32x32b.x8.b32 {$8, $9, $10, $11, $12, $13, $14, $15}, [t];\n\t"
        "tcgen05.wait::ld.sync.aligned;\n\t}",
        ",".join(["=r"] * 16) + ",r",
        loc=loc,
        ip=ip,
    )
    return [v.bitcast(F32) for v in _outs(out, 16, I32, loc, ip)]

tc_ld8(taddr, *, loc=None, ip=None)

tcgen05.ld 32x32b.x8 and its wait in one statement (the values exist only after the wait).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def tc_ld8(taddr, *, loc=None, ip=None):
    """tcgen05.ld 32x32b.x8 and its wait in one statement (the values exist only after the wait)."""
    out = _asm(
        _struct(8, True),
        [I32(taddr)],
        "tcgen05.ld.sync.aligned.32x32b.x8.b32 {$0, $1, $2, $3, $4, $5, $6, $7}, [$8];\n\t"
        "tcgen05.wait::ld.sync.aligned;",
        "=f,=f,=f,=f,=f,=f,=f,=f,r",
        loc=loc,
        ip=ip,
    )
    return _outs(out, 8, F32, loc, ip)

tc_mma_ss(d, da, db, idesc, acc, sfa, sfb, *, loc=None, ip=None)

Block-scaled MMA, A and B from shared memory descriptors.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def tc_mma_ss(d, da, db, idesc, acc, sfa, sfb, *, loc=None, ip=None):
    """Block-scaled MMA, A and B from shared memory descriptors."""
    _asm(
        None,
        [I32(d), I64(da), I64(db), I32(idesc), I32(acc), I32(sfa), I32(sfb)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $4, 0;\n\t"
        "tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale [$0], $1, $2, $3, [$5], [$6], p;\n\t}",
        "r,l,l,r,r,r,r",
        loc=loc,
        ip=ip,
    )

tc_mma_ts(d, ta, db, idesc, acc, sfa, sfb, *, loc=None, ip=None)

Block-scaled MMA, A from TMEM, B from a shared memory descriptor.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def tc_mma_ts(d, ta, db, idesc, acc, sfa, sfb, *, loc=None, ip=None):
    """Block-scaled MMA, A from TMEM, B from a shared memory descriptor."""
    _asm(
        None,
        [I32(d), I32(ta), I64(db), I32(idesc), I32(acc), I32(sfa), I32(sfb)],
        "{\n\t.reg .pred p;\n\tsetp.ne.b32 p, $4, 0;\n\t"
        "tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale [$0], [$1], $2, $3, [$5], [$6], p;\n\t}",
        "r,r,l,r,r,r,r",
        loc=loc,
        ip=ip,
    )

ue8m0(x, *, loc=None, ip=None)

UE8M0 of x rounded up, satfinite (the low byte of cvt.rp.satfinite.ue8m0x2.f32 with a 0 partner).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ue8m0(x, *, loc=None, ip=None):
    """UE8M0 of x rounded up, satfinite (the low byte of cvt.rp.satfinite.ue8m0x2.f32 with a 0 partner)."""
    return I32(
        _asm(
            _i32t(),
            [F32(x)],
            "{\n\t.reg .b16 h;\n\t.reg .b32 w;\n\tcvt.rp.satfinite.ue8m0x2.f32 h, 0f00000000, $1;\n\t"
            "cvt.u32.u16 w, h;\n\tand.b32 $0, w, 255;\n\t}",
            "=r,f",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

ue8m0_value(code, *, loc=None, ip=None)

The float value of a UE8M0 code (through bf16, as __nv_fp8_e8m0 -> float).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def ue8m0_value(code, *, loc=None, ip=None):
    """The float value of a UE8M0 code (through bf16, as __nv_fp8_e8m0 -> float)."""
    return F32(
        _asm(
            _f32t(),
            [I32(code)],
            "{\n\t.reg .b16 h;\n\t.reg .b32 r;\n\tcvt.u16.u32 h, $1;\n\tand.b16 h, h, 255;\n\t"
            "cvt.rn.bf16x2.ue8m0x2 r, h;\n\tshl.b32 $0, r, 16;\n\t}",
            "=f,r",
            side=False,
            loc=loc,
            ip=ip,
        )
    )

zero_byte_mask(t, *, loc=None, ip=None)

0x80 in every byte of t that is zero, else 0 (exact per byte).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/front_kernel.py
@dsl_user_op
def zero_byte_mask(t, *, loc=None, ip=None):
    """0x80 in every byte of t that is zero, else 0 (exact per byte)."""
    return I32(
        _asm(
            _i32t(),
            [I32(t)],
            "{\n\t.reg .b32 z;\n\tand.b32 z, $1, 0x7F7F7F7F;\n\tadd.u32 z, z, 0x7F7F7F7F;\n\t"
            "or.b32 z, z, $1;\n\tnot.b32 z, z;\n\tand.b32 $0, z, 0x80808080;\n\t}",
            "=r,r",
            side=False,
            loc=loc,
            ip=ip,
        )
    )