Qwen 3.8 27B — Serving Configuration Report (Strix Halo, gfx1151)

Verdict. The fastest, most reliable serving configuration for Qwen 3.8 27B on this machine is:

llama.cpp llama-server + Qwen3.8-27B-Q4_K_M.gguf + native MTP speculative decoding, --spec-draft-n-max 4

Measured on the mandatory captured real-world replay (PLAN §2.2): 24.0 t/s mean / 20.1 t/s median generation (re-run: 24.67 t/s), accept_len 3.36, 12/12 turns, 0 errors, ~14 s cold start. No other engine or method comes close: the best vLLM cell is ~10× slower (2.4 t/s), and the best non-MTP llama.cpp method (DSpark) is ~2× slower (12.2 t/s).

This is a clear and reproducible winner, confirmed by running the winner cell twice (L1.4 = 24.0 t/s, L1.4-rerun = 24.67 t/s) and by the fact that the next-best complete cell (L6.5, --prio 3) is within noise of it (23.87 t/s) while every distinct configuration change is measurably worse.

1. Exact winning configuration (reproducible)

# lab/serve_best.sh
/home/ericflo/Development/llama.cpp/build/bin/llama-server \
  -m /home/ericflo/.lmstudio/models/lmstudio-community/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj /home/ericflo/.lmstudio/models/lmstudio-community/Qwen3.8-27B-GGUF/mmproj-Qwen3.8-27B-BF16.gguf \
  --alias qwen/qwen3.8-27b \
  --host 127.0.0.1 --port 1234 \
  --jinja -fa on -ngl 999 -np 1 -c 262144 \
  --reasoning-preserve \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --verbose

Replay benchmark (the ONLY authorized workload, PLAN §2.2):

./run_benchmark.sh local captures-desktop \
  --chunks 3 --chunk 4 \
  --conversation 199ae534158f31e2 \
  --request-timeout 600

Capture: captures-desktop/capture-2026-08-15.jsonl, sha256 cba6a988487e73f4ccac374126e7473f0c4f3aafdae512d224d63f12baa43216 (derived per-request manifest sha 4d3900af5383df75b45c6beb8a31c51f71656c47c25d36f4eed66fe231ab0cda, identical across every cell — the workload is frozen and consistent).

2. Captured-replay leaderboard (all 24 completed cells)

Source: lab/results/LEADERBOARD.md (auto-generated from per-cell JSON). Workload = the §2.2 replay, 12 turns, single conversation, thinking ON. Ranked by generation t/s (mean).

vLLM (excluded from ranking — incomplete lane):

3. What the data says

3.1 MTP draft-depth knee (Q4_K_M) — the core finding

Acceptance length keeps rising past n=4 (3.36 → 3.95) but throughput falls (24.0 → 20.6). This is the classic speculative knee: the draft cost and verification overhead of extra tokens outweighs the marginal extra acceptance. n=4 is the optimum.

3.2 Quantization ladder (MTP n=4)

Lower precision = faster (bandwidth-bound), and MTP's relative speedup is actually larger on the heavier Q8_0 target (2.38× vs 2.07×) because the no-spec baseline is slower. Q4_K_M dominates on throughput and is the right default; Q8_0 is the precision-ceiling option if quality is prioritized over speed (user's Q8-class ceiling).

3.3 Speculative-method comparison (Q4_K_M)

MTP is unambiguous. It is the target's own next-token head (no cross-model feature transfer), so it is the only method with high, stable acceptance. DSpark's Markov head survives cross-gen (accept_len ~1.5–2.4) but is still ~2× slower than MTP. DFlash is a diffusion drafter whose fc projection is trained on the Qwen3.6 target's hidden-state distribution — against the 3.8 target the features are out-of-distribution → 0.12% per-token acceptance (L3.1), effectively zero. DFlash needs a same-generation head (none exists for 3.8 yet).

3.4 Runtime-knob sweep on the winner (Q4_K_M, MTP n=4)

The default configuration is already near-optimal. The single important flag is -fa on (disabling it costs 40%). KV quantization, batch shape, priority, context length, and checkpoint count all trend downward or are neutral — no runtime knob improves on the default.

4. Cross-engine due diligence

4.1 vLLM (kyuz0/vllm-therock-gfx1151, v0.27.1+strix, ROCm 7.14)

• V1.x (GGUF lane) — NOT EXECUTABLE. vLLM's GGUF load path OOM-killed the host (ExitCode=137, OOMKilled=true). The host has only 30 GiB system RAM; vLLM's GGUF loading (weights + page cache) peaks above that, while the identical 28.75 GB FP8 safetensors path streams shards to the GPU with ~6 GB host RAM. This permanently blocks the GGUF lane on this machine.
• V2.1 (FP8 + MTP n=4) — 2.4 t/s mean / 5.7 t/s peak, 9/12 turns. This is the best vLLM config. It is ~10× slower than llama.cpp. The gap is structural: vLLM's FP8 W8A8 decode on gfx1151 runs at ~2–5 t/s, and the fastest single vLLM turn (5.7 t/s) is still 2.7× slower than the slowest llama.cpp turn (15.6 t/s). No speculative method (MTP/DFlash/DSpark) can close a 4× decode gap.
• V2.2 (AWQ-INT4) — not executed. No AWQ-INT4 build/image for gfx1151 (kyuz0 image ships FP8+GGUF only).
• V3.x (DFlash) / V4.x (DSpark) — deprioritized (documented). The vLLM decode ceiling (~5.7 t/s) is already 4× below llama.cpp's minimum; speculative acceptance cannot lift it to 24 t/s. DFlash cross-gen is independently dead (L3.1). Recorded with evidence, not hidden.

4.2 SGLang (Phase 3)

• NOT EXECUTABLE. No public prebuilt SGLang image for gfx1151 exists (ghcr.io/JeremiahM37/strix-halo-sglang is not pullable). The only path is a local build from /home/ericflo/Development/strix-halo-sglang, deferred per the one-GPU-workload-at-a-time rule and because the SGLang README itself notes single-stream decode is weaker than llama.cpp on this stack. RadixArk's DSpark is verified on SGLang in principle, but even its best SGLang acceptance (3.4–4.6) cannot overcome a slower base decode lane. Recorded, not measured.

5. Reproducibility

The winner cell was run twice on the same frozen capture (identical manifest sha 4d3900af…):

Δ = +2.8% (run-to-run variance). The winner is stable and reproducible.

6. Recorded non-executable cells (full audit)

These PLAN cells have no replay result by necessity, each with a documented reason (see results/<engine>/<cell>.json):

7. Caveats & limitations

1. Power mode. The system is in a reduced (stable) power mode since the 2026-08-15 crash (~11% slower than pre-crash). This affects all cells equally (relative ranking is unaffected), but absolute t/s numbers are ~11% below a full-power machine.
2. TTFT dominates wall time. ~92% of replay wall time is TTFT/prefill (mean TTFT ~99–102 s, driven by long agentic prompts up to 88.9 k tok). The generation t/s (the decode metric, where the winner shines) is what this report ranks. If your workload is prefill-dominated, the relative advantage of speculative decoding shrinks (prefill is not spec-decoded).
3. Single-stream scope. PLAN §2.1 excludes concurrency/soak. These numbers are for -np 1 single-conversation serving (the agentic-coding daily-driver use case), not for multi-user throughput.
4. Think-on only. All cells run thinking ON (forced by the replay). Think-off acceptance (often higher) is unmeasurable on this frozen capture.
5. Drafters are third-party. The DSpark/DFlash heads are community models (RadixArk, satgeze, z-lab); MTP (the winner) is the target's own head and needs no external drafter.

8. Deliverables

Generated 2026-08-16. All measurements from the mandatory captured replay (PLAN §2.2), single stream, thinking ON, Q4_K_M→Q8_0 quantization ladder, on Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, gfx1151, 96 GiB).