A while back we got NVIDIA’s and unsloth’s NVFP4 checkpoints of Qwen3.6-27B running on llama.cpp, repackaging the 4-bit safetensors into GGUF so they use llama.cpp’s native sm_120 FP4 tensor-core path. That post was a throughput story: we took an RTX 5090 Laptop from 24 tok/s on a popular Q4_K_M build to 45 tok/s. What we explicitly did not do was measure output quality. “Numerically faithful by construction” is a nice argument, but it isn’t an eval.

This post closes that gap. We shrank the model further, ran it against three benchmarks from the TNG benchmark suite, and, because the numbers held up, we’re releasing it.

TL;DR: The released build is 19.41 GB, runs at 46.48 tok/s on a 24 GB laptop (~1.5× the reference Q4_K_M), leaves ~4.5 GB of headroom for KV cache, draft cache, and compute buffers, and shows no meaningful quality drop versus the full-precision BF16 model on terminal-bench 2.0, MMLU-Pro, and IFBench. It is the smallest and the fastest of everything we built. Grab it here: huggingface.co/tngtech/Qwen3.6-27B-NVFP4-GGUF.

Quick recap: repackage, don’t requantize

The one idea worth repeating from the first post: both vendor checkpoints are mixed precision: 4-bit NVFP4 on the bulk of the MLP, 8-bit FP8 on the attention projections, the Gated-DeltaNet projections, and the LM head. The conversion respects that split:

  • NVFP4 weights are repacked bit-exact: the E2M1 nibbles and their FP8 group scales are copied verbatim, only re-grouped into GGUF’s 64-element super-blocks. No re-rounding.
  • FP8 weights are dequantized to Q8_0 (via --fp8-as-q8), since GGUF has no FP8 weight type. Lossless-or-better relative to the FP8 source.

The result is numerically equivalent to the vendor checkpoint and sidesteps the double-quantization penalty of the naive “dequantize to F16, then llama-quantize to Q4_K_M” route. The NVIDIA (ModelOpt) checkpoint converts out of the box; the unsloth (compressed-tensors) one needed a three-line patch to the convert script, both covered in the earlier post.

Squeezing out the last two gigabytes

On a 24 GB card every gigabyte is KV-cache headroom, so after the initial repack we went after the two remaining full-precision tensors: the token embedding and the MTP (multi-token-prediction) head, both kept at BF16 by default because they’re unquantized source tensors.

We added two opt-in flags:

  • --embd-as-q8 forces token_embd to Q8_0, saving ~1.16 GB.
  • --mtp-as-q8 forces the MTP layer’s (blk.64) 2D weight tensors to Q8_0, saving ~0.39 GB.

Together that’s ~1.55 GB off the file, taking the NVIDIA build from 20.96 GB to 19.41 GB. This shrunk build (NVFP4 MLP repacked bit-exact, FP8 attention/GDN dequantized to Q8_0, and now both the embedding and MTP head at Q8_0) is what we released.

The embedding shrink is not new: the first post introduced --embd-as-q8 and showed the Q8_0 embedding is effectively lossless (rel-RMSE 0.54%, per-row cosine 0.999986 over 30K sampled tokens), since the input embedding is immediately RMS-normalized and is a separate tensor from the LM head, so it never directly distorts logits.

What’s new is the MTP head, and it needed more care. Shrinking it to Q8_0 is a pure speed lever, not a quality one: the MTP head only drafts tokens that the target model then verifies, so it cannot change final output, but a lower-precision draft head could lower the speculation acceptance rate and cost throughput. We expected exactly that regression. It didn’t happen (see below), which is why the released build carries both shrinks.

The measurements

All numbers below are on the target machine: an RTX 5090 Laptop (24 GB), the laptop part with ~half the desktop 5090’s memory bandwidth.

Size

Model Size on disk
unsloth bf16 (reference, full precision) 54.7 GB
NVIDIA straight repackage 20.96 GB
  + embedding → Q8_0 19.80 GB
  + embedding & MTP → Q8_0 (released) 19.41 GB
unsloth straight repackage 22.62 GB
  + embedding → Q8_0 21.46 GB
unsloth Q4_K_M (reference) 17.1 GB

Sizes above are the remeasured, current on-disk figures; the first post quoted 21.5 GB (NVIDIA) and 23.2 GB (unsloth) for the same straight repackages from its earlier build round.

We can’t match the 17.1 GB Q4_K_M, and that’s the point: the extra ~2.3 GB is precision retention where Q4_K_M compresses: attention and the GDN/SSM projections kept at Q8_0, not the FP4 weights (the NVFP4 MLP is essentially the same size as Q4_K_M’s MLP). Further shrinking would require actually re-tuning the affected layers, which is future work. At 19.41 GB the build leaves ~4.5 GB free for KV cache, draft cache, and compute buffers on a 24 GB card.

Performance

Decode throughput on the 5090 Laptop, sweeping MTP speculative-decoding depth (--spec-draft-n-max):

Build MTP 1 MTP 2 MTP 3 MTP 4
unsloth Q4_K_M (reference) 31 (peak) - - -
NVIDIA straight repackage - - 46.46 -
  + embedding Q8_0 - - 45.77 -
  + embedding & MTP Q8_0 (released) 39.18 46.21 46.48 45.85

Three things worth calling out:

  • ~1.5× the reference quant. 46.48 tok/s versus the unsloth Q4_K_M’s own peak of 31 tok/s (which it hits at MTP depth 1). MTP speculative decoding is the dominant lever, and the repackaged builds tolerate deeper speculation. Depth 3 is the sweet spot here (where Q4_K_M peaked at depth 1), consistent with better draft/target agreement through the properly handled MTP layer.
  • The smallest build is also the fastest. The fully shrunk build (46.48) edges out the BF16-embedding base (46.46), within run-to-run margin, but the point is that shrinking cost nothing.
  • The MTP-degradation worry did not materialize. We expected a Q8_0 MTP head to drop the speculation acceptance rate. It didn’t: the 0.4 GB came free.

Quality

This is the part the first post was missing. We evaluated against three benchmarks from the TNG benchmark suite: terminal-bench 2.0, MMLU-Pro, and IFBench. terminal-bench and MMLU-Pro used the TNG benchmarking team’s distilled subsets (v0.1, held to be of approximately full-benchmark hardness); IFBench was run in full. The reference is the full-precision unsloth bf16 (54.7 GB).

To isolate weight-quantization effects from KV-cache effects, benchmarks were run in a best-case quality server config: --reasoning-preserve and f16 KV cache (not the q4_0/q4_0 deployment config).

Build terminal-bench 2.0 MMLU-Pro IFBench
unsloth bf16 (reference) 46.7% 85.15% 72.00%
NVIDIA straight repackage 40.0% 84.16% 66.67%
  + embedding Q8_0 43.4% 90.10% 67.33%
  + embedding & MTP Q8_0 (released) 46.7% 86.14% 66.67%

The headline: no meaningful loss versus BF16. The released build matches BF16 on terminal-bench (46.7%) and sits within noise on MMLU-Pro (86.14% vs 85.15%). The only visible gap is IFBench (66.67% vs 72.00%).

And crucially, shrinking did not hurt quality: the Q8_0-embedding/-MTP builds score at or above the straight repackage on all three, exactly as the lossless-embedding measurement and the draft-only nature of the MTP head predict.

A caveat on noise. The distilled subsets and small absolute counts make single-point differences noisy: the straight repackage scoring below the shrunk builds on terminal-bench and MMLU-Pro is within that noise, not a real inversion. And these are the f16-KV best case; the q4_0/q4_0 deployment config is expected to score somewhat lower. Quantifying the KV-cache quality cost is future work.

Running it

The recommended deployment recipe at 160K context on a 24 GB laptop, using the official CUDA server container:

docker run --rm --gpus '"device=0"' \
    -v $HOME/models:/models -v ./:/local -p 8080:8080 \
    ghcr.io/ggml-org/llama.cpp:server-cuda \
    -m /local/Qwen3.6-27B-NVFP4.gguf \
    -ngl 99 -c 160000 -fa on -np 1 \
    --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 \
    --temp 1 --top-p 0.95 --top-k 20 --presence-penalty 1.5 --min-p 0.0 \
    --port 8080 --host 0.0.0.0

The first post walks through the flags and the KV-cache tuning in detail: why -fa on is required for a quantized cache, why q4_0/q4_0 is the right call at the full 160K (only 16 of the 64 layers carry a growing KV cache, thanks to the Gated-DeltaNet hybrid design), and the K-vs-V precision lever for shorter prompts. Nothing there changed with the shrunk build, so we won’t repeat it here.

Where to get it

The build is live at huggingface.co/tngtech/Qwen3.6-27B-NVFP4-GGUF, with a model card documenting the recipe, the benchmark numbers above, and the runtime config.

A note on naming: although this is the embedding + MTP → Q8_0 build described above, we published it under the plain name Qwen3.6-27B-NVFP4.gguf, the source model plus GGUF, without the internal -embq8-mtpq8 postfixes. The descriptive labels in the tables above are just how we tracked the variants during the investigation; the released file is simply the best of them. There are already a few similar community quants out there, and we’re not looking to compete, just to publish a trusted, measured build for local inference.

Takeaways

  • NVFP4 on consumer Blackwell is no longer blocked by tooling. The FP4 weight kernels are in llama.cpp today; the checkpoints exist; repackaging to GGUF is either trivial (NVIDIA) or a three-line patch away (unsloth).
  • The measured build holds up. ~1.5× the throughput of the popular Q4_K_M build, ~4.5 GB of KV headroom, and no meaningful quality drop versus full-precision BF16 on three benchmarks.
  • Smaller was strictly better here. Shrinking the embedding and MTP head to Q8_0 cost no measurable throughput and no measurable quality, so the smallest file is also the fastest.