Qwen3.6-27B is arguably the best model you can fit on a 24 GB GPU today. But “fit” and “fast” are different problems and the most interesting quantization format for Blackwell hardware, NVFP4, doesn’t run on the tools most of us actually use locally. This is the story of how we got it running at nearly 2x our starting throughput, including a detour into patching llama.cpp’s conversion script to handle a checkpoint it flatly refused to load.

TL;DR: We repackaged the NVIDIA and unsloth NVFP4 checkpoints of Qwen3.6-27B into GGUF, so they run on llama.cpp’s native sm_120 FP4 tensor-core path. On an RTX 5090 Laptop (24 GB), combined with the model’s MTP head for speculative decoding, this took us from 24 tok/s (a popular Q4_K_M build) to 45 tok/s at 160K context. The NVIDIA checkpoint converts out of the box; the unsloth one needed a small patch to the convert script, which we describe in detail.

The wen3.6 27B NVFP4 gguf repack performs around double the baseline of the unsloth Q4_K_M quant
The wen3.6 27B NVFP4 gguf repack performs around double the baseline of the unsloth Q4_K_M quant

The model: a hybrid worth the trouble

Before the quantization story, it’s worth understanding why Qwen3.6-27B is structured the way it is, because nearly every downstream decision follows from its architecture.

Reading the GGUF metadata, the text model is a dense hybrid:

  • 65 blocks: 64 transformer layers plus one MTP (multi-token prediction) head at blk.64.
  • full_attention_interval = 4: only 16 of the 64 layers use full attention (every 4th one). The other 48 layers are Gated-DeltaNet (GDN) linear-attention layers, which carry a fixed-size recurrent state instead of a growing KV cache.
  • GQA with 24 query heads, 4 KV heads, and an unusually large head_dim of 256.
  • Native context of 262,144 tokens, with a RoPE base of 1e7.

That hybrid design is the reason a 27B model with 160K context fits on 24 GB at all: 75% of the layers never grow their memory footprint with context length. Only 16 layers accumulate a KV cache. Hold that thought, it comes back when we tune the runtime.

The problem: NVFP4 is stranded on consumer hardware

Both NVIDIA (via TensorRT Model Optimizer) and unsloth (via llm-compressor) publish NVFP4 checkpoints of this model. NVFP4 is a 4-bit floating-point format (E2M1 values in groups of 16, with an FP8 per-group micro-scale and a per-tensor global scale) designed for Blackwell’s FP4 tensor cores, exactly what an RTX 5090 has.

The catch: those checkpoints ship as safetensors in compressed-tensors / modelopt layouts. You can’t hand them to llama.cpp, and while vLLM supports NVFP4 on Blackwell, its memory overhead needs more headroom than a 24 GB laptop leaves you. So the format that should be ideal for this hardware is the one you can’t run.

The fix is repackaging: convert the NVFP4 safetensors into a GGUF that uses llama.cpp’s native NVFP4 tensor type (GGML type 40), which already has sm_120 weight-GEMM kernels upstream.

Repackage, don’t requantize

The single most important idea here is the distinction between repackaging and requantizing.

Both checkpoints are mixed precision: 4-bit NVFP4 on the bulk of the MLP, and 8-bit FP8 on the attention projections, the GDN projections, and the LM head. The conversion treats the two differently:

  • NVFP4 weights are repacked bit-exact. The converter copies the E2M1 nibbles verbatim and preserves the FP8-E4M3 group scales, merely re-grouping the group-16 layout into GGUF’s 64-element super-blocks. The per-tensor global scale is stored as a separate GGUF tensor and applied at inference via ggml_mul. No re-rounding, no quality loss.
  • FP8 weights are dequantized to Q8_0. GGUF has no FP8 weight type, so with the --fp8-as-q8 flag these become Q8_0 (8.5 bpw) rather than the default BF16 (which would be larger). This is lossless-or-better relative to the FP8 source.

The result is numerically equivalent to the source, and crucially it avoids the double-quantization penalty you’d get from the naive route (dequantize everything to F16, then run llama-quantize to Q4_K_M). You keep the vendor’s carefully-calibrated 4-bit weights exactly as they are.

The easy path: NVIDIA’s ModelOpt export

The NVIDIA checkpoint uses quant_method: modelopt, and llama.cpp’s convert_hf_to_gguf.py already has a mixed-precision-aware ModelOpt branch. Conversion is a one-liner:

export PYTHONPATH=$PWD/llama.cpp/gguf-py
python llama.cpp/convert_hf_to_gguf.py nvidia/Qwen3.6-27B-NVFP4 \
    --outfile Qwen3.6-27B-NVFP4.gguf --fp8-as-q8

Out comes a 21.5 GB GGUF with 1252 tensors: NVFP4 MLP weights repacked bit-exact, FP8 attention/GDN/LM-head dequantized to Q8_0, and the token embedding kept at BF16. The vision tower is dropped (it’s text-only; the vision encoder would need a separate --mmproj file).

The hard path: patching the convert script for unsloth

The unsloth checkpoint uses quant_method: compressed-tensors with format: mixed-precision, and it immediately fails:

NotImplementedError: Can't handle multiple config groups for compressed-tensors yet

Digging in, the checkpoint declares two heterogeneous config groups:

  • group_0 - FP8 (float-quantized, per-channel): attention q/k/v/o_proj, GDN linear_attn.*, lm_head, and - interestingly - the last 8 layers’ MLP (layers 56-63). Tensors are stored as .weight (F8_E4M3) plus a 2D per-channel .weight_scale shaped [out, 1].
  • group_1 - NVFP4 (nvfp4-pack-quantized): the MLP gate/up/down_proj for layers 0-55. Tensors are .weight_packed (uint8) plus a 2D .weight_scale (F8_E4M3, shape [out, in/16]) plus a .weight_global_scale.

The converter’s NVFP4 path gated everything on a predicate that demanded every group be NVFP4:

nvfp4_compressed_tensors = (
    quant_format == "nvfp4-pack-quantized"
    or quant_format == "mixed-precision"
    and bool(groups)
    and all(g.get("format") == "nvfp4-pack-quantized" for g in groups.values() ...)
)

Because group_0 is FP8, this evaluated to False, the NVFP4 handling never engaged, and the multi-group guard raised. The ModelOpt path dodged all of this precisely because it takes a different, mixed-precision-aware branch. The logic we needed already existed in the codebase - it just wasn’t reachable for compressed-tensors. Three small, surgical changes fixed that.

1. Broaden the predicate (all -> any). At both places the predicate is computed, we changed it to recognize a mixed-precision config as NVFP4 when any group is NVFP4:

and any(g.get("format") == "nvfp4-pack-quantized" for g in groups.values() ...)

This flips the internal _is_nvfp4 flag on, enabling the NVFP4 repack and the compressed-tensors -> ModelOpt scale-renaming, and makes the multi-group guard stop raising.

2. Fix the NVFP4-vs-FP8 discriminator. The routine that repacks NVFP4 tensors iterates over everything with a .weight_scale and skipped non-NVFP4 tensors using scale.ndim < 2. That heuristic works for ModelOpt (whose FP8 scales are 1D), but unsloth’s FP8 group uses 2D per-channel scales shaped [out, 1], which would have been wrongly packed as NVFP4. The robust discriminator is the trailing dimension:

# FP8 uses per-channel scales: 1D (ModelOpt) or 2D with trailing dim 1 (compressed-tensors).
if scale.ndim < 2 or scale.shape[-1] == 1:
    continue

NVFP4 block scales always have a trailing dimension of in_features / 16 > 1, so this cleanly separates the two without affecting the ModelOpt path.

3. Dequantize the leftover FP8 group. The NVFP4 repack runs first and removes the NVFP4 tensors, so by the time the dequantization pass runs, the only .weight_scale tensors left are the FP8 ones. We replaced a pass in the mixed-precision branch with the same per-channel FP8 dequant loop the float-quantized path already used, marking those tensors for Q8_0 under --fp8-as-q8.

Validating the patch

Since the ModelOpt GGUF is a known-good reference, we validated structurally: dump both GGUFs and diff tensor names, types, and shapes. The result was exactly what the two vendors’ recipes predict:

  • Zero shape mismatches, no missing or extra tensors.
  • 25 tensors differ in dtype, all NVFP4 -> Q8_0: the 24 MLP tensors of layers 56-63, plus output (the LM head). These are precisely the tensors unsloth assigns to its FP8 group but NVIDIA keeps as NVFP4.
  • The 50 NVFP4-only sidecar scale tensors for those layers are correctly absent from the unsloth build.

Every downgraded tensor maps to an unsloth config target - nothing was misclassified. The unsloth GGUF is 1.7 GB larger than NVIDIA’s for exactly this reason: it keeps those 8 tail MLP layers and the LM head at Q8_0 (8.5 bpw) instead of NVFP4 (4.5 bpw), likely a deliberate precision choice for the sensitive tail of the network.

Anatomy of the resulting GGUFs

Here’s how the three builds break down by function:

Component Q4_K_M NVIDIA NVFP4 unsloth NVFP4
FFN / MLP 10.5 GB (Q4_K/Q6_K) 10.2 GB (NVFP4) 11.2 GB (NVFP4 + 8 layers Q8_0)
Attention 3.6 GB (Q4-Q6_K) 6.3 GB (Q8_0) 6.3 GB (Q8_0)
SSM / GDN 1.1 GB 1.7 GB (Q8_0) 1.7 GB (Q8_0)
token_embd 0.7 GB (Q4_K) 2.5 GB (BF16) 2.5 GB (BF16)
output / LM head 1.0 GB (Q6_K) 0.7 GB (NVFP4) 1.4 GB (Q8_0)
Total 17.1 GB 21.5 GB 23.2 GB

The NVFP4 builds are bigger than Q4_K_M, but not because FP4 is bulky - the NVFP4 MLP (10.2 GB) is essentially the same size as Q4_K_M’s MLP. The extra size is precision retention where Q4_K_M compresses: attention kept at Q8_0 (+2.7 GB), the token embedding kept at BF16 for the 248K-token vocab (+1.8 GB), and the GDN projections at Q8_0. In other words, a “4-bit model” that weighs 21-23 GB is really mixed precision: only the MLP (44% of the file) is actually 4-bit; the numerically sensitive parts spend more bits.

Shrinking the token embedding

On a 24 GB card, every gigabyte counts, and the BF16 token embedding (2.5 GB) is the obvious target. We added an opt-in --embd-as-q8 flag that forces token_embd to Q8_0, saving ~1.2 GB.

We measured the cost directly by quantizing the actual embedding tensor and comparing to the BF16 original over 30K sampled tokens:

Embedding type size rel-RMSE per-row cosine
BF16 (default) 2.54 GB - -
Q8_0 1.35 GB 0.54% 0.999986
Q4_K 0.72 GB ~7% ~0.997

Q8_0 is effectively lossless. That’s better than the raw number even suggests, for two reasons: the input embedding is immediately RMS-normalized, so magnitude error largely washes out and only direction (cosine) propagates; and the embedding is a separate tensor from the LM head here, so quantizing it never directly distorts logits.

We also considered NVFP4 for the embedding and decided against it: there’s no BF16->NVFP4 quantizer in the toolchain (it only repacks existing NVFP4), the per-tensor global scale is applied in the matmul path and not in the embedding gather (get_rows), and CUDA get_rows has no NVFP4 case anyway. It would land in the same quality tier as Q4_K for more effort. Q8_0 is the sweet spot.

What actually made it fast

Now the benchmarks, on the RTX 5090 Laptop (24 GB), pure inference:

Build MTP draft tok/s
unsloth Q4_K_M none 24
unsloth Q4_K_M n-max 1 31
NVIDIA NVFP4 n-max 3 45
unsloth NVFP4 n-max 3 41

Two findings stand out.

MTP speculative decoding is the dominant lever. Decode is weight-bandwidth-bound, and llama.cpp already had the native FP4 weight path, so the quant swap alone (31 -> 45) is worth ~1.45x. Turning on the model’s MTP head for speculative decoding is what closes the rest of the gap to nearly 2x. Notably, the repackaged builds happily run deeper speculation (n-max 3 was optimal) where the Q4_K_M build peaked at n-max 1 - consistent with better draft/target agreement through the properly handled MTP layer.

The NVIDIA-vs-unsloth gap (45 vs 41) is pure bandwidth. Decode streams every matmul weight per token (everything except the embedding, which is a gather). NVIDIA streams ~18.9 GB/token; unsloth ~20.6 GB/token. The ratio, 1.090, matches the throughput ratio, 1.098, almost exactly. That extra ~1.7 GB is precisely unsloth’s higher-precision tail - layers 56-63 MLP and the LM head at Q8_0 instead of NVFP4. So NVIDIA is faster because it runs FP4 on more of the network: a bandwidth win, at a possible small accuracy cost on the tail. You can’t close the gap on the unsloth side without lossy FP8->NVFP4 requantization, since those layers ship as FP8 and never as FP4.

Running it

The benchmark used the official CUDA server container:

docker run --rm --gpus '"device=0"' \
    -v $HOME/models:/models -v ./:/local -p 8080:8080 \
    ghcr.io/ggml-org/llama.cpp:server-cuda \
    -m /local/Qwen3.6-27B-NVFP4.gguf \
    -ngl 99 -c 160000 -fa on -np 1 \
    --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 \
    --temp 1 --top-p 0.95 --top-k 20 --presence-penalty 1.5 --min-p 0.0 \
    --port 8080 --host 0.0.0.0

A few notes for anyone reproducing it. -fa on (flash attention) is required for a quantized KV cache and works fine with this model’s head_dim of 256. 160K is below the native 262K context, so no RoPE scaling is needed. And the sampling temp/top-p/top-k match the model’s own generation_config.json.

KV cache tuning is where the headroom is

Because only 16 layers carry a KV cache, its footprint at 160K is manageable:

K / V type main KV + MTP draft
f16 / f16 10.5 GB +0.65
q8_0 / q8_0 5.6 GB +0.35
q4_0 / q4_0 2.95 GB +0.18
K=q8_0 / V=q4_0 4.26 GB +0.23

With ~19-21.5 GB of resident weights plus ~3 GB of KV plus the draft cache and compute buffers, the run sits right at the 24 GB ceiling - which is why q4_0/q4_0 is the necessary choice here, and why the smaller NVIDIA quant has more breathing room than unsloth.

One quality lever worth knowing: K-cache quantization hurts more than V, because K enters the QK^T dot product before the softmax, so its error shifts attention weights directly, while V error only blurs the weighted average. If your prompts rarely exceed ~100K tokens, note that K=q8_0 / V=q4_0 at ~110K context uses the same VRAM as q4_0/q4_0 at 160K - a strictly better precision trade for the same memory. For the full 160K, q4_0/q4_0 is the right call.

Caveats and takeaways

These results are runtime-validated for throughput - both repackaged builds run and generate under MTP speculative decoding. We have not formally measured generation quality (perplexity, task accuracy); the repackaging is numerically faithful to the source by construction, but a proper quality eval is future work. The conversion patches live in a private fork of the convert script.

The broader takeaway: NVFP4 on consumer Blackwell is no longer blocked by tooling. The weight kernels are in llama.cpp today, the checkpoints exist, and repackaging them to GGUF is either trivial (NVIDIA) or a three-line patch away (unsloth). For a single-user, long-context workload on a 24 GB laptop, the combination of native FP4 weights and the model’s own MTP head turns a “technically fits” model into a genuinely fast one.