2026-06-28 — Dan Billings

The Goldilocks Zone on the 5090: 192k Context on Gemma-4-31B via Q8/Q5_1 KV Cache Quantization

Dan Billings — 2026-06-28

[!NOTE] This post details the search for the optimal context window ceiling on the RTX 5090 (32 GB GDDR7). By leveraging asymmetrical q8_0 Key / q5_1 Value KV cache quantization, compiling customized Flash Attention kernels, and balancing production memory footprints, we've found the ultimate "Goldilocks" zone: running Google's massive Gemma-4-31B at 192k context length with 72 tok/s decode and 1,168 tok/s prefill fully resident on a single consumer GPU.

Our last post detailed how we configured Gemma-4-31B (QAT UD-Q4_K_XL weights) on the RTX 5090 using q8_0/q8_0 KV cache to hit 100 tok/s at 96k context. But when you're running deep agentic workflows (like the Honcho dream loop), a 96k window eventually feels like a cage.

Pushing the context window further on a 32 GB frame buffer requires precision. We swept symmetric and asymmetric quantization configurations across progressive context lengths to map the absolute limits of Blackwell-era hardware.

🛠️ The Science: Compiling Asymmetric Kernels

By default, llama.cpp only compiles GPU-accelerated Flash Attention kernels for symmetric KV cache formats (like f16 or q8_0/q8_0). If you attempt to use an asymmetrical format (like q8_0 Key and q5_1 Value) without custom compilation:

1. It silently falls back to CPU-based attention calculations.
2. Token generation speed collapses from ~72 tok/s to a useless ~2 tok/s.

To prevent this silent performance killer, you must compile llama.cpp from source with the explicit CMake flag:

cmake -B build -S . -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON

This instructs the compiler to build CUDA-accelerated Flash Attention kernels for all supported quantization variations, ensuring that asymmetric dequantization runs entirely on the GPU.

📊 Progressive Benchmark Sweeps

We ran progressive benchmarks on danwin (RTX 5090) comparing q8/q8 and q8/q5_1 KV cache formats under full GPU offloading (-ngl 99, -fa on, -r 1 to isolate static VRAM footprints):

Performance Observations:

• Speed remains intact: Prefill and decode speeds for q8/q5_1 are practically identical to (and at shorter contexts, faster than) q8/q8. The Blackwell architecture handles the asymmetric dequantization math with zero overhead.
• Prefill scaling: Prefilling 192k tokens is an extremely heavy task, clocking in at ~1,168 tokens/s (taking ~160 seconds). Because Gemma-4-31B uses 10 global attention layers, the attention matrix scales quadratically $O(N^2)$, requiring 368 billion operations during prefill. However, in production, you only pay this cost once when loading the conversation, while subsequent turns leverage the fast 72 tok/s decode speed.

📉 Measuring Quantization Decay: Why KL Divergence Rules Long Context

When evaluating the impact of KV cache quantization, standard perplexity (PPL) is a blunt instrument. In ultra-long context scenarios (100k+ tokens), perplexity is dominated by local linguistic structures (grammar, vocabulary) and the model's base training. A quantized KV cache could completely destroy the model's ability to recall specific distant details (like the needle-in-a-haystack task) while showing no visible decay in global perplexity.

To mathematically prove that the q8_0 Key / q5_1 Value cache does not degrade model performance, we must measure Kullback-Leibler (KL) Divergence relative to the unquantized baseline:

$$D_{KL}(P \parallel Q) = \sum_{x \in X} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$$

Where $P(x)$ is the reference probability distribution of the unquantized model's next-token predictions, and $Q(x)$ is the quantized model's distribution.

• Trace the Curve: By measuring $D_{KL}$ step-by-step as a function of context length (from 0 to 192k tokens), we get a granular look at information loss.
• If the curve stays flat and low: The quantized KV cache successfully preserves the exact attention routing of the reference model across the entire context window.
• If the curve spikes at longer contexts: The quantization is introducing noise that corrupts the global attention layers, causing retrieval failure long before it registers as a perplexity drop.

Evaluating KV cache quantization via KL divergence curves over long context sweeps ensures that capacity gains do not come at the expense of retrieval reliability.

💾 Mapping the Goldilocks Zone (VRAM Math)

To run Gemma-4-31B in a production agentic loop, we don't just run raw text. We also host:

1. The MTP speculative draft model (mtp-gemma-4-31B-it.gguf): ~3.8 GB
2. The F16 vision projector (mmproj-F16.gguf): ~0.8 GB
3. Graphics driver overhead & CUDA context: ~4.0 GB

This creates a static baseline memory footprint of ~23.8 GB VRAM before allocating the KV cache. With 32.6 GB of physical VRAM on the RTX 5090, we have an active memory budget of ~8.8 GB for the KV cache.

Here is how the context capacity limits play out:

• Under q8/q8:
  • 128k context uses ~5.96 GB $\rightarrow$ 29.76 GB total (leaves a safe 2.8 GB headroom).
  • 192k context uses ~8.73 GB $\rightarrow$ 32.53 GB total (essentially saturates the card, triggering driver memory paging, collapsing performance to single digits, or crashing WSL2).
• Under q8/q5_1:
  • 128k context uses ~4.96 GB $\rightarrow$ 28.76 GB total (leaves 3.8 GB headroom).
  • 192k context uses ~7.27 GB $\rightarrow$ 31.07 GB total (leaves a perfectly stable 1.5 GB VRAM headroom).

By switching to q8/q5_1, we save 1.3 GB of VRAM at 192k. This is the difference between a running system and a hard crash. Bumping the target to 208k in production would require 32.40 GB of VRAM, leaving less than 200 MB of safety margin—too close to the physical edge.

Thus, 192k is the Goldilocks context size for a fully loaded Gemma-4-31B stack on a single 32 GB card.

🔮 The Future: Blackwell-Optimized Kernels

While 72 tok/s decode and 1.2k tok/s prefill at 192k context are mind-blowing results for a single-GPU workstation, we are still pushing the hardware to its absolute limits.

The next frontier lies in native support for NVFP4 (NVIDIA 4-bit Floating Point) and dedicated Blackwell-optimized tensor kernels. Right now, llama.cpp emulates the dynamic 4-bit weights during execution. When upstream software natively leverages the Blackwell tensor cores' native FP4 arithmetic, we can expect the model weights footprint to drop further, and execution speed to scale even higher—potentially unlocking a full, stable 256k context window on the same 32 GB card.

For now, configuring asymmetrical q8_0 Key / q5_1 Value KV cache is the absolute state-of-the-art configuration for running long-context local LLMs.

← All writings · Home