2026-06-26 — Dan Billings

100 tok/s at 100k Context: Hosting Gemma-4-31B QAT + MTP + Stable Diffusion on the RTX 5090

Dan Billings — 2026-06-26

[!NOTE] This is the definitive guide to configuring the RTX 5090 (32 GB GDDR7) as a local AI powerhouse. We're talking 128k context, QAT-calibrated weights, Multi-Token Prediction (MTP) drafting, real-time telemetry, and high-performance image generation—all running locally on consumer hardware.

We previously pushed the RTX 4090 to its limits, squeezing 84 tok/s out of Qwen 3.6 with MTP. But memory bandwidth and VRAM constraints kept us in a cage.

Now, the RTX 5090 has landed. With 1.8 TB/s of GDDR7 memory bandwidth, 32 GB of VRAM, and native Blackwell FP4 tensor cores, the hardware baseline has shifted. This post walks through the optimal stack for the 5090: hosting Google's massive Gemma-4-31B (QAT-quantized), driving it at ~100 tokens/sec even as context approaches 100k, and parallelizing it with a CUDA-accelerated stable-diffusion.cpp instance.

The entire environment is declaratively defined as type-safe Scala IaC in src/main/scala/ansible/examples/Rtx5090Setup.scala and src/main/scala/ansible/config/DanwinLlm.scala. If you run the playbook, it outputs the exact systemd units, library paths, and model links analyzed below.

The Model: Gemma-4-31B-it-qat (UD-Q4_K_XL)

When choosing the "absolute best" model to host on a 32 GB frame buffer, size-to-quality ratio is everything. A vanilla 31B model at FP16 uses 62 GB—unusable. Standard post-training quantization (like GPTQ or naive GGUF quants) degrades reasoning, logic, and long-context recall.

The solution is Quantization-Aware Training (QAT).

Google’s unsloth/gemma-4-31B-it-qat-GGUF was trained with quantization in the loop. By simulating the loss of precision during training, the model adapts its weights to counteract quantization noise. The result: the UD-Q4_K_XL (Unsloth Dynamic 4-bit Extra Large) weights (~15.2 GB) deliver the quality of a native 16-bit model while running at a fraction of the size.

Speculative Decoding via Multi-Token Prediction (MTP)

Speculative decoding normally requires running a separate, smaller "assistant" model alongside the target. But Gemma 4 MTP uses a unified architecture where the assistant drafter shares the input embedding table with the target.

Our configuration pairs the main QAT model with the root-level MTP draft assistant:

• Target: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
• Drafter: mtp-gemma-4-31B-it.gguf (root MTP draft, ~0.85 GB)

[!IMPORTANT] The MTP drafter is wired directly into the attention stack. Upstream llama.cpp PR #23398 (packaged in our pinned build 02810c7) teaches draft-mtp to source missing embeddings from the target model. Do not use external --model-draft assistant configurations that don't share embeddings, as they will cause a shape mismatch or VRAM leak.

The Blackwell Speculative Curve

On the RTX 4090 (Ada Lovelace), speculative decoding suffered from a brutal performance cliff: --spec-draft-n-max 3 gave 84 tok/s, but bumping to 4 collapsed throughput to 6 tok/s (a 13× regression) due to warp-scheduling overhead in the drafter kernel.

On Blackwell (RTX 5090), the MTP cliff does not exist.

Because of GDDR7 bandwidth and improved Blackwell schedulers, the drafter kernel stays on its fast path. While --spec-draft-n-max 3 remains the mathematical sweet spot for draft acceptance vs. verifier overhead, raising it to 4 or 5 causes only a gradual, graceful decay rather than a collapse. We pin --spec-draft-n-max 3 to achieve an MTP acceptance rate of 76% - 92%.

VRAM Sizing & Llama Server Configurations

Fitting a 31B parameter model with a 128k context window into 32 GB of VRAM requires precision. If you spill a single byte into Windows host memory (WSL2 sysmem fallback), throughput drops from 100 tok/s to single digits.

Here are the critical settings configured in DanwinLlm.scala:

1. Windowed Sliding-Window Attention (SWA) vs. Full Attention

Gemma 4 uses a hybrid attention mechanism. Forcing full attention across the entire context window (--swa-full) is a memory sink.

• At 32k context, --swa-full consumes 32,024 MiB of VRAM, leaving only 0.6 GB free. Running 128k or 256k under this configuration is a guaranteed OOM.
• By keeping --swa-full off, llama.cpp uses windowed SWA (caching only the sliding window for layers that support it). At 128k context, this consumes only 27,262 MiB of VRAM, leaving 5.3 GB free for compute buffers.

2. KV Cache Quantization (q8_0)

We configure --cache-type-k q8_0 --cache-type-v q8_0.

• Capacity: Quantizing the Key and Value caches to 8-bit cuts memory footprint in half. A 128k context window at q8_0 uses only ~5.1 GB of cache space.
• Speed: Unlike q4_0 KV cache (which requires complex per-channel scale and zero-point lookups during decode), q8_0 has negligible dequantization overhead. Switching from q4_0 to q8_0 KV cache yields a 54% decode speedup (moving from 69 tok/s to 106 tok/s) on the 5090.

[!TIP] What's Next? To push the context window to 200k or 256k, context window quantization to q8 (key) / q5_1 (value) (or q5_1 for both) is the next optimization. This will drop the cache footprint further while keeping the perplexity degradation minimal.

3. Physical & Unified Batch Sizes (-b 2048 -ub 2048)

The unified batch size (-ub) controls how many tokens are processed concurrently during prefill.

• On the 4090, setting -ub higher than 512 would OOM under concurrent slot load.
• On the 5090, the GDDR7 capacity allows us to match -b 2048 -ub 2048. This unlocks a prefill throughput of ~1,036 tokens/s, meaning a 30k token prompt ingestion takes just under 11 seconds.

The Complete Service Invocation

Our Ansible runner copies this optimal systemd unit to /etc/systemd/system/danwin-llm.service:

[Unit]
Description=llama.cpp inference server — danwin-llm
After=network.target

[Service]
Type=simple
User=dan
Environment=LD_LIBRARY_PATH=/usr/lib/wsl/lib:/usr/local/cuda-13.3/lib64:/usr/lib/x86_64-linux-gnu
Environment=CUDA_VISIBLE_DEVICES=0
ExecStart=/usr/local/bin/llama-server \
  --model /home/dan/models/gemma31b/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf \
  --model-draft /home/dan/models/gemma31b/mtp-gemma-4-31B-it.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  --ctx-size 131072 \
  --parallel 1 \
  -fa on \
  -b 2048 \
  -ub 2048 \
  --cont-batching \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-ngl 99 \
  --spec-draft-n-max 3 \
  -fit off \
  --metrics
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Real-Time Telemetry & Metrics

We collect telemetry on danarch:9090 (our Prometheus hub) scraping danwin:8080/metrics. The dashboards defined in src/main/scala/ansible/examples/DanarchObservability.scala monitor several critical metrics:

1. Generation Throughput: llamacpp:predicted_tokens_seconds{instance="danwin-5090"} Under real-world loads (Hermes Agent running near-full 100k context), we maintain ~100 tokens/s.
2. Prefill Rate: llamacpp:prompt_tokens_seconds{instance="danwin-5090"} Tracks the ingestion speed, peaking at ~1,036 tok/s.
3. WSL OOM & Sysmem Fallback Alerts:
   • node_memory_MemAvailable_bytes{instance="danwin:9100"} < 4294967296 Fires if available system RAM falls below 4 GiB, signaling a high risk of Windows killing the entire WSL2 VM.
   • llamacpp_llm_sys_rss_bytes > 5368709120 (while VRAM is saturated) Fires if llama-server system RAM RSS exceeds 5 GB. This alerts us that CUDA paged memory fallback is active (which kills token generation speed).

Co-hosting: stable-diffusion.cpp on the 5090

To maximize the 5090's utility, we run image generation side-by-side using Leejet's stable-diffusion.cpp. Because it's written in pure C/C++ with GGML/CUDA bindings, its footprint is tiny and it performs dramatically faster than Python-based webUIs.

The codebase compiles and configures sd.cpp via src/main/scala/ansible/examples/Rtx5090SdCpp.scala.

Compilation & Installation

The playbook clones stable-diffusion.cpp recursively, hooks into the CUDA 13.3 toolchain, and compiles with native CUDA support:

cmake -B /home/dan/sd-build/build -S /home/dan/sd-build \
  -DSD_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCUDAToolkit_ROOT=/usr/local/cuda-13.3 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc

cmake --build /home/dan/sd-build/build --config Release -j$(nproc)

It installs sd-cli and sd-server (an OpenAI-compatible HTTP API) directly to /usr/local/bin.

Model Directory Layout

/home/dan/sd-models/
├── checkpoints/   <-- Drop Realistic Vision v6, Juggernaut XL here (.safetensors)
├── loras/         <-- Style adapters
└── upscale/       <-- Drop RealESRGAN or NMKD Superscale here (.pth)

SSH Integration (Crostini Setup)

Since we want to generate images programmatically or execute CLI calls from our thin-client dev machines, we open a pipeline directly into WSL2. The playbook runs a PowerShell script to pierce the Windows Firewall, allowing incoming SSH traffic on port 2222 from the Crostini (Chromebook) subnet:

New-NetFirewallRule -DisplayName 'Crostini SSH' -Direction Inbound -LocalPort 2222 -Protocol TCP -RemoteAddress '100.115.92.0/28' -Action Allow

Now, we can SSH straight in:

ssh -p 2222 dan@danwin

And immediately kick off image generations or scale requests via sd-cli.

Verdict: The 5090 Capabilities Shift

Running Gemma-4-31B QAT + MTP at ~100 tok/s over a 100k context window crosses a qualitative threshold.

• At 35 tok/s, you watch the model think.
• At 100 tok/s, the output streams faster than comfortable reading speed. The thinking block is a brief flash, and the response is essentially instantaneous.

Co-hosting this with stable-diffusion.cpp creates an uncompromising, fully offline, private AI environment. If you have the budget for a 5090, don't run it with naive defaults. Use QAT weights, set up windowed SWA, quantize your KV cache to q8_0, and watch Blackwell fly.

Next up: sweeping q8 key / q5_1 value KV caches to see if we can stretch the context past 200k without hitting the sysmem fallback threshold.

← All writings · Home