Our MoE Model Started Spewing Garbage — What Auditing All 18,432 Expert Weights Revealed

The first step in debugging is never a fancy tool — it is isolating variables. When a model spits out garbage there are three broad suspects: the model file, the runtime (inference engine), and the configuration. So the very first thing we did was feed the same GGUF file to a different runtime.

That single cross-test cut the investigation in half. The file is innocent, and it is not a config difference — which means something in how the runtime handles weights is broken. Rather than diving straight into the source, we decided to first observe what was actually happening inside the model.

A bit of background first. An MoE (Mixture of Experts) model places several expert networks in each layer and activates only a subset of them per token. Qwen3-30B-A3B activates just 3B of its 30B parameters per token. Which naturally invites the idea: “if only part of the model is used at a time, why not park the idle experts in CPU memory and pull them in on demand — then even a small GPU could run a big model.” That is exactly the approach KTransformers takes.

We ran the test to verify whether a 30B-class MoE could work in a VRAM-constrained environment: experts offloaded to CPU, with only attention and embeddings on the GPU. And the very first inference came back as garbage.

Broken output means the numbers started breaking somewhere inside the model. So we fed identical input to a known-good reference (the original safetensors loaded in bf16) and to the problematic KTransformers (GGUF) model, then compared hidden state statistics side by side across all 48 layers.

Layer 0~1 : statistics nearly identical between the two models
Layer 2   : std dev 6.93 vs 4.69 (+48% vs reference)  ← divergence starts here
Mid layers: max values plateau around 1,400 (reference: 300~400)
Final     : distribution completely different -> output collapse

From Layer 2 onward the standard deviation jumped by more than 48%.When the entry point (embeddings) is clean but the distribution diverges starting at one specific layer, that is a strong signal that either that layer’s weights or the compute path handling it is broken. The suspect list narrowed from “the whole model” to “the expert weights in one particular layer.”

The next question: which weight is actually broken? This model has 48 layers × 128 experts × 3 projections (gate/up/down) = 18,432 expert weight matrices. Sampling a handful risks missing it, so we compared every single one — dequantizing the values from the GGUF and checking them matrix by matrix against the safetensors originals via cosine similarity.

Out of 18,432 weights, exactly 2 were seriously corrupted — and they sat in the very same Layer 2 where Step 1 showed divergence beginning. When the layer trace and the weight audit point at the same spot, you can be confident it is not a coincidence. And here is the important lesson: 18,430 healthy weights (cos > 0.95) still are not enough to save the model. A forward pass is a chain of multiplications, so a single error in an early layer gets amplified exponentially as it travels through 48 layers.

Finally we asked why thatmatrix in particular broke. GGUF’s Q4_K/Q6_K quantization stores weights grouped into blocks, so we dequantized the offending down_proj matrix one block at a time and compared each against the original. The result: one specific block’s dequantized values were off by as much as 14.5versus the original. The matrix was not uniformly bad — one block’s quantization reconstruction was badly out of line.

At that depth the symptom is fully explained: reconstruction error in one block → distorted output from that expert → hidden state divergence starting at Layer 2 → accumulation across 48 layers → output collapse. But one contradiction remained. Why was the same file perfectly fine under llama-cpp-python?

The answer lay in how the two runtimes handle quantized weights.

[GGUF Q4_K/Q6_K quantized weights]
  -> dequantize to bf16, keep permanently   <- 1st precision loss
    -> bf16 matrix multiply                 <- 2nd precision loss
      -> forward pass through 48 layers
        -> exponential error accumulation -> output collapse

In short: “for quantized models, accumulation precision is everything.” Quantization is lossy compression by definition, so it always carries small errors — and where those errors get absorbed is what separates a good runtime from a bad one. MoE is especially exposed: with so many experts, the number of weight matrices is tens of times that of a dense model, so the odds that at least one reconstruction goes wrong are proportionally higher. That is why the same pipeline can survive on a dense model and fall over first on MoE.

With the cause identified, we tried to see whether it could be patched. Every attempt hit a wall.

# Always run large-memory experiments under a hard memory cap
# On overflow only this scope dies and the system survives (cap ~80% of physical RAM)
systemd-run --user --scope -p MemoryMax=80% -p MemorySwapMax=0 \
  python debug_script.py

The lesson is clear. Model debugging is itself a heavy workload, so your experiment code needs production-grade safeguards too (memory caps, swap disabled). The work of finding a fault should not become the cause of an outage.

Our final call on this test was simple. If you have enough VRAM, the whole premise of CPU offloading is unnecessary. When the model fits entirely on the GPU, there is no reason to push experts onto the CPU and take on precision and compatibility risk. We settled on SGLang — a GPU-only engine — with an AWQ-quantized dense model for production serving, and it runs stably at 135 tok/s for a single request.

That does not make KTransformers a bad tool. If you have to run a 30B-class MoE on a system with only 16–24GB of VRAM, CPU offloading is just about the only option, and it earns its keep there. Just verify first that the precision issue described here has been addressed (f32 accumulation or on-the-fly dequantization) before you rely on it.

For measured MoE vs dense comparisons, see MoE vs Dense in Practice, and for the Korean-language quality evaluation that led us to our production model, see our benchmark of 6 local LLMs.