The case against Ollama laid out by Sleeping Robots comes down to a structural fact: Ollama wraps llama.cpp, and llama.cpp ships its own capable server binary. The wrapper introduces version lag, format friction, and a constrained parameter surface. That argument is accurate but tends to stay at the level of principle. The vendor pin becomes more concrete when you look at what it actually delays: importance-matrix quantization, flash attention, speculative decoding, and KV cache quantization. All four have been stable in llama.cpp for months before Ollama’s vendor commit catches up, and none is cleanly exposed through Ollama’s API surface.
IQ Quants: Better Quality at the Same Bit Width
Standard GGUF k-quants like Q4_K_M partition model weights into superblocks and quantize within each block, with larger superblock sizes for attention and feed-forward matrices to preserve quality. The IQ series takes a different approach. Before quantizing, a calibration dataset runs through the full-precision model, and the resulting activations identify which weights are most consequential to output quality. Quantization then assigns bits non-uniformly, spending more precision where activations are large and less where they are not.
The perplexity difference is measurable. Community benchmarks in the llama.cpp repository on Llama-2-13B show IQ3_M achieving lower perplexity than Q4_K_M while fitting in roughly 5.4 GB instead of 7.3 GB. For hardware where VRAM is the binding constraint, this means fitting a larger model into the same memory budget, or getting meaningfully better output quality from the same model size at no additional memory cost.
The bartowski organization on Hugging Face maintains IQ and k-quant builds for most major models with consistent naming conventions. Using an IQ quant in llama-server requires nothing beyond pointing the server at the file:
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
--include "Meta-Llama-3.1-8B-Instruct-IQ4_XS.gguf" \
--local-dir ./models
llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-IQ4_XS.gguf \
-ngl 35 \
--port 8080Ollama’s registry does not carry IQ quant variants for most models, and support for loading them via a Modelfile has been inconsistent across releases depending on the vendor commit.
Flash Attention: Context Length Without the Quadratic Memory Cost
Standard transformer attention materializes the full N×N attention score matrix in memory before computing the softmax. For a sequence of length N, memory usage scales quadratically. Flash attention recasts the computation as a tiled kernel that processes the matrix in blocks fitting in fast SRAM, accumulating the output without storing the full matrix. Memory usage with respect to sequence length becomes linear.
At 16k token contexts on a GPU with 12 GB VRAM, the difference between running with and without flash attention often determines whether inference completes at all. Without it, the KV cache and intermediate attention buffers for a Llama 3 8B model at 16k context can exhaust available VRAM on most consumer GPUs. With flash attention, the same configuration runs within budget and at substantially better throughput, since the attention kernel is no longer memory-bandwidth-bound at longer sequences.
llama-server exposes flash attention as a startup flag:
llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-ngl 35 \
-c 16384 \
--flash-attn \
--port 8080Ollama exposes no equivalent, and whether flash attention is enabled in a given Ollama release depends on the internal decisions baked into that release’s llama.cpp commit.
Speculative Decoding: The Draft Model Speed Multiplier
Autoregressive generation produces one token per forward pass, and forward pass cost scales with model size. Speculative decoding inserts a small, fast draft model into the loop. The draft model proposes several tokens in sequence; the target model then verifies all proposed tokens in a single parallel forward pass, accepting tokens up to the first position where its distribution meaningfully disagrees with the draft. When the draft is accurate, you generate multiple tokens at the cost of one draft sequence plus one target verification pass, rather than multiple target passes.
Acceptance rates are task-dependent. For instruction-following and code generation, where outputs follow predictable patterns, rates of 70-80% are typical with a well-matched draft model, translating to 2-3x effective throughput improvement. The draft model must share tokenization with the target; for Llama 3.1 8B, a Llama 3.2 3B model works as an effective draft.
llama-server exposes this through the draft model path flag and a parameter controlling how many tokens the draft proposes per round:
llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--model-draft ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
-ngl 35 \
--flash-attn \
--port 8080The combined VRAM footprint is the sum of both models. On a 24 GB card, this configuration runs a full 8B model with a 3B draft and achieves throughput that would otherwise require a significantly larger GPU. Ollama has no equivalent configuration surface for speculative decoding.
KV Cache Quantization: The Other Memory Lever
The KV cache stores precomputed attention keys and values for previous tokens. At long context lengths, it becomes the primary VRAM consumer, not the model weights. In float16, the KV cache for a Llama 3 8B model at 32k context grows to roughly 8 GB across all layers. llama.cpp supports quantizing the KV cache to 8-bit integers, halving that cost at a small quality reduction:
llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-ngl 35 \
-c 32768 \
--flash-attn \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--port 8080With q8_0 KV cache and flash attention together, a 32k context configuration for an 8B model fits in 12 GB VRAM. Without KV cache quantization, the same context length exceeds that budget. Ollama does not expose --cache-type-k or --cache-type-v.
What the Pattern Reveals
These four capabilities share a structure. Each is a genuine engineering advance in llama.cpp that has been stable for months; each requires only a flag or a different model file to use; none is cleanly accessible through Ollama. The shared root cause is the vendor pin.
Ollama validates a specific llama.cpp commit before each release, which produces a more consistent end-user experience. The cost is that every Ollama installation is a snapshot of a rapidly moving inference engine, and that snapshot ages as llama.cpp continues to ship. Users building on Ollama are not building on llama.cpp; they are building on Ollama’s most recently validated subset of it.
The migration path is the same for all four features: download a current llama.cpp binary from the releases page, download a GGUF from Hugging Face, and start llama-server with the flags your workload needs. Any client targeting Ollama’s /v1/chat/completions endpoint works against llama-server with one change: replace http://localhost:11434/v1 with your llama-server base URL.
The broader Sleeping Robots argument is that the local LLM ecosystem has outgrown the abstraction. Looking at what is sitting on the other side of the pin commit makes that argument specific. The features are not hypothetical: they exist, they work, and llama-server exposes them today.