Ollama arrived at the right moment. Running a large language model locally used to mean building llama.cpp from source, tracking down a GGUF file on Hugging Face, and deciphering a pile of command-line flags before you saw any output. Ollama collapsed all of that into ollama run mistral and normalized the idea that local inference should feel like pulling a Docker image. The mental model worked, and adoption followed.
A recent post on SleepingRobots argues that the local LLM ecosystem has matured past the point where Ollama provides net value, and the 600-point Hacker News response suggests the argument landed with real traction. Having worked through several local inference setups for bot and automation work, I find myself largely agreeing, for reasons worth unpacking carefully.
The Wrapper Is Thicker Than It Looks
Ollama is written in Go and ships with a statically compiled build of llama.cpp embedded inside. When you run ollama serve, you get a persistent HTTP daemon on port 11434 that exposes an OpenAI-compatible API alongside Ollama’s own endpoints. The daemon handles GPU detection, model loading, context management, and routing.
The wrapping is convenient, but it introduces a lag between llama.cpp improvements and what Ollama actually ships. llama.cpp moves fast: speculative decoding, flash attention, grouped-query attention optimizations, new quantization schemes like IQ2_XXS and Q8_0_R4, and architecture support for new model families often land in llama.cpp days or weeks before Ollama surfaces them. When you install Ollama 0.x.y, you are effectively pinned to whatever llama.cpp commit was vendored in that release, with no upgrade path short of waiting for a new Ollama release.
This is not hypothetical. Early Phi-3 support, Mamba SSM inference, and several of the more aggressive k-quant improvements went through exactly this pattern: available in llama.cpp main, inaccessible to Ollama users for weeks. With direct llama.cpp usage, you build from main or grab a nightly binary and move on.
The Model Storage Problem
Ollama does not simply point at GGUF files on your filesystem. It maintains its own model registry under ~/.ollama/models/, storing models as blobs with content-addressed manifests, structured similarly to how container registries store image layers. When you run ollama pull llama3, Ollama fetches from its registry at ollama.com, which hosts a curated but limited subset of what is available on Hugging Face.
This creates friction in both directions. If you already have a GGUF downloaded from Hugging Face, getting Ollama to recognize it requires creating a Modelfile:
ollama create my-model -f - <<EOF
FROM /absolute/path/to/model.gguf
EOFDepending on whether your filesystem supports reflinks (Btrfs, APFS), this either duplicates the file or creates a hard-link. On ext4 or NTFS, you may end up with two copies of a 20 GB model consuming double the disk space. The registry covers popular model families well but lags behind Hugging Face on fine-tuned variants, uncommon quantizations, and anything that the Ollama maintainers have not prioritized pulling in.
Parameter Control at Inference Time
The Ollama API exposes a subset of llama.cpp’s generation parameters: temperature, top_p, top_k, repeat_penalty, seed, num_predict, and a few others. llama.cpp itself has a much larger surface area. Grammar-constrained generation using GBNF grammars, logit bias arrays, mirostat sampling (both v1 and v2), per-token logprob output, and fine-grained context management are either absent or baked into static per-model Modelfile configuration rather than per-request options.
For grammar-constrained output, this matters considerably. If you are building something that needs deterministic structured JSON from a local model, llama.cpp’s grammar support gives you that without retry loops:
from llama_cpp import Llama, LlamaGrammar
json_grammar = LlamaGrammar.from_string(r'''
root ::= object
value ::= object | array | string | number | ("true" | "false" | "null")
object ::= "{" ws (string ":" ws value ("," ws string ":" ws value)*)? "}"
array ::= "[" ws (value ("," ws value)*)? "]"
string ::= "\"" ([^\"\\] | "\\" ["\\/bfnrt] | "\\" "u" [0-9a-fA-F]{4})* "\""
number ::= "-"? ([0-9] | [1-9] [0-9]*) ("." [0-9]+)? ([eE] ["+\-"]? [0-9]+)?
ws ::= ([ \t\n] ws)?
''')
llm = Llama(model_path="models/mistral-7b-instruct.Q4_K_M.gguf", n_gpu_layers=-1)
result = llm("Generate a JSON object with name and age fields.", grammar=json_grammar)Ollama has no equivalent of this at request time. The Modelfile can set a PARAMETER, but that applies to the model globally, not per-call.
What llama-server Gives You Directly
llama.cpp ships its own HTTP server binary, llama-server, which provides the same OpenAI-compatible API surface as Ollama but with access to the full parameter set. Starting it is straightforward:
./llama-server \
--model models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
--port 8080 \
--n-gpu-layers 35 \
--ctx-size 8192 \
--threads 8You point it at any GGUF on your filesystem, specify your hardware configuration explicitly, and get a server that is always running the llama.cpp version you chose, not the one Ollama decided to vendor. The /completion and /v1/chat/completions endpoints accept the full parameter surface including grammar, logit_bias, and sampling mode selection.
The thing you give up is Ollama’s model management UX. There is no llama-server pull command; you download GGUF files manually or with a script. For most developers this is not actually a hardship, since Hugging Face’s CLI (huggingface-hub) handles it cleanly and gives you access to everything in the Hub.
The Broader Alternatives Landscape
llamafile: Mozilla’s llamafile project produces single-file executables using cosmopolitan libc that run on Linux, macOS, and Windows without installation. You download one file, make it executable, and run it. For reproducible distribution across machines or sharing a working setup with someone who should not need to manage dependencies, this is genuinely useful in a way Ollama is not.
vLLM: If you have a multi-GPU machine and care about throughput rather than convenience, vLLM is not in the same category as Ollama at all. Its PagedAttention mechanism and continuous batching handle dozens of concurrent requests at throughput that llama.cpp-based servers do not approach. The tradeoff is real: CUDA required, full-precision or bfloat16 weights rather than GGUF quantization, and significantly more VRAM. Ollama does not compete here, and neither does llama-server. vLLM is a production inference engine; the others are developer tools.
mlx-lm: On Apple Silicon, Apple’s MLX framework offers native performance that llama.cpp’s Metal backend does not always match, particularly for attention-heavy workloads on M-series chips. The integration with Hugging Face is direct:
pip install mlx-lm
python -m mlx_lm.generate \
--model mlx-community/Mistral-7B-Instruct-v0.2-4bit \
--prompt "Explain GGUF quantization in two paragraphs."The mlx-community organization on Hugging Face maintains MLX-converted versions of most popular models. If you are on an M2 or M3 Mac, this pipeline often outperforms Ollama on the same hardware.
Where Ollama Still Earns Its Place
None of this means Ollama is poorly built. The ollama ps output showing loaded models, the automatic model eviction when VRAM pressure increases, the multimodal support that surfaces cleanly through its API, and the Modelfile system for encoding system prompts alongside model weights are all well-considered features. For teams where the audience is not comfortable with command lines, or for use cases where the Docker-like mental model genuinely accelerates onboarding, Ollama’s UX advantage is real.
The problem arises when Ollama gets adopted for production services or research workflows primarily because it was the convenient starting point. At that juncture, its costs compound: you cannot update the underlying llama.cpp independently, you have a running daemon to manage as a service with its own failure modes, parameter control is constrained, and the model storage layout creates its own maintenance surface.
The local LLM ecosystem has developed enough that the bootstrapping problem Ollama originally solved is largely solved by the ecosystem itself. llama-server gives you the same API surface with direct hardware control. llama-cpp-python gives you in-process inference with the full parameter set. llamafile gives you distributable, reproducible single-binary setups. mlx-lm gives Mac users native performance. The choice between these tools is a real choice now, with real tradeoffs, rather than a choice between Ollama and a much steeper climb.
Ollama got a lot of people to run their first local model. That matters. The question in 2025 is whether it remains the right tool once you know what you are actually doing with local inference, and for most serious use cases, the answer has shifted.