How the Local AI Inference Ecosystem Matured: GGUF, Ollama, and Hardware Trade-offs

A site called canirun.ai recently surfaced on Hacker News with nearly 400 points. The concept is borrowed from the old “Can You Run It?” PC gaming compatibility sites: enter your hardware specifications, get a verdict on which AI models will run and how well. The fact that this category of tool now exists for LLMs says something meaningful about where the local inference ecosystem has landed.

Two years ago, running a language model locally meant compiling llama.cpp from source, hunting for the right quantized weights on Hugging Face, and doing mental arithmetic about VRAM limits with no tooling to help. Today you install Ollama with a single command and pull a model by name. The ecosystem has matured to the point where there are enough models, enough hardware configurations, and enough tooling that a compatibility matrix actually makes sense as a product.

The GGUF Format and the Supply Chain It Created

The dominant format for local inference is GGUF, which replaced GGML in August 2023. A GGUF file is self-contained: it bundles model weights, tokenizer, and configuration in a single binary. Tools built on llama.cpp, including Ollama, LM Studio, GPT4All, and koboldcpp, all speak GGUF natively. This standardization enabled a supply chain of pre-quantized models that makes local deployment straightforward. The Hugging Face model hub hosts thousands of GGUF variants across dozens of base models, covering multiple quantization levels for each.

The naming convention is worth understanding. Q4_K_M means 4-bit weights using the k-quant method, medium size variant. The _K suffix marks k-quants, a mixed-precision technique that applies different bit widths to different tensor types rather than quantizing everything uniformly. This preserves quality better than naive uniform quantization at the same average bit depth. The _M and _S suffixes indicate medium and small variants within a given k-quant level, trading a small amount of quality for reduced file size.

For a 7B model, the practical reference points are: Q4_K_M produces a roughly 4.1 GB file and fits in 8 GB VRAM alongside a modest context cache. Q8_0 at about 7.7 GB is nearly lossless, within 1% of float16 quality, and worth choosing if VRAM allows it. Q6_K and Q5_K_M split the difference cleanly for 8-12 GB configurations. Below Q4, quality degrades noticeably on complex reasoning tasks, though Q3 is viable when VRAM is the binding constraint.

More recent work on importance-matrix quantization (imatrix) improves on these baselines. IQ4_XS uses a calibration dataset to identify which weights most affect model output and quantizes those at higher precision, often outperforming Q4_K_M at a similar file size. The llama.cpp project has shipped imatrix variants since mid-2024, and they are worth reaching for when quality matters at a given size budget.

Beyond GGUF, there are GPU-specific formats with different trade-offs. GPTQ and AWQ store weights as .safetensors files and run on NVIDIA hardware through tools like ExLlamaV2 or text-generation-webui. EXL2 from the ExLlamaV2 project supports variable bit widths from 2.5 to 8 bits and often achieves higher throughput than GGUF on NVIDIA hardware at a given quality level, though it sacrifices portability. For Apple Silicon, Apple’s MLX framework (released late 2023) uses its own format and provides inference that rivals or exceeds llama.cpp Metal on some model architectures.

The Tools That Removed the Friction

llama.cpp is the foundation. Written in C/C++ with no required dependencies, it compiles and runs on CPU (with AVX2 and AVX512 acceleration), NVIDIA via CUDA, AMD via ROCm, Apple Silicon via Metal, and cross-platform via Vulkan. The project ships a local HTTP server with an OpenAI-compatible API, meaning any client library targeting the OpenAI API can point at a local llama.cpp instance by changing one URL. Recent versions support speculative decoding, where a small draft model proposes candidate tokens that the main model verifies in parallel, yielding 2-3x throughput improvements in favorable configurations. Flash Attention 2 and continuous batching for multi-user serving are also built in.

Ollama wraps llama.cpp with a Docker-inspired CLI and model registry. The workflow maps onto container tooling directly: ollama pull mistral fetches the model, ollama run llama3 pulls and starts it, ollama list shows what is local. GPU detection, quantization selection, and layer offloading happen automatically. The REST API it exposes on port 11434 is OpenAI-compatible, so integrating local models into existing code is usually a one-line change. Recent versions added multi-model serving with automatic GPU memory management.

LM Studio is oriented toward users who prefer a GUI. Its integrated Hugging Face model browser lets you search for and download GGUF files without leaving the app, and it provides a chat interface alongside the API server. The 0.3 release added an MLX backend for Apple Silicon and multi-model serving. For quickly testing a model’s behavior or for non-technical users, it removes significant friction.

For more demanding use cases, text-generation-webui supports multiple backends, LoRA adapter loading, and detailed sampling parameter control. vLLM is the production option for multi-user serving on NVIDIA hardware, using PagedAttention for efficient KV cache management and continuous batching to maximize GPU utilization under concurrent load.

Hardware Tiers and What to Expect

For a Llama 3 8B model at Q4_K_M, an RTX 4090 (24 GB VRAM, 1008 GB/s memory bandwidth) generates around 100-130 tokens per second. An RTX 3080 with 10 GB VRAM fits the same model and runs at 50-70 tokens/sec. An Apple M2 Pro with 32 GB unified memory handles the same model at 45-60 tokens/sec. For interactive chat, anything above 30 tokens/sec feels responsive. Below 15 tokens/sec, the wait between sentences becomes disruptive.

The picture changes for 70B-class models. A Q4 Llama 3 70B file is about 39 GB. A single RTX 4090 has 24 GB VRAM, so the model does not fit. Offloading layers to system RAM through PCIe drops throughput to 5-10 tokens/sec in mixed mode. An M2 Ultra with 192 GB of unified memory holds the full model and generates at 20-35 tokens/sec. An M3 Ultra manages 30-45 tokens/sec. Apple Silicon’s architectural advantage for large models is genuine: the unified memory pool accessible to both CPU and GPU at high bandwidth (800 GB/s on M2 Ultra, 400 GB/s on M3 Max) makes it the most practical single-device option for 70B inference without committing to a multi-GPU setup.

NVIDIA’s RTX 5090, which shipped in late 2024 and early 2025, changes the single-GPU ceiling: 32 GB GDDR7 at approximately 1.8 TB/s bandwidth is roughly double the 4090’s bandwidth. A 70B model at Q3 quantization runs around 29 GB, putting it within reach of a single consumer card with solid throughput.

For CPU-only inference, a mid-range desktop chip generates at 8-15 tokens/sec on a 7B Q4 model. That is workable for batch processing but frustrating for interactive use. The efficient small models have improved enough that CPU-only inference deserves reconsideration for constrained hardware: Phi-3 Mini at 3.8 billion parameters fits in about 2.3 GB at Q4 and achieves GPT-3.5-class quality on many benchmarks. Gemma 2 2B at roughly 1.8 GB Q4 runs on nearly any hardware with tolerable speed.

What a Compatibility Checker Does and Does Not Solve

The value of canirun.ai is that it short-circuits a tedious feedback loop most new users go through: browse model cards, estimate VRAM requirements, download several gigabytes, watch the inference engine run out of memory, search for a smaller quantization, repeat. For someone who does not yet have these numbers memorized, a compatibility database saves real time.

The limitation is that compatibility and usability are different things. A site that says a model “runs” on your hardware is technically correct even if generation happens at 8 tokens/sec with partial CPU offloading and a 2 GB context limit. The tool works best as a filter that eliminates obvious mismatches, not as a substitute for understanding what the numbers mean.

For someone starting out, the practical path is Ollama with one of the smaller Llama 3 variants. The 3B Llama 3.2 model is a reasonable default: modest hardware requirements, fast generation, and quality that holds up for most casual uses. For Apple Silicon users, LM Studio with the MLX backend is worth trying, especially on M2 or M3 chips where the Metal optimization path is well-developed.

That canirun.ai landed near the top of Hacker News reflects where the community is. Local inference has moved from a niche hobby requiring manual compilation to something enough people want to do seriously that tooling, format standardization, and compatibility databases all make sense as investments. The engineering work that enabled that shift, starting with llama.cpp’s portability and continuing through GGUF’s standardization and Ollama’s packaging, happened fast. A compatibility checker is a useful artifact of that maturity.