· 7 min read ·

FreeBSD and the Case for Coherent Systems

Source: hackernews

The Hacker News thread around this love letter to FreeBSD has 256 comments and 506 points at the time of writing, which is a pretty good signal that the FreeBSD community still has things to say that Linux users find worth reading. What strikes me every time these threads surface is that the strongest arguments for FreeBSD are not about any single feature. They are about coherence.

Linux is remarkable engineering. It powers most of the internet, most phones, most supercomputers. But it is also, in a meaningful sense, an assemblage. The kernel is Linux. The init system might be systemd or OpenRC or runit. The C library is usually glibc but sometimes musl. The toolchain is GNU or LLVM depending on the distribution. The firewall is iptables or nftables. Each layer has its own release cycle, its own documentation, its own compatibility surface, and its own set of people who can tell you when something breaks across layer boundaries.

FreeBSD does not work this way. The base system, from the kernel through the C library through the userland utilities, is developed and released as a single unit from a single source tree. This is not a minor organizational detail. It is load-bearing.

The Single Source Tree

FreeBSD maintains one repository at src.freebsd.org that contains everything: the kernel, libc, the compiler infrastructure, ssh, the firewall, the virtual machine monitor, and the container system. When you check out FreeBSD HEAD, you get a complete operating system, not a kernel waiting to be assembled into one.

The practical consequence is that API and ABI guarantees can be maintained across the entire base system simultaneously. When the kernel team changes a system call interface, the userland change lands in the same commit or the same review cycle. There is no version matrix to consult. There is no “this version of the kernel requires this version of glibc” problem because the kernel and the C library are the same project.

The branch model reflects this. FreeBSD runs three branches: CURRENT (the development trunk), STABLE (the integration branch for a given major version), and RELEASE (the tagged, supported release). FreeBSD 14, released in December 2023, follows this pattern. FreeBSD 14.1 shipped in June 2024 with accumulated fixes and driver updates. Each release is a snapshot of the entire system, not just the kernel.

Jails Before Containers Were a Thing

FreeBSD Jails were introduced in FreeBSD 4.0 in 2000, roughly five years before Linux namespaces and a decade before Docker. The mental model is simpler than Linux containers in some respects: a jail is a restricted process tree with its own filesystem root, network stack, and process namespace. You can create one with jail -c:

jail -c path=/jails/myjail host.hostname=myjail ip4.addr=192.168.1.10 command=/bin/sh

What makes jails interesting architecturally is that they are a kernel primitive, not a userspace composition of several kernel primitives (namespaces, cgroups, seccomp, capabilities) that tools like Docker assemble into something container-shaped. The FreeBSD kernel understands jails natively. The security boundary is enforced at the syscall level, not constructed from orthogonal mechanisms that have to be carefully layered.

Jails also compose cleanly with ZFS. A common pattern is to create a ZFS dataset per jail, which gives you snapshot-based backup, cloning for fast provisioning, and quota enforcement essentially for free:

zfs create zpool/jails/myjail
jail -c path=/jails/myjail ...
zfs snapshot zpool/jails/myjail@baseline

This is not impossible on Linux. It is just something you assemble yourself from btrfs or ZFS-on-Linux and your container runtime, with varying degrees of integration depending on which versions of which tools you have.

ZFS as a First-Class Citizen

Sun Microsystems released ZFS under the CDDL in 2005. FreeBSD integrated ZFS into the base system in 2008, making it available as a boot filesystem and a fully supported part of the installer. Linux distributions, because the GPL and CDDL have licensing incompatibilities, could not include ZFS in the kernel directly. ZFS on Linux exists as an out-of-tree module, maintained by the OpenZFS project, and works well, but it requires installation as a separate package and can lag kernel releases.

On FreeBSD, ZFS is just there. bsdinstall can put your root filesystem on a mirrored ZFS pool during installation. Boot environments, which let you take a snapshot of your entire system before an upgrade and roll back if something breaks, are a standard workflow:

bectl create pre-upgrade
bectl activate pre-upgrade  # on next boot, if needed
pkg upgrade

The bectl command manages boot environments backed by ZFS clones. If the upgrade goes sideways, you reboot, pick the old environment from the bootloader, and you are back. This is available on Linux through tools like zsysctl on Ubuntu or zectl elsewhere, but it is optional tooling built on top of an optional filesystem module, not a standard part of the system.

Capsicum and the Capability Model

Capsicum is a capability-based security framework developed at the University of Cambridge and integrated into FreeBSD 9.0 in 2012. It extends the POSIX API with capability mode, where a process can voluntarily enter a restricted sandbox and can only operate on file descriptors it already holds, not open new ones by path.

The model is elegant: rather than listing what a process cannot do (as seccomp does on Linux), Capsicum describes what it can do. A process in capability mode has no ambient authority. Every resource it needs must be handed to it as an explicit capability before the sandbox is entered.

Base system tools like tcpdump and dhclient use Capsicum. OpenSSH on FreeBSD uses it for privilege separation. The sandboxing is available to any developer with cap_enter() and the caph_* helper functions, and it composes correctly with jails. A jailed, Capsicum-sandboxed process is constrained by both the jail boundary and the capability boundary simultaneously.

Linux has seccomp-bpf, which is powerful and widely used (Chrome, Firefox, systemd, and most container runtimes use it). But writing seccomp policies requires reasoning about which system calls to block, which is hard to get right and easy to get wrong. Capsicum’s approach of reducing ambient authority is architecturally cleaner, even if seccomp covers more platforms.

The Network Stack and bhyve

FreeBSD’s network stack has a long history of being used in production at scale. Netflix’s Open Connect CDN appliances run FreeBSD, and Netflix has been a significant contributor to the FreeBSD network stack. Their work on sendfile(2) optimizations, TLS offload, and HTTP/2 push are in the base system. When Netflix says they are serving around 40 percent of North American internet traffic at peak from FreeBSD appliances, that is a meaningful data point about the network stack’s production readiness.

netmap is a framework for high-speed packet I/O that bypasses the kernel’s normal network stack and gives userspace programs near-wire-rate access to network interfaces. It originated in FreeBSD and is part of the base system. The DPDK project, which Linux users often reach for in similar situations, is a separate userspace library that needs to be installed and configured separately.

bhyve is FreeBSD’s type-2 hypervisor, included in the base system since FreeBSD 10.0. It supports running Linux, Windows, and other BSDs as guests, uses virtio for paravirtualized devices, and has a reasonably clean architecture. It is not as feature-rich as KVM, but it does not need to be a separate kernel module because it is already part of the system.

Production Deployments and the Quiet Infrastructure

The PlayStation 4 and PlayStation 5 operating systems (Orbis OS and Prospero OS respectively) are derived from FreeBSD. The networking stack, filesystem layer, and process model come directly from FreeBSD. Sony has contributed code back upstream over the years.

WhatsApp ran FreeBSD for its server infrastructure during the years when it was scaling to hundreds of millions of users with a very small engineering team. The combination of a stable, well-understood base system and a clean network stack was part of why it worked.

These are not edge cases or curiosities. They are deployments at a scale most software will never reach, running on an operating system that most developers have never touched.

The Documentation

The FreeBSD Handbook is one of the best pieces of technical documentation for any operating system. It is comprehensive, accurate, and maintained as part of the same repository as the system it documents. The man pages are part of the base system. When something in the base system changes, the documentation is expected to change with it in the same review cycle.

This is a direct consequence of the single-tree philosophy. Documentation drift, where the docs describe a system that no longer exists because they live in a separate repository with separate maintainers, is structurally harder to sustain.

Why This Matters Now

The argument for FreeBSD is not that Linux is bad. Linux is the right answer for most things, and the ecosystem around it is enormous. The argument is that there is a real engineering cost to building systems from loosely coupled, independently versioned components, and that cost is often invisible until it surfaces at 2am when a kernel update breaks an out-of-tree module that breaks the container runtime that breaks production.

FreeBSD’s coherent system model does not eliminate operational complexity, but it localizes it. When you upgrade FreeBSD, you are upgrading a system. The compatibility guarantees are made by people who can see the entire picture at once.

For someone building infrastructure who can afford to operate a system that is not Linux, that coherence has real value. For someone building a Discord bot, the choice probably does not matter. But it is worth understanding why the FreeBSD model works the way it does, because the design decisions embedded in it, single-tree development, first-class capability security, integrated filesystem and containerization, are worth carrying into whatever system you do work on.

Was this interesting?