The Design Boundary FreeBSD Hides in Plain Sight: /usr Versus /usr/local

The Hacker News discussion around Dragas’s “Why I Love FreeBSD” reliably surfaces the same topics: jails, ZFS, DTrace, the unified source tree. These are the right topics. But there is a quieter feature of FreeBSD’s design that enables all of them to behave predictably in production: the hard line between the base system and everything else. On FreeBSD, that line runs between /usr and /usr/local.

What the Division Actually Means

FreeBSD ships two distinct software management tools with non-overlapping scopes. freebsd-update(8) manages the base system: the kernel, libc, the standard utilities in /bin, /sbin, /usr/bin, and /usr/sbin, and the libraries under /lib and /usr/lib. It does not touch anything in /usr/local. The pkg(8) tool manages third-party packages, which live in /usr/local/bin, /usr/local/lib, /usr/local/etc, and related directories. It does not touch anything above that boundary.

This is not a convention that emerged organically. It is an architectural decision that both tools are built around. When FreeBSD ships a security advisory and you run freebsd-update fetch install, it patches the base system. Your installed packages stay untouched. When you run pkg upgrade, it upgrades third-party software without modifying any base system files. The two toolchains cannot interfere with each other because they own separate namespaces by design.

Compare this to a Debian or Red Hat system. There, a single package manager governs the entire software universe: the kernel, glibc, the standard utilities, and third-party applications all live in the same package graph. A apt upgrade might update glibc and curl and nginx in a single transaction. Shared libraries under /usr/lib can be owned by system packages and application packages depending on the library version in question. There is no structural boundary between OS and application. That is not a design failure; it reflects a different goal of providing a unified ecosystem across tens of thousands of packages. But it means that upgrading the OS and upgrading applications are intertwined in ways that FreeBSD deliberately avoids.

On FreeBSD, the OS is one managed unit. The applications are another. That sentence sounds obvious until you try to automate upgrades on production infrastructure and realize how much ambiguity comes from not knowing which invocation of apt upgrade will touch which system library.

The Ports Tree

Third-party software on FreeBSD originates from the ports collection, a tree of Makefiles covering over 34,000 packages. A port is not a binary; it is a build recipe. Running make install in a port directory fetches upstream source, applies FreeBSD-specific patches, builds against the system’s libc and toolchain, and installs into /usr/local. The result is software compiled for the exact FreeBSD version and hardware you are running.

The OPTIONS framework allows compile-time customization before building. A database port might offer options like SASL, OPENSSL, PAM, and LDAP. Setting them with make config records your choices for future rebuilds:

# Configure options interactively before building
cd /usr/ports/databases/postgresql16-server && make config install

# Show what a port will build with current options
make showconfig

For most workloads the binary pkg tool is the right choice. It resolves dependencies, installs from binary packages built by the FreeBSD project’s build cluster, and tracks installed software for upgrades and removal. The ports tree and pkg share the same package metadata format; packages built from ports are compatible with the binary package management infrastructure. You can mix source-built and binary packages on the same system without inconsistency, because they install to the same /usr/local tree under the same metadata scheme.

Security Tracking With pkg audit

One operationally useful feature of the pkg ecosystem is pkg audit. The FreeBSD project maintains a vulnerability database keyed to package names and version ranges. Running pkg audit -F fetches the current list and checks every installed package against it:

pkg audit -F

Output identifies vulnerable packages with CVE identifiers and the upstream advisory:

nginx-1.24.0 is vulnerable:
  nginx -- HTTP/2 rapid reset attack
  CVE: CVE-2023-44487
  WWW: https://nginx.org/en/CHANGELOG

1 problem(s) in 1 installed package(s) found.

This checks your exact installed versions against precise version boundaries in the vulnerability database, executed locally without an agent or external service. The /usr/local boundary matters here as well. pkg audit covers third-party software. The freebsd-update tool handles base system CVEs through the security.freebsd.org advisory mechanism. The two scanning concerns never overlap because the software sets do not overlap. On a Linux distribution with a unified package graph, distinguishing OS CVEs from application CVEs requires reading the advisory metadata carefully; the tooling does not enforce the distinction.

Upgrading Across Major Versions

The separation becomes most visible during a major version upgrade. Moving from FreeBSD 13.x to 14.0 involves freebsd-update -r 14.0-RELEASE upgrade, which replaces the base system binaries, kernel, and libraries. After rebooting, pkg upgrade rebuilds or reinstalls third-party packages linked against the new base libraries:

# Upgrade base system to new major version
freebsd-update -r 14.0-RELEASE upgrade
freebsd-update install
reboot

# Second freebsd-update install pass after reboot
freebsd-update install

# Now upgrade packages against the new base libraries
pkg upgrade

The base system upgrade and the package upgrade are sequential, independent operations. The base system comes up in a known state first. Packages are then rebuilt against it. On a Linux distribution, a major version upgrade typically moves the kernel, glibc, and application packages in one large operation, with the distribution maintaining compatibility across the transition. Both approaches can work reliably; the FreeBSD approach is easier to reason about because the phases are explicitly discrete and the failure modes are bounded.

If freebsd-update fails, you can activate a ZFS boot environment to revert the base system. If pkg upgrade breaks a specific application, you can reinstall a prior version: pkg install nginx-1.24.0. Each recovery path corresponds to the management domain where the failure occurred. There is no ambiguity about whether a broken system state was caused by a base system patch or a package upgrade, because they happened in separate, identifiable steps.

The Historical Root

The /usr versus /usr/local split was not invented by FreeBSD. It traces back to early BSD Unix systems, where /usr/local was the conventional home for locally installed software as opposed to the distribution itself. What FreeBSD did was elevate that convention to a hard architectural constraint and build its upgrade tooling around it. freebsd-update was explicitly designed to never write to /usr/local. pkg was explicitly designed to never write above it. The constraint is documented in hier(7), which lives in the base system source tree alongside the tools that enforce it.

Linux distributions inherited the same filesystem hierarchy conventions but did not build upgrade tools around the boundary in the same way. On a typical Debian system, /usr/local is available for manually installed software, but nothing stops a Debian package from installing to /usr/bin or /usr/lib directly, and nothing stops the administrator from placing system-wide configuration in /etc alongside distribution-managed configuration. The hierarchy exists but the tooling does not enforce it as a management boundary.

The Practical Consequence

For anyone running FreeBSD on infrastructure they need to understand, the /usr/local boundary changes how you think about the system. You can look at freebsd-update fetch output and know precisely which files it will modify: base system binaries and libraries, nothing else. You can run pkg upgrade with confidence that it will not disturb kernel modules or the libc your processes are linked against. You can reason about your system state as two separately managed layers rather than one unified package graph.

This is the part of FreeBSD that the original article is gesturing at when it describes the system as coherent and predictable, even if it does not call out /usr/local by name. The jails work because the jail subsystem is part of a coherent base system. ZFS boot environments work because freebsd-update is designed around the base system boundary. DTrace covers base system kernel functions because DTrace is part of the same tree. All of it depends on the OS knowing what it is and what it is not, and building tooling that respects that distinction.

The line at /usr/local is not the most visible feature of FreeBSD. It does not compete with jails or ZFS for attention. But it is the concrete expression of the design philosophy that makes the rest of the system trustworthy on production infrastructure that cannot afford to be surprised by a routine upgrade.