· 5 min read ·

What FreeBSD Gets Right About Production Observability

Source: hackernews

A Hacker News thread around a post titled “Why I love FreeBSD” hit 500 points this week. The comments followed a familiar pattern: jails, ZFS, the unified source tree, the BSD license. These are the right things to talk about. But there is one part of the FreeBSD story that gets less attention in these discussions, and it matters to anyone trying to understand what a production system is actually doing under pressure: DTrace.

What DTrace Is

DTrace was developed at Sun Microsystems by Bryan Cantrill, Mike Shapiro, and Adam Leventhal, and shipped in Solaris 10 in 2005. It was open-sourced under the CDDL license and integrated into FreeBSD in version 7.1, released in January 2009. It is now part of the FreeBSD base system, available on every installed machine without additional packages.

The model is a set of probes organized by provider, module, function, and name into a four-part tuple: provider:module:function:name. Probes fire when execution crosses a specific point in the kernel or in userland. You write D scripts to select which probes to enable, what data to collect when they fire, and how to aggregate that data.

A minimal example: count system calls by process name.

syscall:::entry
{
    @calls[execname] = count();
}

Run with dtrace -s script.d, wait ten seconds, terminate with Ctrl-C. DTrace prints an aggregation sorted by call count. No reboot, no kernel rebuild, no package installation. The probes are there because the base system includes them.

The Probe Model

The provider is the source of probes. FreeBSD’s base DTrace providers cover a broad surface:

  • syscall fires on entry and return of every system call
  • io fires when I/O requests are submitted and complete
  • proc fires on process creation, execution, and exit
  • sched fires on context switches, wakeups, and sleeps
  • tcp, udp, and ip fire at key points in the network stack
  • fbt (Function Boundary Tracing) fires on entry and return of nearly any kernel function
  • profile fires on a timer, useful for sampling-based profiling

FBT is the most powerful of these. It instruments kernel function entry and return dynamically, without modifying the kernel binary or rebooting. You can trace specific kernel functions on a live system:

fbt::tcp_output:entry
{
    printf("tcp_output from pid %d (%s)\n", pid, execname);
}

Disabled probes have zero overhead. They cost nothing when not in use, which means shipping DTrace-ready instrumentation in production carries no continuous performance penalty.

USDT: Tracing Application Logic

USDT (Userland Statically Defined Tracing) allows applications to embed probes that DTrace can enable. A server process defines probes like request-start, cache-miss, or connection-accepted. When DTrace is not running, these probes are no-ops, replaced at compile time with NOP instructions. When DTrace attaches and enables them, they fire with application-defined arguments.

PostgreSQL ships with USDT probes on FreeBSD. You can trace transaction commits at the application level without modifying PostgreSQL source or adding any external instrumentation:

postgresql$target:::transaction-commit
{
    @commits[pid] = count();
}

Node.js, Python, Ruby, and Java all support USDT probes. The same D script structure applies across runtimes because the probe interface sits at the OS level, not the language runtime level. You instrument the application from the outside, without changing it.

Why the License Creates the Integration

DTrace is licensed under the CDDL, the same license as ZFS. CDDL is incompatible with GPL v2, so DTrace cannot be merged into the Linux kernel. Linux’s answer has been eBPF and bpftrace, which are genuinely capable. eBPF programs run in a kernel-side virtual machine, verified for safety, and hook into tracepoints, kprobes, and uprobes. bpftrace provides a DTrace-like scripting layer on top of eBPF.

The difference is depth of integration. DTrace on FreeBSD instruments the network stack, the scheduler, the I/O subsystem, and userland application probes through a single coherent model that was designed alongside the kernel. eBPF on Linux is a powerful framework layered on top of infrastructure that was not originally designed for dynamic tracing. The eBPF verifier has historically been a source of security vulnerabilities because it is a complex subsystem added later rather than designed from the start as a core primitive. CVE-2021-3490, CVE-2021-31440, and several others in recent years all trace back to verifier edge cases.

Brendan Gregg, who spent years at Netflix developing DTrace-based performance tools before moving to eBPF work at Meta, has written about both systems in depth. His Systems Performance book covers both. His view is that eBPF has largely caught up for many use cases, but DTrace’s user experience is cleaner because the D language was purpose-built for exactly this task rather than assembled from general-purpose VM machinery.

Practical Usage

Profiling where a process spends CPU time:

profile:::profile-1000hz
/pid == $target/
{
    @[ustack()] = count();
}

This samples the process’s user stack at 1000 Hz and produces a sorted count of stack traces, revealing where CPU time goes without instrumenting every function individually. Run with dtrace -s cpudist.d -p <pid>.

Tracing file opens for a specific process:

syscall::openat:entry
/pid == $target/
{
    printf("open: %s\n", copyinstr(arg1));
}

Measuring disk I/O latency by file path:

io:::start
{
    ts[arg0] = timestamp;
}

io:::done
/ts[arg0]/
{
    @lat[args[2]->fi_pathname] = quantize(timestamp - ts[arg0]);
    ts[arg0] = 0;
}

The quantize aggregation produces a power-of-two histogram in the output, showing not just average latency but the distribution. You can tell the difference between a disk that is uniformly slow and one with occasional high-latency outliers, which are different problems with different solutions.

The Coherence Point

DTrace’s presence in FreeBSD’s base system is not incidental. The base system is a controlled product maintained under one source tree, the CDDL license is compatible with FreeBSD’s overall licensing model, and the project chose to maintain DTrace alongside the kernel rather than distribute it separately. When the network stack changes in a new release, the tcp and udp probe providers are updated in the same commit. When a new syscall is added, it appears in the syscall provider without additional work from a separate team.

Netflix’s use of FreeBSD was partly driven by this. Brendan Gregg’s public DTrace-based tooling, developed while at Netflix, depended on the ability to dynamically trace the network stack, identify TCP retransmit patterns affecting specific HTTP requests, and measure I/O latency at the kernel level without modifying application code or rebooting servers. His DTrace tools collection documents what this looks like in practice: scripts that answer specific operational questions about production systems, run on demand, with no persistent overhead.

The original article that prompted the HN discussion focuses on FreeBSD’s design philosophy: a coherent, unified system where components are developed together and tested against each other. DTrace is a clean expression of that philosophy. It works well on FreeBSD because it was designed as part of the operating system, maintained by the same people who maintain the kernel it traces, and documented in the same handbook as the system calls it intercepts. When a post about loving FreeBSD leads with jails and ZFS but says nothing about DTrace, it is leaving out the tool that, for many production engineers, is the clearest argument for being there at all.

Was this interesting?