· 7 min read ·

Knowing What Your BPF Program Actually Needs Before It Hits the Kernel

Source: lobsters

Every BPF program you write carries hidden kernel version requirements. They live in the helpers you call, the map types you use, the program type you attach to. None of this is printed on the tin. You compile your program, it runs on your development machine with a 6.x kernel, and then you deploy it to a production fleet running 5.4 and something fails silently or panics the verifier. bpfvet is a tool that answers the question you should have been asking all along: what is the oldest kernel this compiled BPF object can run on?

The tool works on compiled ELF files, not source. Feed it a .bpf.o and it tells you the minimum kernel version (derived from helper usage, program types, and map types), which transport mechanism the program uses, whether BTF and CO-RE are present, and whether you’re using any superseded helpers. That last one matters more than it sounds.

Why BPF portability is structurally harder than userspace portability

In userspace, you have ldd. You can look at an ELF binary and know exactly which shared libraries it needs, and those libraries have stable versioned symbol tables. The OS loader will refuse to execute the binary if a dependency is missing. The contract is explicit.

BPF has no equivalent. The BPF verifier in the kernel will reject your program if you call a helper that doesn’t exist in that kernel, but it won’t tell you which version introduced the one you’re missing. The feature set is a function of the specific kernel you’re running, and there is no single authoritative table that maps every helper, map type, and program type to its introduction version that you can query from userspace at compile time.

The Linux BPF helpers documentation lists helpers and their descriptions but is light on version history. Tracking this down has historically meant reading kernel git history or relying on institutional knowledge. libbpf maintains an internal feature probing mechanism that probes the running kernel at load time, but that’s runtime detection, not static analysis. bpftool has bpftool feature for the same reason: it tells you what the current host supports. Neither tells you what a specific compiled program requires.

What lives inside a .bpf.o

A compiled BPF object is a standard ELF file. The sections encode most of what you need to know. Program sections use names like kprobe/vfs_read or tracepoint/syscalls/sys_enter_openat that tell the loader the program type and attach point. The .maps section (or maps for older formats) describes map definitions. If the program uses CO-RE, there will be a .BTF section containing BPF Type Format data and a .BTF.ext section with line info and CO-RE relocation records.

BTF itself was introduced in kernel 4.18 but CO-RE, which uses BTF to perform struct field offset relocations at load time, became practically usable with kernel 5.2 when the kernel started exporting its own BTF via /sys/kernel/btf/vmlinux. Programs using CO-RE technically compile against a vmlinux.h generated from that BTF, allowing them to adapt to struct layout differences across kernel versions. This is the mechanism that lets a single compiled BPF binary handle task_struct layout changes between kernels without recompilation.

But CO-RE only helps with struct layouts. It cannot conjure a helper that doesn’t exist in the target kernel.

The helper version problem in practice

Consider bpf_probe_read. This helper has existed since kernel 4.1 and is used to safely read memory from kernel context in tracing programs. For years it was the standard way to dereference kernel pointers. In kernel 5.5, the kernel introduced two replacements: bpf_probe_read_kernel and bpf_probe_read_user. The distinction matters because the original function was ambiguous about which address space it was reading from. On architectures with separate user/kernel address spaces (like x86 with SMAP), calling the wrong variant is a correctness issue, not just a style issue.

If you write a new tracing program today and use bpf_probe_read_kernel, you have implicitly required kernel 5.5 or later. That excludes RHEL 8 (ships with a 4.18 kernel), Ubuntu 18.04 LTS (4.15 base), and various hardened enterprise distributions that don’t backport BPF features. bpfvet detects this usage and flags it, which is the kind of information that should live in your CI pipeline, not in a production postmortem.

The pattern repeats across the helper table. bpf_skb_load_bytes_relative arrived in 4.18. bpf_sk_fullsock in 5.1. bpf_tcp_sock in 5.1. bpf_get_current_task_btf in 5.10. bpf_ktime_get_boot_ns in 5.7. Each call you make narrows the set of kernels your program can run on, and without tooling you accumulate these constraints invisibly.

Transport mechanism matters too

BPF_MAP_TYPE_RINGBUF was introduced in kernel 5.8. Before it, the standard way to stream events from BPF programs to userspace was BPF_MAP_TYPE_PERF_EVENT_ARRAY with bpf_perf_event_output, which dates back to 4.4. Ring buffer is the right choice for new programs: it has lower overhead, supports variable-length records without the per-CPU memory waste of perf event arrays, and provides atomic reserve-commit semantics that prevent torn reads. But using it immediately sets your floor at 5.8.

This tradeoff is exactly what bpfvet surfaces. If you query a .bpf.o and it reports RingBuf as the transport, you know the program requires 5.8+. If it reports PerfEventArray, you have more headroom, down to 4.4. For a tool you’re distributing broadly, like a security monitoring agent or a network observability component that runs on customer infrastructure, that three-and-a-half-year gap in kernel releases represents a significant portion of the fleet you might encounter.

Static analysis versus runtime probing

The existing art in this space is mostly runtime. libbpf’s feature probing creates small BPF programs internally and tries to load them, using the verifier’s rejection as a signal that a feature is unsupported. bpftool wraps this for interactive use. This approach is comprehensive and accurate but requires access to a kernel. You cannot run it in a CI environment that doesn’t have BPF available, and you cannot use it to analyze programs before deployment.

bpfvet takes the other side of this tradeoff. Because it works purely from ELF content, it runs anywhere: CI pipelines, code review tooling, package metadata generation. It can be part of a build system step that gates distribution on portability criteria. The tradeoff is that a database of helper-to-version mappings must be maintained. When new helpers land in the kernel, the tool’s understanding needs to update. But that’s a tractable problem.

The approach is also not unprecedented. The Linux kernel itself has long maintained BPF_CALL_x macros with associated feature guards, and the verifier internally tracks which helpers are available for each program type. Tools like BCC have shipped with kernel version checks at the Python layer for years, but these are hand-coded guards scattered through individual tools rather than derived from the compiled artifact.

The ELF as a bill of materials

The useful mental model for bpfvet is that a .bpf.o is a bill of materials, and bpfvet reads it. The map section lists map types and their configurations. The program sections identify program types. The BTF section, if present, provides the function call graph and type information that allows resolving which specific helpers are called. Without BTF the analysis has to work from the BPF bytecode directly, inspecting BPF_CALL instructions and looking up the helper ID. With BTF the analysis can be more precise.

This is close to how the kernel verifier itself works when loading a program. The verifier builds a call graph, tracks register types through every instruction, and validates helper calls against the program type. bpfvet does a subset of this offline and cross-references helper IDs against a version table rather than against the currently running kernel’s supported set.

Where this fits in a BPF development workflow

For a tool like this to be useful, it needs to be integrated early. The obvious integration point is CI. A compiled .bpf.o artifact is small and can be analyzed quickly. A CI step that runs bpfvet and prints the minimum kernel version, any superseded helpers, and CO-RE status gives you immediate feedback. If your stated support target is 5.4 and bpfvet reports your program requires 5.8, that’s a merge-blocking finding, not a production incident.

The superseded helper check has value even if you don’t care about old kernels. bpf_probe_read still exists in modern kernels for compatibility, but using it in new code is a correctness smell. On x86_64 with CONFIG_X86_SMAP, the kernel distinguishes user and kernel addresses at the hardware level; bpf_probe_read handles both but provides no compile-time or verifier-level enforcement that you’re using the right one. Tools that flag this help enforce the cleaner API without requiring reviewers to remember the deprecation.

The BTF and CO-RE presence check is similarly useful. A program without BTF is opaque to bpftool’s introspection, harder to debug at runtime, and unable to use CO-RE relocations. Knowing a compiled artifact lacks BTF can prompt you to check compiler flags (-g is required for DWARF-to-BTF conversion via llvm-strip and pahole) before you’ve already deployed.

Closing thoughts

BPF has grown into a remarkably capable subsystem, but the tooling around it still has gaps. The kernel verifier is a rigorous gatekeeper at load time, but it gates against the running kernel, not against your stated target. libbpf and bpftool cover runtime detection. What bpfvet covers is the space between compilation and deployment: the static, portable question of what a compiled artifact actually requires.

This is the kind of tool that becomes invisible when it works. You run it in CI, it reports nothing concerning, and you ship. The value shows up when it catches a 5.8 ringbuf dependency in a program you planned to deploy on a fleet with mixed kernel versions, before you find out the hard way. For anyone maintaining BPF-based tooling that runs on infrastructure you don’t fully control, that’s a meaningful addition to the build pipeline.

Was this interesting?