Part two of a BSD TCP troubleshooting series on omaera.org does something relatively rare: it follows an investigation all the way to an actual fix. Not a workaround, not a “further investigation is needed” conclusion, but a root cause and a resolution.
That matters because most network debugging write-ups stop somewhere in the middle. You get the symptom, maybe some tcpdump output, and then a vague gesture toward socket buffers or congestion control. The omaera.org post traces the problem from observed throughput degradation down into the NetBSD kernel itself.
What NetBSD Does Differently
NetBSD has a reputation for conservatism in its defaults, and nowhere is that more visible than in the TCP stack. Auto-tuning for socket buffers is not on by default the way it is in Linux. The relevant sysctls, net.inet.tcp.recvbuf_auto and net.inet.tcp.sendbuf_auto, exist, but the out-of-box configuration can leave you with buffer sizes that made sense for hardware from fifteen years ago.
The consequence is that on modern gigabit or multi-gigabit links, especially with any meaningful round-trip latency, your TCP window fills before the pipe does. Throughput plateaus, and nothing in the obvious monitoring output tells you why. The link looks healthy. The interface counters look fine. The problem is invisible until you go looking for it specifically.
The Debugging Path
What the series documents is the kind of methodical elimination that kernel debugging actually requires:
- Establish a baseline with
iperf3between two machines - Confirm the hardware is not the bottleneck by checking NIC and driver offload settings via
ifconfig - Work through the sysctl tree, adjusting one variable at a time and measuring again
- When userspace tuning is not enough, trace the issue into the kernel source
NetBSD is a reasonable environment for that last step. The source base is compact compared to Linux, the TCP implementation is well-organized, and the sysctl net.inet.tcp namespace gives you a relatively direct view into what the stack is doing. You can read from an observed behavior back to the relevant code path without losing days to it.
TSO (TCP Segmentation Offload) is another place the series looks. Driver support in NetBSD varies across NICs, and the interaction between TSO and features like VLAN tagging or IPsec is a known source of subtle regressions. Toggling it through ifconfig and measuring the delta is one of the faster ways to rule it in or out as a factor.
Why This Kind of Post Is Worth Reading
There is a lot of general BSD networking documentation, and there are changelog entries that tersely note fixes. What is scarcer is a narrative that shows the dead ends, the hypotheses that did not pan out, and the reasoning that eventually landed on the right answer.
For anyone running NetBSD in a context where throughput matters, whether that is a home router, a build server, or an embedded system, the specific changes documented here are directly applicable. The broader methodology applies to FreeBSD and OpenBSD too, with adjustments for how their respective sysctl trees differ.
If you have not looked at your BSD TCP buffer settings recently, this series is a good reason to.