The Two Kinds of Undefined Behavior in C, and Why C2Y Needs to Separate Them
Source: lobsters
The paper’s title, Ghosts and Demons: Undefined Behavior in C2Y, riffs on decades-old C folklore. In the early 1990s, on comp.std.c, someone pointed out that the C standard’s definition of undefined behavior imposes no requirements on the implementation whatsoever, which technically includes “demons flying out of your nose.” The phrase nasal demons became shorthand for the category: UB is not just “wrong answer” or “crash”; it is the compiler permitted to do anything at all.
N3861’s title refines that joke into a taxonomy. The question WG14 is implicitly asking for C2Y is which undefined behaviors are ghosts, haunting the standard as artifacts of 1989 portability concerns, and which are demons, actively enabling security vulnerabilities and allowing compilers to eliminate security-critical checks.
That distinction has never been drawn cleanly in the standard, and the ambiguity has served both sides of the debate for over thirty years.
Why undefined behavior has two different origins
The C standard’s three-tier behavioral taxonomy, undefined behavior, unspecified behavior, and implementation-defined behavior, was designed to accommodate hardware that no longer exists. In 1989, C ran on machines with ones’-complement integers, sign-magnitude representations, and word-addressed memory. Rather than mandate specific behavior for signed integer overflow, which differed across architectures, the standard declared it undefined. Each implementation could do what the hardware did naturally, with no emulation overhead.
That was a reasonable design for 1989. The problem is those machines are gone. The architectural diversity that justified many of C’s undefined behaviors has collapsed, but the undefined behaviors remain in the standard.
At the same time, a second category developed: behaviors undefined specifically to enable compiler optimizations. Signed integer overflow being UB allows the compiler to assume x + 1 > x is always true for any signed integer x, which enables loop induction variable analysis and simplifies range reasoning. Strict type aliasing lets the compiler assume that an int * and a float * never point to the same memory, so values can stay in registers without spurious reloads between calls. These UBs have genuine optimizer benefits.
Both categories sit in the standard under the same label with no distinction, and C2Y needs to separate them.
What C23 already handled
C23 (published as ISO/IEC 9899:2024) did the easier half of this work. It mandated two’s complement representation for all signed integer types, following C++20’s lead. The change means the representation of signed integers is defined everywhere. Signed integer overflow itself remains undefined, because that particular UB still provides real optimizer latitude even on two’s complement hardware.
C23 also added unreachable() to <stddef.h>, giving a standard name to what programs had been communicating implicitly through UB. Code that signals “this path cannot be taken” now has a proper mechanism:
#include <stddef.h>
int classify(int x) {
if (x > 0) return 1;
if (x < 0) return -1;
if (x == 0) return 0;
unreachable(); /* compiler may assume this path never executes */
}The semantic contract is identical to __builtin_unreachable(), but the intent is now legible to both the compiler and future maintainers.
nullptr as a type-safe null pointer constant addresses a related problem. The old (void*)0 / 0 ambiguity was a source of bugs where null pointer checks could be optimized away: the compiler, treating null dereference as UB and therefore unreachable, could eliminate the preceding check that guarded it.
The harder cases waiting for C2Y
What C23 deliberately left untouched are the UBs where optimizer benefit is contested and security costs are concrete.
Signed integer overflow is the obvious candidate. With two’s complement mandated, the mathematical justification for overflow being UB has narrowed. The optimizer argument survives, but the security argument against it is concrete. Consider:
/* Attempting to detect overflow before it happens */
int safe_add(int a, int b) {
if (a > 0 && b > INT_MAX - a)
return -1; /* would overflow */
return a + b;
}A compiler exploiting signed overflow as UB may eliminate the check entirely. The reasoning: if overflow is UB, overflow cannot occur, so the guard is always false, so remove it. The check becomes evidence against itself. This pattern has produced real CVEs in production C code, and sanitizers catch it during testing but not in release builds where the optimization fires.
Strict aliasing violations are another category where the UB enables meaningful optimizations but practice in the field diverges sharply from the standard. Most embedded and systems code violates strict aliasing somewhere. GCC’s -fno-strict-aliasing is nearly a default flag in major projects. When a UB is endemic in real-world code and requires a compiler flag to suppress its exploitation, the standard’s position has become untenable.
Reading an uninitialized variable is a third case. The current standard makes this UB in most contexts. The C++ committee’s proposal P2795 introduced a new category called “erroneous behavior”: a programming error that produces an unspecified value but does not permit the compiler to do arbitrary things. The program reads garbage, but the behavior is bounded. This is a principled middle ground between full UB and fully defined behavior, and C2Y may adopt something similar.
What other languages chose
Rust’s answer is the most comprehensive: the safe subset has no undefined behavior by design. The borrow checker enforces memory safety at compile time; bounds checks prevent out-of-bounds access at runtime; integer overflow panics in debug builds and wraps in release builds, both documented and explicit. The cost is that the language demands significantly more from the programmer at compile time.
Zig is more direct about integer arithmetic. The +% operator wraps, +| saturates, and plain + traps in safe builds and wraps in release-fast. The programmer selects the overflow mode per operation; there is no silent path.
Neither approach is available to WG14. C cannot adopt a borrow checker, and the compatibility requirements, installed base, and intended use cases make wholesale elimination of UB impossible. The committee can reduce and clarify, and the question is how much.
What the taxonomy is for
The value of N3861’s framing is that it separates the purpose of undefined behavior from its current effect. Some UBs are ghosts: they exist because of hardware that is gone, and removing them would cost no meaningful optimizer latitude. Some UBs are demons: exploited by compilers, source of security vulnerabilities, harmful when they escape from test environments into production.
The committee has historically been slow on UB reform because compiler writers and security researchers both participate in WG14 and pull in opposite directions. Compiler writers want to preserve optimizer latitude; security researchers want to stop undefined behavior from eliminating their checks.
C23 demonstrated that progress is possible when the committee identifies ghost UBs clearly enough: mandating two’s complement cost no optimizer latitude because every relevant compiler was already generating two’s complement arithmetic. C2Y has more latent decisions of that kind available, and a paper like N3861 is presumably trying to surface them.
The most useful outcome for C2Y would be a formal addition to the behavioral taxonomy: a new category for programming errors that produce bounded outcomes rather than arbitrary behavior, something analogous to C++‘s proposed erroneous behavior, alongside clear annotations distinguishing which undefined behaviors are retained for optimizer latitude and which are simply obsolete.
C has carried this ambiguity since 1989 because the costs were distributed across programs that nobody could fully audit. The costs are more visible now, the tools to measure them exist, and the committee is paying attention.