Daniel Lemire’s recent article on the cost of a function call captures the basic premise cleanly: calling a function is cheap, but not free, and in tight loops the overhead accumulates. The concrete numbers bear this out. On a modern x86-64 core, the CALL instruction pushes the return address onto the stack, decrements RSP, and jumps to the callee. The callee saves any registers it uses, does its work, restores those registers, and executes RET. Add up the prologue, epilogue, and argument marshaling, and a predictable direct call to a cached function costs roughly 5 to 10 cycles. At 3 GHz, that is about 2 to 3 nanoseconds per invocation.
For a function that does a few additions, that overhead is enormous relative to the work. But this is not actually the most expensive part.
The Optimization Fence
When a compiler encounters a call to an external symbol, a symbol whose definition is not visible in the current translation unit, it has to treat that call as a black box. It cannot know whether the function reads or writes memory. It cannot know whether it has side effects. It must assume the worst: any pointer that was live before the call might now point to different data. Any global variable might have changed. Any carefully maintained invariant about memory layout might be gone.
This conservative assumption is not a compiler bug. It is the only safe choice given incomplete information. But the practical consequence is severe: the compiler inserts an analysis barrier at every non-inlined call site. Alias analysis stops. Constant propagation stops. And, most consequentially, auto-vectorization stops.
Consider what happens to this loop:
__attribute__((noinline))
float scale(float x) { return x * 2.0f; }
void process(float* data, int n) {
for (int i = 0; i < n; i++) {
data[i] = scale(data[i]);
}
}Compile this with -O3 -mavx2 and inspect the output. The loop will be scalar. Each iteration calls scale, which the compiler cannot vectorize across because it cannot see inside it. The body of scale is trivially vectorizable, a multiply by a constant, but the boundary between translation units hides that fact.
Now change scale to an inline function or move its definition above process:
inline float scale(float x) { return x * 2.0f; }
void process(float* data, int n) {
for (int i = 0; i < n; i++) {
data[i] = scale(data[i]);
}
}The output changes completely. LLVM and GCC will both emit an AVX2 loop that processes 8 floats at a time using vmulps. The throughput on a Skylake core goes from roughly 1.0 ns per element to 0.08 to 0.12 ns per element. That is an 8 to 12x difference, and it comes almost entirely from the optimizer seeing across the function boundary, not from eliminating the CALL instruction.
Why the Vectorizer Stops at Function Calls
LLVM’s Loop Vectorizer and GCC’s tree-vectorizer share the same fundamental requirement: the loop body must contain no operations the vectorizer cannot reason about. A call to an opaque function breaks this in two ways.
First, the call is an alias analysis barrier. The vectorizer needs to prove that loads and stores in the loop do not overlap in ways that would make reordering them incorrect. An opaque call might modify any pointer in scope, which makes that proof impossible.
Second, the call is a side-effect barrier. Vectorization reorders and merges memory operations. An opaque call might have observable effects that depend on execution order, so the vectorizer cannot safely move operations across it.
There are escape hatches. GCC and Clang support __attribute__((const)) for functions that are pure computations, taking no memory inputs and producing no memory outputs, and __attribute__((pure)) for functions that read but do not write memory. A function annotated with const can be vectorized across even without inlining, because the compiler knows it only depends on its arguments.
__attribute__((const, noinline))
float scale(float x) { return x * 2.0f; }With this annotation, the vectorizer treats scale as a pure mathematical operation. The loop will vectorize, though the vectorized version still pays the actual call overhead. For most cases, inlining is the right answer. The const attribute is most useful for non-trivial functions where inlining would cause code bloat.
Forcing the Compiler’s Hand
When the compiler’s cost model decides a function is too large to inline, but you know from profiling that it belongs in a hot inner loop, __attribute__((always_inline)) overrides that decision:
__attribute__((always_inline))
inline float scale(float x) { return x * 2.0f; }MSVC uses __forceinline for the same purpose. These attributes tell the compiler to ignore its heuristics and inline unconditionally, even at higher optimization levels where the function might otherwise be kept out of the hot path.
The inverse also matters. Marking rarely-called functions with __attribute__((noinline, cold)) keeps them out of the instruction cache entirely. The .cold section of the binary collects these functions, improving the density of hot code in L1i and reducing cache pressure in the paths that matter.
The Icache Tradeoff
Aggressive inlining has a cost of its own. When a function is inlined at N call sites, the binary contains N copies of its body. If those call sites are hot and scattered across different loop iterations, the L1 instruction cache has to hold all of them simultaneously. The L1i on a Skylake core is 32 KB. Template-heavy C++ can exceed this budget quickly.
Agner Fog’s optimization guides document cases where inlining an inner loop across cache line boundaries slowed the result down by 1.4x compared to a non-inlined version with better cache reuse. The 32 KB budget is not theoretical.
This is where Profile-Guided Optimization becomes the principled solution rather than a nice-to-have. With PGO, the compiler instruments the binary, collects real execution traces, and then recompiles using that data to guide inlining decisions. Hot call sites get inlined. Cold call sites stay out of the icache. The compiler moves cold functions into separate sections, tightening the hot path.
Chromium, Firefox, and the Linux kernel all use PGO combined with post-link optimizers like BOLT or Propeller to manage this tradeoff in production. The reported gains range from 5 to 15 percent on end-to-end workloads, which is substantial for code that has already been through - O3.
Cross-Translation-Unit Inlining
All of the above assumes the function definition is visible at the call site. For functions defined in separate .cpp files, the compiler cannot inline across translation unit boundaries at compile time. Link-Time Optimization changes this by deferring optimization until the linker has access to all translation units simultaneously.
-flto on GCC and Clang enables full LTO. -flto=thin enables ThinLTO, which performs most of the same cross-module inlining with significantly faster link times by processing modules in parallel. For a library or application with hot paths split across multiple .cpp files, enabling ThinLTO can recover inlining opportunities that would otherwise be invisible to the compiler.
Other Languages Handle This Differently
Rust mirrors C++ closely because both compile through LLVM. The #[inline] attribute marks a function as a candidate for cross-crate inlining. Without it, Rust’s generics monomorphize correctly within a crate, but a function in a library crate cannot be inlined by its consumers unless the attribute is present. The standard library applies #[inline] pervasively on small functions for exactly this reason.
Java’s HotSpot JIT takes a different approach: speculative inlining at runtime. After a method is called roughly 10,000 times, the JIT compiles it and speculatively inlines virtual call targets based on observed types, inserting a type guard to handle deviations. The threshold for unconditional inlining is 35 bytecodes (-XX:MaxInlineSize=35). This makes Java JIT behavior inherently dynamic; it improves over time as the JIT collects more profile data.
Go’s inliner is more conservative by design, using a node budget to limit inlining depth. Go also does not auto-vectorize (intentionally, as of recent versions), so the vectorization angle does not apply. The tradeoff looks different when SIMD is not part of the picture.
What This Means in Practice
The diagnostic flags tell you exactly which loops are affected. On GCC:
-fopt-info-vec-missedThis reports every loop the vectorizer attempted and failed to vectorize, with a reason. “Function call may clobber memory” is the message for loops blocked by opaque calls. It is one of the most actionable pieces of feedback a compiler can give.
For hot numerical code, the path to maximum throughput runs through inlining. Not because CALL/RET is expensive in isolation, but because the compiler’s optimizer needs to see the full computation to transform it. The 5 to 10 cycle function call cost is a footnote. The vectorization multiplier it blocks, up to 16x on AVX-512, is the actual story.