The source article on isocpp.org opens with a deceptively simple observation: function calls are not free, and in tight loops their overhead can dominate. That framing is correct, but it undersells what is actually happening. The call instruction itself is almost never the bottleneck. The real cost is what the compiler cannot do when it cannot see across a call boundary.
The Mechanical Cost Is Small
On a modern x86-64 processor, a direct function call involves pushing a return address, jumping to the callee, setting up a stack frame, saving any callee-preserved registers the function uses, doing work, restoring registers, and returning. With a correctly predicted return address in the CPU’s Return Stack Buffer, all of that overhead runs to roughly 5-10 cycles for a small function. Agner Fog’s optimization manuals document this in detail, including per-register save/restore costs and calling convention differences between System V (Linux/macOS) and the Windows x64 ABI, which mandates a 32-byte shadow space regardless of use.
That overhead is real, but small. On a CPU running at 4 GHz, 10 cycles is 2.5 nanoseconds. For a function called a thousand times at startup, this is irrelevant. For a function called inside a loop running a billion iterations, the math changes, but it is still not the main story.
The Real Cost: Optimization Barriers
When the compiler encounters a call to a function whose definition is not visible, it must make a conservative assumption: anything could happen. Any globally reachable memory may have been modified. All caller-saved registers are clobbered. Instructions cannot be reordered across the call boundary.
This is the optimization barrier, and it blocks three things that matter far more than saving 10 cycles per call.
First, constant propagation. If you call clamp(val, 0, 255) and the compiler can see the body of clamp, it knows lo and hi are constants at that call site. It can fold comparisons, eliminate branches, and reduce the whole operation to a pair of min/max instructions. Without visibility, none of that is possible.
Second, loop hoisting and unswitching. A loop that calls an external function cannot have its body transformed in ways that require knowing the function’s behavior. Branches on parameters that are constant at the call site cannot be hoisted or eliminated without seeing the body.
Third, and most significantly: auto-vectorization.
Inlining as the Gate to SIMD
Modern compilers can transform scalar loops into SIMD (Single Instruction, Multiple Data) code that processes 4, 8, or 16 elements per iteration using vector instructions. On a machine with AVX2, vmulps and vaddps can process eight floats per cycle. On AVX-512, sixteen. The theoretical throughput improvement over scalar code is 4-16x, and measured gains in numerical kernels often land in that range.
The auto-vectorizer requires exactly one thing: it must be able to analyze the loop body. If the body calls an opaque function, the vectorizer must assume arbitrary side effects and arbitrary memory accesses. The loop stays scalar.
// Compiler cannot vectorize this: process_element is opaque
for (int i = 0; i < N; i++) total += process_element(arr[i]);
// Compiler can vectorize this after inlining
static inline float process_element(float x) { return x * 2.0f; }
for (int i = 0; i < N; i++) total += process_element(arr[i]);In the second version, the compiler inlines process_element, sees the scalar multiply, and emits a vectorized loop over 8-element chunks. Daniel Lemire’s benchmarks have documented this pattern repeatedly: the overhead people attribute to the function call is usually the vectorization they are not getting.
This insight explains several library design decisions that otherwise look peculiar. Eigen uses expression templates to ensure that C = A + B * x compiles to a single fused vectorized loop rather than three separate passes over the data. SQLite ships its recommended build as a single amalgamated source file, explicitly to give the compiler full inlining visibility; the documentation attributes 5-10% performance improvements to this approach over separately compiled translation units. Google’s move away from reflection-based protobuf serialization to generated type-specific code enabled vectorization of inner byte-copying loops and produced roughly 2x serialization throughput in some workloads.
The inline Keyword No Longer Means What You Think
There is a widespread misconception worth addressing directly: the C++ inline keyword does not tell the compiler to inline a function. Compilers began ignoring it as a performance hint in the early C++03 era. GCC, Clang, and MSVC all have internal cost models, and they make inlining decisions based on those models regardless of the inline specifier. Bjarne Stroustrup documented this explicitly; cppreference states it clearly.
What inline means today is a statement about the One Definition Rule: an inline function can be defined in multiple translation units without a linker error, provided all definitions are identical. This is the mechanism that lets you define functions in header files. C++17 extended the same semantics to variables, enabling static inline int count = 0; as a class member defined in a header.
If you want to force inlining, you need __attribute__((always_inline)) on GCC and Clang, or __forceinline on MSVC. These override the compiler’s heuristics. They belong in specific contexts: SIMD wrapper libraries like xsimd where every wrapper must expose vector registers to the optimizer, cryptographic inner loops where ABI register saves would corrupt timing guarantees, and hot paths where profiling has confirmed the overhead is meaningful.
Link-Time Optimization and Cross-TU Visibility
The optimization barrier is most severe between translation units. A function in foo.cpp calling a function in bar.cpp gives the compiler no visibility during the compilation of foo.cpp; the callee is entirely opaque. Link-Time Optimization (LTO) resolves this by having the compiler emit its internal IR into object files rather than machine code. The linker then combines all IR into a single module, runs the full optimization pipeline including cross-module inlining, and emits machine code.
LLVM’s ThinLTO, developed at Google and available since LLVM 3.9, addresses the memory and build-time cost of full LTO at scale. Rather than loading all IR simultaneously, ThinLTO builds a function summary index per module, uses it to identify cross-module inlining candidates, imports only those function bodies, and optimizes each module in parallel. The Chrome team has reported 5-10% binary size reduction and 3-7% performance improvement from ThinLTO. The Clang documentation covers the build flag specifics.
For most projects, enabling ThinLTO at the link step is the highest-leverage change for recovering performance lost to cross-TU barriers. Both compilation and link must include the flag: clang -O2 -flto=thin. GCC uses -flto at both steps, emitting GIMPLE IR into object files.
Virtual Calls, Function Pointers, and the Spectre Factor
Virtual function calls add a layer on top of the basic call problem. Each virtual call loads the object’s vtable pointer, loads the function pointer from the table at the appropriate offset, and makes an indirect call. When the CPU’s indirect branch predictor has learned the target for a monomorphic call site, this costs 1-3 cycles over a direct call. When the site is polymorphic, arriving with different concrete types in unpredictable order, mispredictions cost 15-20 cycles each on current hardware.
Post-Spectre mitigations, deployed broadly starting in 2018, made this substantially worse. Retpoline, the software mitigation for indirect branch speculation attacks, replaces indirect branches with a call/ret sequence that prevents speculative execution of the target. On affected hardware, this adds 10-30 cycles per indirect call. Vtable dispatch and function pointer calls are affected equally. The compiler cannot inline through either, so the vectorization barrier remains regardless of whether the call site is monomorphic.
The C++ solution is templates. std::sort with a lambda comparator inlines the comparison at the call site; with a function pointer, it cannot. std::for_each with a lambda vectorizes; with a function pointer, it stays scalar. This is not an accident of standard library design. It reflects exactly this constraint, and it is why the standard algorithm headers are written using template parameters for callables rather than function pointers.
Devirtualization offers a partial escape for virtual calls. Marking a class final tells the compiler no further subclasses exist, enabling it to replace virtual dispatch with a direct call and then potentially inline. With LTO, whole-program devirtualization can identify virtual functions with only one override across the entire program and treat them as direct calls throughout. Profile-guided optimization adds a third path: if profiling shows a call site is monomorphic 99% of the time, the compiler emits a type check followed by a direct call for the common case.
Knowing When to Care
Most code is not in hot numerical loops. For the large majority of call sites, none of this matters, and the compiler’s default inlining heuristics are well-tuned for typical use. Annotating everything with always_inline increases code size and instruction cache pressure, which can make things slower by pushing hot code out of the L1 instruction cache.
The cases where it matters are specific: numerical kernels, serialization and deserialization hot paths, string processing on large inputs, and any loop where the bottleneck is element-level throughput rather than latency. Standard profilers show call overhead; they say nothing about the vectorization that a function boundary prevented. Understanding that the two are connected, and that inlining is the mechanism linking compiler visibility to SIMD throughput, is the useful mental model for diagnosing performance problems in tight loops.