The Real Cost Isn’t the Call Instruction
The usual framing is cycles: a function call costs roughly 4-8 cycles on a modern x86-64 processor, covering the call instruction, the stack frame setup, and the ret. Daniel Lemire’s piece on isocpp.org states it plainly: cheap, but not free, and consequential in tight loops. That framing is correct, but it understates the problem considerably.
The cycles from call and ret are the visible portion. The invisible portion is what the compiler gives up the moment code lives in a separate function: auto-vectorization, constant propagation, and dead code elimination. For code that processes arrays or operates in tight numerical loops, failing to vectorize can mean leaving a 4x to 8x speedup on the table. The call itself is rarely the bottleneck; the blocked optimization is.
What Actually Happens at the CPU Level
On x86-64 with the System V AMD64 ABI (Linux, macOS), the first six integer arguments arrive in registers (rdi, rsi, rdx, rcx, r8, r9), so passing arguments to a small function costs nothing in additional memory loads. The call instruction pushes a return address and jumps; the callee’s prologue adjusts the stack pointer and saves any callee-preserved registers it needs; ret reverses this. At -O1 and above, compilers enable -fomit-frame-pointer, eliminating the frame pointer save and restore entirely.
The net mechanical cost for a direct, warm-cache call on a Skylake-class processor runs about 3-6 cycles for the call/ret pair, plus register pushes. A function doing 20 cycles of arithmetic pays roughly 30% overhead. A function doing 1-2 cycles of arithmetic pays several times its own weight.
The instruction cache matters more in practice than the call instruction itself. If the callee is not already in L1 I-cache, the penalty is 12-40 cycles depending on whether L2 or L3 absorbs the miss. For functions called infrequently or from many call sites that don’t all stay hot simultaneously, this cold-cache cost dominates everything else. Inlining eliminates the miss entirely by folding the callee into the caller’s already-hot instruction stream.
The Vectorization Barrier
Auto-vectorization, the transformation that processes multiple array elements per clock cycle using SIMD instructions, requires the optimizer to see the entire loop body. A function call boundary prevents that analysis completely.
void scale(float* dst, const float* src, int n) {
for (int i = 0; i < n; i++) {
dst[i] = multiply(src[i]); // non-inlined call
}
}The compiler cannot vectorize this loop. It cannot determine whether multiply reads or modifies dst. It cannot batch eight calls to multiply into a single AVX vmulps instruction because the calling convention requires scalar arguments and scalar return values. The ABI offers no mechanism to invoke an arbitrary function with a SIMD register’s worth of inputs.
Inline the function, and the optimizer sees:
// After inlining multiply(x) = x * 2.0f:
for (int i = 0; i < n; i++) {
dst[i] = src[i] * 2.0f;
}With -O3 -mavx2, GCC or Clang generates something along these lines:
vmovups ymm0, [rsi + rax] ; load 8 floats
vmulps ymm0, ymm0, ymm1 ; multiply all 8 by 2.0
vmovups [rdi + rax], ymm0 ; store 8 floats
add rax, 32
jl loopEight elements per iteration instead of one, with zero call overhead. On float arrays, the throughput gain from this transformation alone runs 4x to 8x depending on loop structure and the surrounding instruction mix.
Vectorization is not the only optimization blocked by function boundaries. Constant propagation cannot fold computations when arguments are compile-time constants unless the callee is inlined. Dead code elimination cannot remove branches gated on those constants. Loop unswitching cannot hoist invariant conditions if they live inside a called function. Inlining is the precondition for the transformations that make tight loops fast, not a minor nicety on top of them.
What the inline Keyword Does
There is a persistent misconception, present in C++ written by experienced engineers, that the inline keyword instructs the compiler to inline a function. It does not. The C++ standard defines inline as relaxing the One Definition Rule: it allows a function to be defined identically in multiple translation units without triggering a duplicate-symbol linker error. The specification explicitly states that the implementation “is not required to perform this inline substitution at the point of call.”
Modern compilers have treated inline as a hint with no guaranteed effect for roughly two decades. GCC’s auto-inliner at -O2 uses an internal instruction-count threshold of about 40 GIMPLE instructions for automatic inlining. Clang/LLVM uses a cost model with a threshold around 225 abstract cost units. A function marked inline that exceeds these thresholds will not be inlined unless a call site is detected as especially hot through profile-guided optimization data. A function without inline that falls under the threshold will be inlined anyway.
The word inline in a header file serves a genuine purpose: ODR compliance for functions defined in headers included by many source files. It is not a performance tool, and treating it as one leads to code that looks like it is requesting optimization while producing none.
Two related facts worth knowing: all functions defined inside a class body are implicitly inline, and all constexpr functions are implicitly inline since C++11. Neither of these properties implies inlining at call sites.
Forcing the Compiler’s Hand
For functions that must be inlined to unlock SIMD or other critical optimizations, GCC and Clang provide __attribute__((always_inline)):
__attribute__((always_inline)) inline float multiply(float x) {
return x * 2.0f;
}MSVC provides __forceinline. Both bypass the cost model entirely and force inlining at every call site, with the physical exception that recursive inlining is impossible. GCC requires the inline keyword alongside __attribute__((always_inline)) for the attribute to take full effect across translation units. Clang does not.
The most cited industrial example of systematic forced inlining is simdjson, the high-performance JSON parser from Lemire and colleagues. The library defines:
#define really_inline __attribute__((always_inline)) inlineThis macro appears on essentially every function in the hot parsing path. The intent is to present the optimizer with a single large inlined function body for the entire parsing loop, enabling SIMD vectorization throughout. simdjson achieves 2.5 to 3.5 GB/s JSON parsing throughput; parsers built on conventional function decomposition typically measure around 0.5 GB/s. The difference is not algorithmic cleverness; it is the direct consequence of compiler visibility over the hot path.
To diagnose where the compiler declined to inline a call, Clang provides -Rpass-missed=inline and GCC provides -fopt-info-inline-missed. Both emit per-call-site diagnostics explaining why inlining was rejected, which is considerably more useful than guessing at cost-model thresholds.
Cross-Translation-Unit Inlining with LTO
Forced inlining works within a single translation unit. Functions defined in separate .cpp files are invisible to each other during compilation, so inlining across that boundary requires link-time optimization, which defers code generation to the link step when all translation units are visible together.
Clang’s ThinLTO (-flto=thin) compiles each source file to LLVM IR and emits a lightweight summary of each module. At link time, the linker identifies profitable cross-module inlining opportunities, imports only the needed IR from remote modules, and recompiles each module independently. ThinLTO is designed for parallel execution; large projects that find full LTO impractical on compile time typically find ThinLTO acceptable.
# ThinLTO with Clang
clang++ -flto=thin -O2 -o program *.cpp
# With incremental cache for fast rebuilds
clang++ -flto=thin -Wl,--thinlto-cache-dir=/tmp/thinlto-cache -O2 -o program *.cpp
# GCC full LTO
g++ -flto=auto -O2 -o program *.cppBeyond inlining, LTO enables interprocedural constant propagation, dead code elimination across module boundaries, and virtual call devirtualization using whole-program type analysis. Combined with profile-guided optimization, it is the most complete performance tool available without changing source code structure.
When Not to Inline
Aggressive inlining increases binary size, and a larger binary evicts other hot code from the L1 instruction cache. A function inlined at twenty call sites appears twenty times in the instruction stream. For a function of nontrivial size, this can harm the cache locality that inlining was meant to improve.
Error-handling and diagnostic paths are the natural candidates for the opposite treatment. Marking them noinline and cold keeps them out of the hot code region:
[[gnu::noinline, gnu::cold]] void throw_range_error(size_t idx, size_t size);The cold attribute places the function in a cold section of the binary (.text.cold on ELF targets) and biases branch prediction to treat calls to it as unlikely. The hot path’s instruction cache footprint shrinks; the cold path remains callable but stays physically separated. The combination is particularly effective for error paths that must exist for correctness but are almost never taken at runtime.
There is also a profiling argument for noinline: inlining collapses stack frames, making profiler output and backtraces harder to read. A function inlined at ten call sites appears as part of its callers in a profile, obscuring where time is actually being spent. For any function you might want to observe in a profiler, noinline preserves the frame boundary.
The Right Mental Model
A function call in a tight loop carries two costs: the visible one (call/ret cycles, register saves, potential I-cache miss) and the invisible one (the optimizer is now working blind on whatever is inside that function). For any loop where performance matters, the invisible cost usually dominates the visible one. The gap between scalar and AVX2 execution is 8x; the gap between a warm cache hit and an I-cache miss is 12-40 cycles; the raw call/ret overhead is 4-8 cycles.
The practical approach follows from this. Keep hot inner functions small enough to fall within the compiler’s default inlining thresholds. Use __attribute__((always_inline)) for functions that must be inlined to enable downstream SIMD or constant-folding opportunities. Use ThinLTO for any project where performance-critical code spans multiple files. Mark cold error paths with noinline and cold to protect the instruction cache for the paths that run constantly.
The source of the overhead is rarely the call instruction. It is the optimization surface the compiler loses sight of.