· 7 min read ·

What It Takes to Make a Docker Image Bit-for-Bit Reproducible

Source: lobsters

Arch Linux now produces a Docker base image that is bit-for-bit reproducible. Build it twice from the same inputs on two different machines, on different days, and you get the same SHA256 digest. That sounds like a modest infrastructure improvement until you start listing the things that have to go right for it to be true, and then it looks considerably more interesting.

What a Docker Image Actually Is

Before getting into the reproducibility mechanics, it helps to be precise about what a Docker image consists of. Under the OCI Image Specification, an image is a JSON manifest referencing a sequence of layers, each of which is a gzipped tar archive containing filesystem deltas. The manifest references layers by their SHA256 digest. The manifest itself has a digest, and that digest is what docker pull and container registries treat as the canonical identity of an image.

This structure means that for two builds to produce the same image digest, the following must all match: the manifest JSON, the layer count and order, and the exact byte sequence of every layer archive. Because layers are tar files, this means that the file ordering inside each archive, the file metadata (permissions, modification timestamps, ownership), and the compressed byte stream must all be identical.

Gzip compression is not deterministic by default. Even with the same input bytes, different runs can produce different compressed output depending on the compressor version, the compression level, and certain timing-dependent state. The OCI spec leaves compression as an implementation detail, so image builders have to be deliberate about this.

The Sources of Non-Determinism

Container image builds accumulate non-determinism from several independent sources, and each one has to be addressed separately.

Timestamps. Files in a tar archive carry modification timestamps. If you write a file during a build, it gets the current wall clock time. Two builds run at different times produce different timestamps, different archive bytes, and therefore different layer digests. The standard fix is SOURCE_DATE_EPOCH, an environment variable defined by the Reproducible Builds project that instructs build tools to use a fixed timestamp instead of the current time. Tools in the build chain need to respect it, and for a Linux bootstrap tarball that includes pacman, the package manager and its install hooks, and the tools that write the filesystem, all of them need to behave correctly when SOURCE_DATE_EPOCH is set.

File ordering. Tar archives preserve insertion order, not filesystem order. If the code that builds the archive iterates over a directory using readdir(3), the order depends on the filesystem, the kernel, and sometimes on hash table internals that vary between runs. Sorting the entries before archiving them is the standard fix, but every tool in the chain that writes archives needs to do this.

Package metadata and database state. Pacman writes a local package database under /var/lib/pacman/. The database entries contain installation timestamps. When you bootstrap an Arch Linux root with pacstrap or a bootstrap script, each installed package records the time it was installed. Two bootstraps run at different times produce different database contents, different files, different layer bytes.

Locale and environment variables. Some build tools produce locale-dependent output. Others embed the hostname of the build machine, or the username, or a build identifier. These need to be stripped or normalized.

Compression. As mentioned, gzip is not reproducible by default. The fix is to use a deterministic compressor, often gzip with specific flags, or to use zstd in deterministic mode, or to configure the Docker image builder to use reproducible compression.

What the Arch Linux Bootstrap Does

The Arch Linux Docker image is not built with a Dockerfile from scratch. It is built from a bootstrap tarball, which is itself produced by mkarchroot from the arch-install-scripts package. The bootstrap process installs a minimal Arch Linux system into a temporary directory, archives it, and imports the archive as the base layer.

This is a good foundation for reproducibility because the build path is relatively short and controlled. There is no long Dockerfile with many RUN layers accumulating non-determinism. There is one root filesystem archive, created by a known set of tools, importing a known set of packages at known versions.

The work that went into making this bit-for-bit reproducible involved auditing and fixing each non-determinism source along that path. Pacman’s installation timestamps had to be normalized. The archive creation code had to sort entries and clamp timestamps. The compression had to be made deterministic. The Docker layer construction had to produce identical compressed archives across runs.

The Reproducible Builds project has been running this kind of analysis on Debian packages since around 2013, building the tooling and methodology that other distributions now draw on. Their diffoscope tool is the standard way to compare two build artifacts and identify exactly where they diverge. A diff showing that two builds of a tar archive differ only in the modification timestamps on five files tells you exactly what to fix. Without tooling like this, the debugging process would be guesswork.

Why This Matters for Security

Reproducible builds are often framed as a quality property, which they are, but the security implications are more concrete.

The core problem that reproducibility addresses is the gap between source code and binary artifact. You can read and audit source code. You cannot directly audit a compiled binary or a container image layer. The claim that a given image was produced from a given source commit is ordinarily just a claim, backed by your trust in the build infrastructure and the humans who operate it. If a build server is compromised, a malicious actor can modify the build output without touching the source code. The compromise would be invisible to anyone who only audits the source.

With a reproducible build, the claim becomes verifiable. Anyone can independently reproduce the build and check that their output matches the published digest. If two independent builds from the same inputs always produce the same output, then a build output that does not match the expected digest is evidence that something went wrong, either in the build inputs or in the build infrastructure. The XZ Utils backdoor discovered in early 2024, inserted into the release tarball rather than the source repository, is an example of exactly the kind of attack that independent reproducibility verification can catch.

This is the argument the Reproducible Builds project has been making for years, and it is why reproducibility matters more for base images than for application images. The Arch Linux base image is a trust anchor. Every container built on top of it inherits whatever properties it has. A reproducible base image means that the anchor can be independently verified.

The Comparison With Other Distributions

Arch Linux is not the first distribution to achieve a reproducible container base image, but the approach differs from other distributions in instructive ways.

NixOS takes the most aggressive stance. Because the Nix package manager builds everything from source in hermetically isolated environments, the entire system is designed around reproducibility. A NixOS container image’s SHA256 can be computed from the store paths of its inputs without running the build at all. The tradeoff is substantial complexity: NixOS is not a general-purpose distribution in the way Arch is, and its learning curve is steep.

Debian’s reproducible builds effort operates at the individual package level. The project tracks which packages build reproducibly and which do not, and the numbers have improved significantly over the years. Getting from reproducible packages to a reproducible base image requires additional work at the image assembly layer, which the Debian Docker image maintainers have worked on separately.

Alpine Linux, the most popular base image for production containers by pull count, has its own approach rooted in the simplicity of musl libc and its package format. Alpine images are small because the base system is small, which reduces the surface area for non-determinism.

Arch Linux’s approach sits in a different position in this space. It is a rolling release distribution with bleeding-edge packages. The bootstrap tooling is straightforward and not hermetically isolated in the Nix sense. Achieving reproducibility here required making the specific tools in the bootstrap path behave deterministically, rather than redesigning the build system from the ground up. That is a more pragmatic approach, and it is one that other conventionally-structured distributions can follow.

Verification in Practice

The practical value of reproducibility depends on someone actually doing independent verification. Publishing a SHA256 digest alongside the image is a necessary condition; it is not sufficient on its own.

The Arch Linux infrastructure now supports this: the build artifacts are produced in a way that a third party with access to the same package versions and the same bootstrap tooling can independently reproduce the build and compare digests. The Arch Linux reproducible builds status page tracks which packages meet the standard.

For users pulling the base image, the immediate benefit is that docker pull archlinux:latest now gives you an image whose digest you can cross-reference against an independently reproduced build. That is a meaningful improvement over trusting the registry and the build pipeline unconditionally.

Making the base image reproducible does not make the entire container supply chain reproducible. Application layers built on top of it are usually not reproducible, because most Dockerfiles embed timestamps, download remote resources, and produce output that varies between runs. But the foundation being solid is a prerequisite for everything built on it to have any chance of being solid too, and Arch Linux has now made its contribution to that foundation verifiable.

Was this interesting?