In March 2024, a Postgres developer named Andres Freund noticed SSH logins on a test machine used a little more CPU than they should. This was a nit, but just enough to be annoying if you happen to be the kind of person who profiles things that are already working. Freund pulled on that thread and found a backdoor deliberately inserted into xz utils, a compression library sitting quietly inside most Linux distributions.

Most people expect these backdoors are in the project's source code itself. In this case, the build scripts were compromised. This is the often-overlooked infrastructure that takes clean source, the stuff usually reviewed, and turns it into the binaries people actually run. Savvy attackers love to put their exploits in the dark corners of projects where they go unnoticed.

If you cannot independently verify that a given binary came from a given source, you're implicitly trusting the build process itself. The xz incident attempted to exploit that trust. Fortunately, Freund noticed the problem before it spread, but collectively we could catch problems like this even earlier, if we're more deliberate about what we build and how.

I think about that case a lot, because it is the cleanest possible argument for why "we reviewed the code" and "we know what we're running" are not the same claim.

Reproducibility Means the Build, Not Just the Code

A reproducible build environment is a setup where identical source code always produces an identical, verifiable binary, provable by comparing a cryptographically secure hash. Two developers on different machines, or even an internal CI server compiling the same commit a year later, get a byte-for-byte match.

To achieve this goal, you must lock down every layer that could otherwise cause any drift between outputs: the language runtime, direct and transitive dependencies, the compiler version, system libraries, and the environment variables the build happens to inherit, including things like timestamps of the builds themselves! If you miss any one of those, you lose the ability to trust that the same outputs come from the same inputs always under your control.

The Debian Reproducible Builds project has been doing this work in the open for over a decade, with actual tooling built specifically because "trust me, I built it correctly" stopped being good enough a long time ago. I recommend especially their diffoscope tool, to compare two build outputs and identify exactly where they diverge.

Three Places Build Environments Actually Drift

What will you find if you go down this path?

Unpinned transitive dependencies. A developer pins the exact version of the library they explicitly chose. The dependencies that library pulls in by itself, without human scrutiny, often go unpinned. When one of those updates on its own schedule, a build that worked last week fails this week, with no single commit anyone can point to as the cause. That failure rarely stays contained. Someone has to stop mid-sprint to figure out which of a dozen transitive packages moved, whether the fix is safe, and whether it breaks anything else downstream. A two-hour investigation becomes a lost afternoon, a lost afternoon becomes a missed sprint commitment, and nobody planned for any of it because the dependency that changed wasn't one anyone was watching.

If you've seen the news about exploits such as the axios library on the JavaScript npm repository, you'll know that updating to the latest released version of any package arbitrarily, deliberately or not, can provide attackers a way into your codebase.

Reliance on scan and pray. Many organizations let developers pull packages freely from public registries during a sprint, then run a scanner against the finished build and a remediation backlog back to the developers. That is reactive by construction. At best, it costs you time and velocity, because a developer has to stop feature work to untangle a dependency conflict that a security patch just introduced. Worse, it does nothing to prevent the next conflict.

System-level drift. Application package managers mostly ignore the operating system underneath them. If a package needs a specific C library to compile from source, and that library's version differs between a laptop and the build server, compilation may fail, and someone has to debug it. Worse, compilation may work but the software may silently fail in production as it's running.

Functional Package Managers Treat a Build Like a Pure Function

Feed a pure function the same inputs and you get the same output, every time, guaranteed. I keep my eyes on two projects that apply the same logic to the dependency graphs of software builds, pun intended: Nix (likely the most popular) and GNU Guix (a Nix-derived alternative).

Nix demonstrates the impressive level of discipline necessary to get this to work. Every package lives in its own directory under /nix/store, named by a cryptographic hash of its exact inputs. This allows multiple versions of the same library to coexist on one machine without stepping on each other. When a developer shares a project's configuration, the recipient's machine can recreate the exact same environment graph rather than an approximation of it.

This takes real discipline to adopt. Nix's learning curve is not gentle! The payoff for that investment is an onboarding process that takes minutes instead of days, and a security team that can audit a fixed, immutable snapshot of the stack instead of asking engineering what's actually running and hoping the answer is current. I prefer to pay up front for a repeatable process than to pay unestimable costs later, under difficult and stressful circumstances.

Curate and Govern, Not Scan and Pray

Scaling a reproducible build process across an organization means changing how you handle open source consumption in the first place.

Curate what enters instead of scanning what's already there. An automated system that builds and maintains artifacts with known provenance, where developers pull from a curated, approved catalog rather than raw public registries, is a different and safer approach than scanning after the fact. This is the point at which I point to ActiveState's Curated Catalog. It applies exactly this model, governing ingestion before a package reaches a developer rather than flagging it afterward.

Enforce governance at the point of origin. Policy has to apply before a package enters a workflow, not after it shows up in a release candidate. The strongest discipline in open source software supply chain security is about where in the pipeline you can stop something unexpected from propagating.

Make the secure path the easy path. Governance that developers can route around is a suggestion instead of a policy. That is not a criticism of developers; it happens to any control that adds friction without adding a corresponding benefit at the point where the friction lands. Sure, Nix onboarding will take some time, but the friction in such a system pushes people to do the right thing, not the lazy and insecure thing.

Treat security and engineering as one team with two jobs, not two teams with opposing incentives. Security cannot enforce policy on decisions it never sees. Engineering cannot ship predictably when a security finding interrupts urgent work already underway. A reproducible, governed catalog removes that structural conflict instead of asking either team to absorb it.

Ten Steps to a Reproducible Environment

1. Define your dependencies explicitly. Document every library, framework, and runtime your application needs. Use a declarative configuration file as your single source of truth.

2. Lock all transitive dependencies. Lockfiles pin exact versions of both direct and transitive dependencies, so an update three layers deep doesn't break your build in six months with no obvious cause.

3. Curate your open source ingestion. Change your process away from scanning what already arrived to curating what's allowed to arrive. A catalog of components built from source gives you provenance you don't have to reconstruct after the fact.

4. Standardize OS and system libraries. Reproducibility has to cover the operating system layer too. Use carefully-maintained container images or a functional package manager such as Nix to isolate the exact compiler and system binary versions your application requires.

5. Build from source. Pre-compiled binaries from public registries ask you to trust a build process you didn't run and can't verify. Building from source in an isolated environment is the baseline of Reproducible Builds. It's the only way to guarantee provenance.

6. Remove network access during builds. A genuinely reproducible build doesn't depend on live internet access mid-build. Fetch source and artifacts before compilation starts. If the build can reach the network, it can silently pull in something you didn't intend.

7. Eliminate time and user variables. Timestamps, user IDs, hostnames, and anything local to the machine running the build changes the final hash even when nothing about the actual source changed. This is fiddly to get right across toolchains, but the confidence you get is worth the effort.

8. Establish SLA-bound remediation. The industry average mean time to remediate a critical vulnerability is 54.8 days. Remember the long tail of Log4Shell: many organizations were still finding vulnerable instances more than a year after the initial disclosure, not because nobody knew, but because nobody could quickly and confidently rebuild what needed rebuilding. A contractual SLA, such as ActiveState's 5 business days for critical CVEs, is a bet that speed of remediation matters more than speed of initial patch availability.

9. Automate environment provisioning. Developers shouldn't hand-configure new laptops. When you've invested in deterministic, single-source-of-truth, reproducible builds, you can have a single script that provisions the locked, reproducible environment on any machine.

10. Verify cryptographic hashes. Build a CI check that compares output hashes against a known baseline. If a build produces a different hash from identical source, stop the deployment and find out why before it ships, not after.

Common Use Cases

What do you get from this work, besides fewer arguments between developers and security folks and higher confidence in what you've deployed?

Vulnerability remediation. Security finds a critical CVE in a logging library. Because the environment is reproducible and dependencies are governed at the point of origin, engineering can test the safest upgrade path immediately, without first fighting a build that won't cooperate.

Developer onboarding. A new engineer joins a distributed team and runs one command instead of losing three days to local system dependencies and a wiki that's a year out of date. Coder and SlashData's State of Development Environments 2025 report, surveying more than 550 enterprise developers, found that only 63% of organizations can stand up a new development environment within a day, and the slowest 2% take more than a month. A reproducible, automated environment is how you land on the right side of that number.

Regulatory compliance auditing. A regulated financial institution needs to prove exactly what code is running in production. Reproducible builds let an auditor compile the source independently and match the resulting hash to the deployed artifact. That is a much stronger claim than "we scanned it and it looked fine."

Now the caveat. I don't think reproducibility is a solved problem for most ecosystems, and I'd rather say that than pretend otherwise. Some language ecosystems make this genuinely harder than others, particularly ones with native, compiled extensions or GPU-dependent packages, like Python and Node.js. There are legitimate cases where full byte-for-byte reproducibility isn't practical yet.

However, I do believe this is the direction we're all going to go, sooner or later: you don't have to be on the absolute bleeding edge of every dependency update. Running a few days or weeks behind is fine. Even so, you cannot be blind to what's actually in your build and how each piece gets there.

The gap between organizations that understand this risk and organizations that have actually done something about it is where incidents like xz utils find room to sit undiscovered for years.

Contact us if you want to see how the ActiveState Curated Catalog handles the ingestion side of this argument.

Shane Warden is Principal Architect at ActiveState, where he works on package management, build reproducibility, and open source security across ActiveState's language ecosystems. He is also a practicing open source maintainer with direct experience triaging his own project's security disclosure process.