Hackers are increasingly targeting software development systems, open source resources and software build pipelines in order to gain economies of scale when it comes to compromising software supply chains. A single compromised software artifact in a popular application can potentially compromise hundreds or even thousands of downstream customers. It’s one of the key reasons that Sonatype found a 742% increase in software supply chain attacks over the past three years.
As a result, Gartner is predicting that “By 2025, 45% of organizations worldwide will have experienced attacks on their software supply chains, a three-fold increase from 2021” (Gartner report: How Software Engineering Leaders Can Mitigate Software Supply Chain Security Risks). Online marketplace vendor Capterra confirms the trajectory, citing the fact that “three-fifths (61%) of US businesses have been directly impacted by a software supply chain threat” in 2022. And 2023 saw twice as many software supply chain attacks as 2019-2022 combined.
According to IBM, the average cost of a supply chain attack is $4.46M, and requires an average of 26 days to identify and contain (IBM 2022 report: “Cost of a Data Breach”). Just as impactful, more than 50% of organizations have lost trust in legacy vendors due to supply chain attacks.
But ensuring the integrity and security of the software development process can be a confusing undertaking given the plethora of commercial and open source software positioning their solutions in the software supply chain security space, as well as the breadth and depth of the software supply chain, which affords multiple points of entry for malicious actors who are always looking for the weakest link in the chain to exploit:
- Breadth – most organizations work with multiple open source languages, and import their code from more than one public repository. Because there are no industry-wide standards in place today, each language and repository may require its own solution.
- Depth – There is a large set of security & integrity best practices that can help, but only the largest enterprises can hope to implement and maintain them all.
- Change – most software artifacts have a short shelf life before bugs, vulnerabilities, incompatibility with newer systems and other issues crop up. Correcting all these issues is difficult for all but the most dedicated organizations to do in a timely manner.
For all these reasons, it’s imperative that organizations implement tools, processes, applications and solutions that can help automate many of the issues that arise during key phases of the software development lifecycle, including:
- Importing/Sourcing – the process of importing third-party tools, libraries, code snippets, packages from online resources, as well as pulling proprietary code from local repos.
- Building – the process of compiling, building and/or packaging code, usually via an automated system that also executes tests on built artifacts.
- Deploying/Using – the process of deploying and remediating customer software, as well as working with built artifacts in development, test and production environments.
ActiveState is working on a comprehensive Buyers Guide that can help organizations navigate each stage of the software supply chain with confidence by choosing the right solution for the issues you may be experiencing. This blog post provides an introduction and overview of the Guide to help you get started.
Securing the Software Supply Chain Import Process
The import process is arguably the weakest link in the software supply chain due to the open nature of open source ecosystems that allow anyone to publish anything without a rigorous process to eliminate threats prior to publication.
As a result, each organization must create their own process, implemented with appropriate tools and solutions, to ensure the integrity and security of the code they import. For example:
In this example:
- If prebuilt components are downloaded, they should be staged in an artifact repository like Sonatype Nexus, JFrog Artifactory, or similar where they can be examined before being made available to developers.
- If source code is downloaded instead (for example, if you are vendoring all the dependencies your application requires), you might want to stage it in a code repository like GitHub, Bitbucket, GitLab, etc instead.
- All imported artifacts must be scanned for malware, trojans, backdoors, typosquatting and other malicious vectors using a code scanner such as UpGuard, BluEye, or similar solution.
- If prebuilt binaries are also being imported, you may require a separate binary scanning tool that can disassemble and examine compiled code, such as the Static Application Security Testing (SAST) Tools from Veracode, Checkmarx, or similar.
- You may want to consider implementing a second, “quarantine” repository where components can be further analyzed for security and integrity using Software Composition Analysis (SCA) Tools like BlackDuck, Snyk or similar to identify vulnerabilities.
- Some enterprises may want to perform further checks for compliance with corporate guidelines by identifying non-commercial-friendly licensing.
- Finally, a third “development” artifact repository is where all “ready to use” artifacts are placed for use by development teams. This repository should provide continuous vulnerability scanning, generate a vulnerability status, and send automated notifications when a vulnerability is found. Notifications should also include patch/upgrade recommendations. The aforementioned Sonatype Nexus, JFrog Artifactory will serve here, as well.
There are numerous other tools that can be considered along this chain to help with automation, including:
- Policy management tools that can prevent the importation of any artifact that does not conform to corporate governance guidelines. Both general solutions like Open Policy Agent (OPA) and IDE-based solutions like Snyk can be considered here.
- Automated remediation tools like GitHub’s Dependabot, Debricked or similar can help quickly fix vulnerabilities through automating the upgrade process when a newer version of the component is released.
Implementing these kinds of best practices and tools can help avoid common exploits and malware associated with open source, including using known vulnerable components, avoiding typosquatted resources, and incorporating GPL licensed code in commercial applications. For example, Sonatype reports “2.1 billion OSS downloads with known vulnerabilities in 2023 could have been avoided because a better, fixed version was available.”
Securing the Software Supply Chain Build Process
The software artifact build process is typically automated through the use of Continuous Integration (CI) systems that must be hardened against attack and intrusion. The pipeline itself also offers a number of opportunities to better ensure the security and integrity of the artifacts being built, including:
- Source Code Integrity – the provenance or source of the code must be ensured before the build process can be kicked off. Consider adopting software attestations that help verify the origination of a component and how it was built such as those provided by TestifySec Witness, GitHub Actions or similar.
- Runtime Integrity – prior to populating any container with its runtime environment, consider verifying the integrity of the runtime against its Software Bill Of Materials (SBOM) to ensure all components (and no extra components) are present. SBOM vendors include Anchore, FOSSA, Mend, etc.
- Hardened Build Service – implement strong tamper protection for your build service by following best practices that include isolated, ephemeral, hermetically sealed containers, as well as build scripts and a signing service that are inaccessible to users. Signing services include examples like SigStore, Microsoft Authenticode, or similar.
- Reproducible Builds – use of a deterministic build service to ensure that the output is always identical given the same inputs. CI solutions that can be made deterministic include CircleCI, Bazel, or similar.
- Code & Executable Testing – the CI process should incorporate both Static and Dynamic Application Security Testing. SAST and DAST tools include those from Acunetrix, SonarQube, Fortify, etc
Implementing these kinds of best practices and solutions can help avoid becoming the next Solarwinds, whose build process was compromised resulting in hundreds of downstream customers being affected including the US government, all arms of the US military, the big five accounting firms, and more.
Securing the Software Supply Chain Deployment Process
Deployment systems, whether used for releasing on premise versions of an application or hosted SaaS deployments must be appropriately hardened. The same is true for any system that delivers automated security updates/patches.
Post-release, vulnerabilities should continue to be tracked, with customers being provided notifications and timely updates.
More on this section will be provided in the Buyers Guide, along with other areas including:
- Secrets management
- Cloud / PaaS systems
- Dependency management
- Threat modeling
- OS and language distributions
- And more
Conclusions: The Best Way to Secure the Software Supply Chain
While multiple Secure Software Development Frameworks (SSDFs) have existed for decades, they proved insufficient to address software supply chain attacks. Cue the market, with vendors of all stripes jumping in to fill the gap with everything from new security frameworks to specific point solutions to one-size-fits-all platforms.
Integrating multiple best-in-class point solutions can be time and resource consuming, and require extensive maintenance to upkeep as newer versions are released and integrations break. On the other hand, adopting a monolithic platform may bring with it numerous limitations since it’s unlikely to be flexible enough to address all use cases.
There’s also the question of how much of the solution is proprietary, engendering lock-in versus those that follow open source principles. ActiveState, for example, takes a hybrid approach, offering a platform that supports open source standards designed to integrate seamlessly with existing software development processes.
With organizations facing a veritable tidal wave of choices, it’s no wonder that most of them are knee deep in analysis paralysis, unsure of where to even get started. This is why we created the Journey to a Secure Software Supply Chain ebook that can help you evaluate your current stance, and create a plan to improve it. Coupled with our upcoming Software Supply Chain Security Buyers Guide, organizations will be able to obtain a comprehensive view of the market landscape so they can pick and choose the best solutions at each stage in their journey.
This blog is a preview of our Buyers Guide which will help organizations implement tools, processes, applications and solutions that can help automate many of the issues that arise during key phases of the software development lifecycle.