Reducing Mean Time To Remediation for Open Source Vulnerabilities

Reducing Mean Time To Remediation (MTTR) is top of mind for most security-conscious organizations using open source software.  The likelihood of vulnerabilities in any given open source package is an acceptable risk for most organizations given the convenience and speed that public repositories of prebuilt software like PyPI offer.

Unfortunately, when you Combine the prevalence of vulnerabilities with the fact that 80% of codebases are not updated once they’re built for fear of breaking the build, you have an attack surface primed for hackers. A quick Google search (or even just conventional wisdom) will tell you that implementing tools designed to continuously monitor and identify vulnerabilities is the first step in a successful remediation process.  But this is still truly a reactive approach in which the steps for remediation look something like this:

source: snyk

Essentially a continuous cycle made up of a few steps that include:

• Finding vulnerabilities through scanning, which can be done with open source or paid tooling.
• Prioritizing vulnerabilities by severity or other internally-defined criteria like the applications impacted or the type of potential exposure.
• Fixing or remediating through patching, taking the application down, or rolling back to an older version.
• Monitoring for new vulnerabilities, and the cycle continues.

Even with a continuous stream of vulnerability identification, prioritization and patching, on average ittakes 2 to 5 months for an organization to complete this cycle!! It is difficult to cut this down much further using the aforementioned methodology, which is why we recommend implementing a proactive approach by establishing four key principles:

1. Understand what’s in your open source
2. Know where that open source package and all its dependencies are deployed
3. Have audit-ready records of what open source is in use and where it came from
4. Provide secure, reproducible developer environments

 

Understanding what’s in your open source

This is a little more difficult than it may sound at first. Most large organizations Know which packages are being actively used by the development team, typically through the use of a binary repository like JFrog Artifactory or Sonatype Nexus that contains only “approved” packages.  But identifying all the transitive dependencies down to and including native libraries remains a challenge for most organizations.

Transitive dependencies hidden beneath the first visible layer can also hide other dependencies and libraries, which are undetectable through traditional SCA or AST tools (infrastructure, static and dynamic application security testing). For example, Lots of commonly used Python packages depend on C libraries, or have multiple dependency layers contained within them.  One option to gain visibility is to manually or automatically create dependency trees and graphs for each package in use. Even once this is established, tracking where every package originated from and whether it was built securely in the first place can be a challenge, especially for languages like Java, where source code is less available than in languages like Python.

The other option is to use a platform like ActiveState, which has its own catalog of millions of open source packages originally sourced from public repositories, but that have been traced down to the lowest dependency level in order to be able to build everything securely from source code in a reproducible manner..  These are denoted in the platform with a “Secure” and “Built from Source” tag.  This is a key principle in preventative supply chain security: knowing exactly what is in your open source and how it was built.

Knowing where your open source packages & dependencies are deployed

Tracking who pulled which open source packages can also be done with the use of tools like JFrog Artifactory or Sonatype Nexus, which host approved packages in a central repository.  But tracking the system or environment where packages have been deployed can be a highly manual process. Instead, most organizations employ SCA tools to identify at least top-level dependencies, but again, those sitting below the surface may go undiscovered.  This means that scanning for vulnerabilities can take longer to identify, or even go without identification altogether.

Alternatively, a solution like ActiveState quickly shows all dependencies in use, and it is as simple as looking at your CVE view (Common Vulnerabilies and Exploits view) to see which packages are in use across a specific organization within your enterprise.  In the ActiveState Platform, CVEs associated with a dependency can be tracked and searched for by CVE number, language, severity level or any other free text you choose. This view can also be used to help prioritize triage since it shows the impact of any one CVE on all the projects in an organization:

Generate audit-ready records of your open source

Speaking of knowing who is using what, where it came from, and where it’s being used, down to the lowest transitive dependency, keeping those records audit-ready also greatly reduces the remediation cycle.  In the case that you do not have immediate visibility into where every dependency is deployed, or if you’re using a manual vulnerability discovery process, having auditable records on what is contained in each package is a great start.  Records that are machine-readable are even better, and these can come in a couple of (largely automated) forms that work better together, such as SBOMs (Software Bill of Materials) and attestations.

An SBOM is an auditable version of all of an application’s dependencies.It is akin to the ingredients list on a package of cookies in that you can see all the things which make up the cookie.  Even better than a cookie, SBOMs also provide hashes and external links to known vulnerabilities which can help make remediation even quicker.

There are three output standards, CycloneDX, SWID (Software ID), and SPDX (Software Package Data Exchange®), which all provide slightly different information in different downloadable formats.  If you are generating SBOMs each time you create a runtime environment for your application, this is a great step toward providing the audit trail of what is in use.

ActiveState also provides ultra-simple SBOM generation for any project built with the ActiveState Platform GUI or CLI (the State Tool) in an SPDX format.  You simply log into your ActiveState account, and locate the “Get SBOM” button (contact us to test if not an enterprise customer).

Now you can see all the components in use, but where it came from is still missing.  This is where attestations come into play.  Attestations provide documented assurance of where the components were sourced and how they were built – was it built on a developer’s laptop or was it created using a secure build service?

Attestations allow you to determine how much you trust the components you are using, as well as whether you are taking an acceptable level of risk by integrating that open source into your own build.  Attestations also come in two formats;

1. Software Attestation – this tells you how an application was created, including who built it, how was it compiled, and where was it constructed
2. Provenance Attestations – these tell you the origin of the source code for all open source dependencies/ transitive dependencies in a build. Provenance attestations will also tell you whether a dependency was  pre-built rather than being built from source.

Provenance Attestations are also available in the ActiveState Platform, along with the SLSA Levels for all dependencies in a completed build (bonus!).  If you want to try it for yourself, check out this blog with three quick steps to generate a Provenance Attestation for yourself.

If you want to start leveling up with SLSA (Supply Chain Levels for Software Artifacts), attestations are fundamental to Level 1.  In addition, if you are one of those organizations selling to the government, consistently generating SBOMs and attestations will keep you compliant with regulations like Executive Order 14028.

Provide secure, reproducible developer environments

Once you know where your open source components came from and have assessed your level of acceptable risk, the final principle is to make this a repeatable process for your developers.

Solutions to this include:

1. Vendoring your own dependencies and managing each distribution with a dedicated build or DevOps team
2. Providing only approved, pre-built packages in a binary repository for use
3. Providing a template for secure developer environments, which can easily be built from source every time

In options 1 and 2, the burden of effort to secure developer environments requires another manual intervention of human screening, or the combination of human and machine based screening to determine which dependencies or packages are permitted.  Then a repository is needed to host them, and other guidelines or policies need to be set to enforce usage.

Many companies set up DevOps, DevSecOps, AppSec and Build/Infrastructure Engineering teams to do exactly this! Issues tend to arise when developers violate policy or attempt to use non-permitted packages, which then may get kicked back for updates after application development has already started, or even before.  This creates developer vs security tension that is familiar to most organizations, where development speed is paramount but security risk also needs to be managed.

ActiveState also offers a solution to this problem, by providing a template for developer environments (called Projects in the ActiveState Platform UI), which can then be reproduced on any developer’s machine and will be built from source each time. 

A Project can be created by:

• Any developer using the entire ActiveState package catalog, if you are comfortable with assessing vulnerabilities post-build.
• Any developer using a subset of packages in a Managed Distribution, which features a curated set of approved packages and dependencies.
• Your security team and then shared with the development team, leaving it up to your security team to determine vulnerability exposure limits, update dependencies, and set the package list for each development environment.

Once you build a secure, reproducible developer environment (aka Project) in ActiveState, it’s as simple as hitting the “Share” button!

Reducing mean time to vulnerability with the ActiveState Platform

Ultimately, the first three principles discussed above are really about open source observability and the ability to monitor who is using which open source packages, where they came from, and where they’re currently deployed.  Open source observability is the backbone that helps stand up supply chain security, and coupled with the ability to reproduce secure developer environments at scale, provides a true 3SC (Secure Software Supply Chain) solution.

At ActiveState, we believe that avoiding vulnerabilities before they enter your supply chain is the best course of action, followed by understanding where they are at all times for rapid remediation.

Read Similar Stories