Best Practices: How To Update Your Codebase Without Breaking The Build

Based on research by Veracode, we know that most organizations prefer to limit the number of times they update or upgrade their codebase:

“When developers add an open source library to their application, 79% of the time they never go back to update it,” according to Chris Eng, chief research officer at Veracode.

At ActiveState, we’ve seen a similar pattern by users of the ActiveState Platform who create open source runtime environments, but rarely (only ~28% of projects) update them. 

There are a number of reasons why organizations are reluctant to make changes to their open source codebase, including:

• Dependency Management Issues – any complex application will have dozens of dependencies and hundreds of transitive dependencies. Updating any one dependency is likely to have a cascading effect, requiring multiple components to be updated. This process can result in:
  • Dependency conflicts, which may lead to dependency hell and/or corrupted environments.
  • Breaking the build, since newer dependencies may introduce breaking changes to the way existing code has been implemented.
  • Pulling in new dependencies that may violate licensing and/or compliance rules.
• Recoding – rewriting existing code spends time and resources just to get back to the status quo, providing no benefit to the organization or its customers. In some cases, it can be straightforward. In other cases, you may end up removing the vulnerable dependency altogether, and either coding the functionality yourself or else replacing it with a different but similar, non-vulnerable dependency. Developers have more than enough work on their hands just to meet their sprint deliverables.
• Rebuild Issues – rebuilding the runtime environment means rebuilding each of the new dependencies and transitive dependencies. In some cases, the build process may not have changed much. In other cases, new requirements, new dependencies and changes to the build environment can throw a monkey wrench into the process.
• Redeploy Issues – whenever a fully CI/CD process is in place, redeployment can be trivial. For most organizations, though, redeployment can be delayed by:
  • The need for extensive manual testing.
  • The need to wait for a deployment window in order to avoid disrupting existing customers/users.
  • The need for interdependent components, such as third-party APIs, services or applications to be updated prior to redeployment.

All of this means that time to remediation can be delayed. In fact, Mean Time To Remediation (MTTR) can vary widely from 65 days for a critical security issues to more than 100 days for other issues, depending on a number of factors including language:

Source: Veracode State of Software Security

But sooner or later, End Of Life (EOL) catches up with all technologies, forcing an upgrade process that can be far more complex and resource intensive than ongoing updates:

Figure 2: cost of upgrading older codebases

As the diagram shows, the further your upgrade target is from your original codebase, the more expensive the upgrade will be. This is primarily due to the fact that while your codebase’s dependencies change minimally between patch versions for a language, they will typically introduce breaking changes between minor versions of a language (to say nothing of differences between major versions), meaning that a greater portion of your application will need to be rewritten.

But it doesn’t need to be like this.

Upgrading Your Codebase With Minimal Risk

Security-conscious organizations have long dedicated 10-30% of every sprint to addressing technical debt, which includes updating and/or upgrading aging and vulnerable codebases. 

But as new projects proliferate, it’s often difficult to split a team’s focus between maintenance and new development work. In these cases, some organizations prefer to adopt the best practice of dedicated maintenance teams who are measured by their ability to remediate aging codebases.

Smaller and/or faster moving organizations often feel they cannot dedicate resources to regular maintenance. When their codebase inevitably becomes EOL, there are some best practices that can be implemented to help minimize the cost of upgrading:

• Set up automated Continuous Integration (CI) systems for both the existing version and the target upgrade version.
• Ensure adequate automated test coverage.
• Halt active development, or at least scale it back significantly.
• Minimize your current set of dependencies by removing redundant, outdated or unused ones.
• Create an incremental upgrade strategy to minimize the amount of broken code between minor language versions.
• Use advanced environment management.

Advanced environment management refers to the ability to work with multiple versions of an open source language at the same time as you perform incremental upgrades, but also to be able to manage your open source codebase (and its changes) from version to version. 

With traditional, ecosystem-native tools, this can be a complex, time-and-resource-intensive initiative. The ActiveState Platform has been designed to automate many of the tasks associated with maintaining and upgrading your codebase. 

For example, within a single project, you can create multiple branches for each of the incremental upgrade targets to take your existing codebase (say, Python 3.7) to the final target (such as Python 3.11):

Each branch is automatically built from source code (including linked C libraries) and packaged (in this case) for deployment on Windows or Linux. Each branch will be installed automatically into its own virtual environment, allowing you to work on upgrading your code in a sequential manner without stepping on previously installed environments. 

To get a sense of the amount of work it will take to upgrade, you can quickly visualize the delta between upgrade targets by just changing the version of the language. The ActiveState Platform will automatically resolve all the dependency changes for you:

• Newer dependency versions are highlighted in blue.
• Transitive dependencies that are no longer required are highlighted in red.
• Newly required transitive dependencies are highlighted in green.

In this way, you can judge how many intermediary steps may be appropriate for your upgrade project. Obviously, in this case, jumping from Python 3.7 all the way to Python 3.11 will likely cause far too many breaking changes to be easily managed, encompassing as it does not only changes in dependency versions but also the replacement of some transitive dependencies and the introduction of brand new transitive dependencies, as well.

The process is identical even if you’ve just remediating a single dependency’s vulnerability:

1. Get notified of a newly discovered vulnerability
2. Create a new branch
3. Update the affected dependency and visualize/approve the necessary changes
4. Automatically rebuild the updated environment
5. Redeploy with a single command

Conclusions – Update Codebases Incrementally

Never updating a codebase is a habit that needs to be done away with sooner rather than later. And if the software industry won’t do it themselves, there’s plenty of government legislation pending that will force you to do it. 

But keeping your codebase current doesn’t need to be painful. Just remember that updating a codebase should never be a one-and-done affair. Rather, consistent, incremental changes applied regularly will ensure your application remains secure, performant and maintainable over time. In this way, you can minimize risk by spreading changes out over several releases, which will make resolving any issues that may crop up with each build much easier to deal with.

Using the ActiveState Platform to help automate changes in your codebase goes a long way to minimizing the manual work that needs to be done:

1. Select newer and/or non-vulnerable versions of dependencies from our continually updated catalog.
2. Automatically resolve dependencies/ transitive dependencies and review the changes BEFORE you rebuild/redeploy so you can understand the ramifications.
3. Automatically rebuild the runtime environment in minutes for all target platforms.
4. Redeploy the updated runtime with a single command.  

You can test it out for your project by creating your own free ActiveState Platform account.

Next Steps

Discover the ultimate solution for language version management with ActiveState’s Get Current, Stay Current. Say goodbye to dependency pains and hello to seamless coding. Click here to explore Get Current, Stay Current now!