Easy Python Dependency Management With Shareable Environments

Last week’s discussion of Python dependency management on Reddit was indicative of how much confusion developers new to Python (and not so new) still experience when installing and managing their dependencies. It all started with a post entitled:

“After using Python for over 2 years I am still really confused about all of the installation stuff and virtual environments”

The more than 300 responses were dominated by posters commiserating with the original opinion. XKCD, the source of all geek wisdom, represents the problem succinctly:

Source: https://xkcd.com/1987 

In other words, contrary to the Zen of Python, there are multiple ways to install and manage Python dependencies, and they can often step on each other. Worse, none of the commonly used tools provide a simple, comprehensive solution to the problem of managing multiple versions of Python/dependencies on a per project basis:

• pip is fine for managing dependencies, but you’ll want something else to manage your virtual environments.
• Conda can manage dependencies and virtual environments, but is not very good at resolving dependency conflicts, which can lead to corrupt conda environments.
• Poetry is good at dependency conflict resolution, but far slower than many users would like.
• Homebrew is fine for Mac or Linux, but doesn’t support Windows.

And the list of exceptions goes on. All these issues are infinitely more important these days given the popularity of Python. Stack Overflow’s latest developer survey shows that almost 60% of all respondents are currently learning Python:

Not having a simple, standard way to manage Python dependencies and environments is especially limiting for beginners who will want to try out multiple projects (likely requiring multiple different versions of Python) on their system in order to get their feet wet. 

But even if you’re a seasoned Python developer, getting everyone on your team set up with identical versions of the dev environment for your project is far more painful than it should be. While you can always use a config file like requirements.txt or pipfile.lock, you still need to remember to update it before distributing it. Even then, configurations can get out of sync as team members update their local environment for their own purposes. And what happens when a team member is on a different OS? Your config file may be missing key, OS-level dependencies. 

Shareable Python Environments

Do we really need yet another package management solution for Python? Don’t take our word for it – here’s some of the thoughts from the aforementioned Reddit thread:

• The overhead necessary to juggle virtual environments quickly becomes very, very burdensome.
• This [venv] will not give you reproducible builds across a team. Python is a mess on this front.
• My tech lead is a very experienced Python developer and he’s told me multiple times that he hates dependency management in python.
• It [poetry] would be much nicer if it wasn’t build upon sand (of pip/setuptools).
• 15 years with python, and every second year they put additional something on top of existing terror.
• Yeah open secret is that it’s bad even if you follow best practices.

And most pointedly:

“I’m noob thinking of getting deeper into Python .. but reading all these comments is giving me indigestion”

Rather than scare off those that are new to Python, the ActiveState Platform is designed to take all the pain and guesswork out of managing Python projects and their dependencies. But it really shines when it comes to sharing your project with others by reducing a complicated dev environment setup down to a single command.

As an example, assume a project has a main branch for your current production release that requires Python 3.9. But there’s also a branch with a new feature (aptly named “awesome-new-feature”) that requires Python 3.10.

With the ActiveState Platform, it doesn’t matter whether you have Python installed, or even if you have the wrong version of Python. It also doesn’t matter what OS you’re running. The ActiveState Platform will always ensure you have the right dependencies for the given OS and git commit you’re working on.

Assuming you have curl installed on your machine, this single command (for Mac or Linux) is all you need to get your environment set up and ready to start coding: 

sh <(curl -q https://platform.activestate.com/dl/cli/655424048.1642518345_pdli01/install.sh) -c'state activate --default ActiveState/onboarding'

Once the installation has finished, you can run the unit tests. We’ve set up a simple entry point in the activestate.yaml, which gets installed in your environment called “testit” that invokes pytest for you:

Note from the output, it’s using Python 3.9.15.

But what if you have to work on the other branch which requires Python 3.10? Simply checkout the GitHub branch, and then run testit again:

That’s it! Thanks to the git hook and State Tool, whenever you change branches or pull in new commits from a remote repo we ensure all your dependencies are up to date. And we mean, ALL YOUR DEPENDENCIES, including compilers, interpreters, C libraries, dev tools. And so on.

Conclusions: Python Dependency Management Made Easy

Managing dependencies and environments should be the last thing you need to worry about when you’ve got pressing deadlines. But as many Python newbies (as well as some experienced Pythonistas) have noted, the current package management and environment solutions don’t offer a simple solution that just works. 

This is one of the key reasons the ActiveState Platform exists. Not only will it automatically build all the dependencies your project requires securely from source code, but it will also resolve all your dependencies, or else tell you how to resolve any conflicts. No more dependency hell. 

And because every project is installed into a virtual environment by default, there’s no need for a secondary tool to manage your environment. Sharing your project then becomes a single command that finally solves the problem of reproducible builds across teams.

• If you’d like to try out the project used in this post, you can find it on Github. The repo also includes instructions for Windows users.
• If you’d like to set up git hooks for your existing projects, refer to our docs.

Read Similar Stories