How to Package Python dependencies with PIP setuptools
Try a faster and easier way to manage your Python dependencies. Use Python 3.9 by ActiveState and build your own runtime with the packages and dependencies you need. Get started for free by creating an account on the ActiveState Platform or logging in with your GitHub account.
If you’re packaging a Python package for publication, how do you ensure that all of the required dependencies are included? Just like pip is the standard package manager for Python, setup.py is the heart and center of Python projects installed with pip. Simply put, setup.py is a build script template distributed with Python’s setuptools package.
Setuptools is the Python Packaging Authority (PyPA) package development process library and utility for building Python projects based on packages and their dependencies listed in a setup.py script. A Python file that relies only on the standard library can be redistributed and reused without the need to use setuptools. But for projects that consist of multiple files, need additional libraries, or need a specific version of Python, setuptools will be required.
This article provides a brief tutorial on how to work with setuptools.
Update Python Tools
Setuptools is typically installed with Python downloaded from python.org, so there’s no need to separately install setuptools. Instead, your first step should be to use pip to update your Python installation to the latest version of setuptools on popular operating systems like Windows, Linux or macOS:
$ python -m pip install –upgrade pip setuptools
If you get a permissions error, instead of using sudo, consider creating a virtual environment with virtualenv, venv, pipenv or pyenv. If you’re working with more than one project, rather than installing everything in /site-packages, you’ll want to work with virtual environments to ensure each project has just the dependencies it requires.
The following script is an example of how to create a setup.py script for your Python 3 (not Python 2) project. The script syntax spells out the information you need to provide, including metadata such as:
- Project name – enter a name for your project in quotes
- Python packages and Python modules to include in the distribution (dist) – the find_packages(‘,’) default argument will incorporate all packages that include an __init__.py file and are located in the local directory (dir) where setup.py is installed
- Project version number – enter a version number for the project in quotes
- List a license for the project – enter the name of the license you are licensing your project for use under in quotes
- Short description of your library (lib) – enter a brief description of your project in quotes
- Long description of your library (lib) – enter a more detailed description of your project either in text or using markdown
- Your name – enter your name in quotes to denote who the author is
- Your email address – optionally provide an email address in quotes where users can contact you
- Link to your GitHub repository or website – optionally provide a link to your project’s repo, list your maintainers, etc
- Download Link from where the project can be downloaded from – provide an URL to your project’s source code in quotes
- List of keywords – provide a list of keywords in square brackets associated with your project to make it easier to search for
- List project dependencies – provide a list of all the dependencies your project requires in square brackets. The easiest way to provide this is to copy and paste from the install_requires section (see below).
- https://pypi.org/classifiers – provide a list of all classifiers that apply to your project in square brackets. You can find a comprehensive list of classifiers at https://pypi.org/classifiers
from distutils.core import setup from setuptools import find_packages import os # Optional project description in README.md: current_directory = os.path.dirname(os.path.abspath(__file__)) try: with open(os.path.join(current_directory, 'README.md'), encoding='utf-8') as f: long_description = f.read() except Exception: long_description = '' setup( # Project name: name='', # Packages to include in the distribution: packages=find_packages(','), # Project version number: version='', # List a license for the project, eg. MIT License license='', # Short description of your library: description='', # Long description of your library: long_description=long_description, long_description_content_type='text/markdown', # Your name: author='', # Your email address: author_email='', # Link to your github repository or website: url='', # Download Link from where the project can be downloaded from: download_url='', # List of keywords: keywords=, # List project dependencies: install_requires=, # https://pypi.org/classifiers/ classifiers= )
install_requires is a section within the setup.py file in which you need to input a list of the minimum dependencies needed for a project to run correctly on the target operating system (such as ubuntu). When pip runs setup.py, it will install all of the dependencies listed in install_requires.
For example, if your project includes matplotlib, you’ll need to list it, as well as its dependency of NumPy in install_requires as shown below:
setup( ... install_requires=[ '<matplotlib>', '<numpy>' ] ...)
Example of Install_requires with Platform Specific Dependencies
You can modify install_requires by adding in platform-specific and version-specific dependencies, depending on the environment that a project is to be installed on. For instance, if our previous example included matplotlib v3.2.1 and numpy v1.17.4 on Python 3.6.6 for Linux, we could specify:
setup( ... install_requires=[ "<matplotlib>;python_version<'<3.6.6>'", "<numpy> >= <1.17.4>;platform_system=='<Linux>'" ] ...)
To install a setup.py file including dependencies listed in install_inquires:
$ python setup.py install
When the command is run, all of the dependencies not already installed will be downloaded, built (if necessary), and installed. Any scripts that require specific dependencies at runtime will be installed with wrappers that ensure the correct versions are added to sys.path (system path).
Dependency Not Found in PyPI
If your project has a dependency requirement that is not currently in the Python Package Index (PyPI), you can still include them if they can be accessed via http and are packaged as either an egg, .py file, or a VCS (Version Control System) repository, such as Git or Subversion, rather than just a tarball (tar.gz).
In order to download dependencies not found in PyPI, you will need to add URLs to a dependency_links section under setup() in the setup.py file. Assuming that the dependencies are packaged correctly, they will be automatically installed:
setup( ... dependency_links=['http://github.com/<username>/<reponame>/tarball/master#egg=<packagename>-<version#>'] ...)
To check a dependency not found in PyPI, prefix the package name and version # with _ (instead of -) in the dependency_links argument:
A modern solution to Dependency management – Try ActiveState’s Platform
Dependency resolution is at the core of the ActiveState Platform. When you create a project and start adding requirements, the Platforms tell you what dependencies those requirements have.
The ActiveState Platform is a cloud-based build tool for Python. It provides build automation and vulnerability remediation for:
- Python language cores, including Python 2.7 and Python 3.5+
- Python packages and their dependencies, including:
- Transitive dependencies (ie., dependencies of dependencies)
- Linked C and Fortran libraries, so you can build data science packages
- Operating system-level dependencies for Windows, Linux, and macOS
- Shared dependencies (ie., OpenSSL)
- Find, fix and automatically rebuild a secure version of Python packages like Django and environments in minutes
The ActiveState Platform aims to handle every dependency for every language. That means handling libraries down to the C/C++ level, external tools, and all the conditional dependencies that exist. To take things even further, our ultimate goal is to support multi-language projects. That means that you can create a project using both Python and Perl packages, and we’ll make sure that both languages are using the same (up to date) OpenSSL version.
Python Dependency Management In Action
Get a hands-on appreciation for how the ActiveState Platform can help you manage your dependencies for Python environments. Just run the following command to install Python 3.9 and our package manager, the State Tool:
powershell -Command "& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString('https://platform.activestate.com/dl/cli/install.ps1'))) -activate-default ActiveState-Labs/Python-3.9Beta"
sh <(curl -q https://platform.activestate.com/dl/cli/install.sh) --activate-default ActiveState-Labs/Python-3.9Beta
Now you can run state install <packagename>. Learn more about how to use the State Tool to manage your Python environment.
Let us know your experience in the ActiveState Community forum.
Frequently Asked Questions
Source Distributions (sdist) and Wheels (whl) are distribution formats for Python code. Wheels are precompiled, platform-specific binaries that provide faster installation compared to Source Distributions.
pip can install from either sdists or whls, but will prefer wheels.
There are currently two major ways to package your project in Python: using setuptools and using project.toml.
With setuptools, you’ll need to:
- Lay out your project on your local machine using an appropriate directory structure.
- Create a setup.py script that describes all the metadata for your project.
- Create your first release.
- Register your package with the Python Package Index (PyPI).
Pyproject.toml is a more modern way to do Python packaging. In this case, all the above steps still apply except #2. First:
- Add pyproject.toml to your project. For example, the following configuration won’t change how your project gets built today, but it will explicitly tell users what you want to happen when a build is started:
[build.system] requires = [”setuptools”, “wheel”] build-backend = ”setuptools.build_meta:__legacy__”
- Instead of setup.py you should use setup.cfg to describe your project instead.
The result is a project with far more explicit, readable, and dependable build information. For more details on how to use pyproject.toml to package your project, read ‘how to package Python projects in 2021‘.
Pip is the default package manager for Python, and is included with every version of Python you install. However, their a number of alternative package managers available for Python, including:
- Conda – the package manager for Anaconda’s Python ecosystem
- Poetry – a third-party package manager best known for its ability to perform dependency resolution – a feature that is still missing in pip.
- ActiveState Platform – a package management solution that includes a number of advanced package management features, including:
- Automated builds from source code, including linked C libraries
- Automated dependency resolution plus dependency conflict handling
- Vulnerability remediation
Read more about alternatives to pip in our Python Package Management Guide for Enterprise Developers white paper.
Setuptools is typically included within the Windows, macOS or Linux installer of most popular Python distributions. If not, you can install it by running:
pip install setuptools
sudo pip install setuptools
Recent versions of ActivePython include setuptools. Download your version of ActivePython for free.