How to download Python dependencies

how to download Python dependencies

Most of the time, you’ll use a package manager like pip or conda to install or download Python dependencies. The package manager will automatically download the dependency along with any requirements for you. As a result, in normal practice, you’ll never need to manually download a dependency.

However, there are a few instances in which you may choose to manually download your dependencies, such as:

  • The system you’re working on is off-line, air-gapped or has an unreliable connection.
  • Your organization wants to create a local repository of Python packages.
    • You might need to do this if the set of packages you work with need to be approved by Legal (ie., for licensing), or similar
  • Other reasons

If you choose to create your own repository, you’ll need to be able to download all the packages you require, along with their dependencies – all without installing them.

Note that in these cases, you may also need to download OS-specific dependencies if they are meant to be deployed on multiple operating systems.

Download Python Dependencies with Pip

The pip download command can be used to download packages and their dependencies to the current directory (by default), or else to a specified location without installing them.

Download Package and Dependencies

For example, to download the requests package and all its dependencies to the current directory without installing them, do the following:

(current directory) $ pip download requests

The output should look similar to:

    certifi-2020.4.5.1-py2.py3-none-any.whl
    chardet-3.0.4-py2.py3-none-any.whl
    idna-2.9-py2.py3-none-any.whl
    requests-2.23.0-py2.py3-none-any.whl
    urllib3-1.25.9-py2.py3-none-any.whl 

Download Dependencies Only

In some cases, you may want to download just the dependencies for a package, excluding the package itself. In this case, you have two options: 

  • Use the pipdeptree utility to gather a list of all dependencies, create a requirements.txt file listing all the dependencies, and then download them with the pip download command.
  • Get the list of dependencies for a package from the setup.py file

Get Dependencies with Pipdeptree

  1. Install pipdeptree:
a.$ pip install pipdeptree 

2. As an example, generate a dependency tree for requests

 $ pipdeptree -p requests 
requests==2.23.0
  - certifi [required: >=2017.4.17, installed: 2020.4.5.1]
  - chardet [required: >=3.0.2,<4, installed: 3.0.4]
  - idna [required: >=2.5,<3, installed: 2.9]
  - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.9]

3. Copy the dependencies and version information into a requirements.txt file:

  certifi>=2017.4.17
  chardet>=3.0.2,<4
  idna>=2.5,<3
  urllib3>=1.21.1,<1.26,!=1.25.1,!=1.25.0

4. You can now download the dependencies to the current directory without installing them:

(current directory) $ pip download -r requirements.txt

Get Dependencies from Setup.py

To get a list of dependencies from a setup.py file, do the following:

  1. Open the setup.py file and scroll to the install_requires section, which should look similar to:
setup(
...  
install_requires=[
'<depname>',
'<depname>'
] 
...)

2. Copy the dependencies listed in the install_requires section and paste them into a requirements.txt file.

3. Download the dependencies for the setuptools project:

(current directory) $ pip download -r requirements.txt -d <location>

For more information about setup.py files and the install_requires specification, refer to: How to use Python Dependency Management Tools

Download Python Dependencies for a Pipenv Project 

If you`re working with pipenv, the commands for downloading dependencies are slightly different. 

Download Pipenv Package and Dependencies

For example, to download the requests package and all its dependencies to the current directory without installing them, do the following:

(current directory) $ pipenv download requests

The output should look similar to:

    certifi-2020.4.5.1-py2.py3-none-any.whl
    chardet-3.0.4-py2.py3-none-any.whl
    idna-2.9-py2.py3-none-any.whl
    requests-2.23.0-py2.py3-none-any.whl
    urllib3-1.25.9-py2.py3-none-any.whl 

Download Pipenv Dependencies 

To download just the dependencies for a pipenv project to a specific location without installing them:

  1. Open the pipfile.lock file for your pipenv project
  2. Copy the dependencies listed in the requires section, and paste them into a requirements.txt file. For example:
  "_meta": {
        "hash": {
            "sha256": "<hashcode>"
        },
        "pipfile-spec": 6,
 "requires": {
            "<depname>": "version#"
            "<depname>": "version#"
  }
...}

3. To download the dependencies for the pipenv project to a specified location without installing them:

$ pip download -r requirements.txt -d <location>

Download Python Dependencies in a Conda Environment

Conda environments also require different commands to create a repository of dependencies.

Download Conda Package and Dependencies

For example, to download the requests package and all its dependencies to the current directory without installing them, do the following:

(current directory) $ conda install --download-only requests

Download Conda Dependencies Only

The simplest way to download just the dependencies for a package in a conda environment is to run the following command:

$ conda install  --download-only --only-deps requests

You can also download the dependencies for any package using the conda info command to first list all the dependencies for a specific package, and then copying those dependencies into a requirements.txt file. 

  1. For example, to view a list of dependencies for the requests package, enter:
$ conda info requests 
   requests 2.13.0 py36_0
   ----------------------
   file name   : requests-2.13.0-py36_0.tar.bz2
   name        : requests
   … 
   dependencies:
    certifi >=2017.4.17
    chardet >=3.0.2,<4
    idna >=2.5,<3
    python
    urllib3 >=1.21.1,<1.26,!=1.25.0,!=1.25.1

2. Copy the list of dependencies, including version information into a requirements.txt file.

3. Run the following command to download the dependencies without installing them:

$ conda install --download-only --file requirements.txt 

The dependencies will be downloaded to the \pkgs folder in the Anaconda\Conda root directory. The root directory in both Linux and Windows is the Anaconda\Conda installation directory.

Download Dependencies in a Poetry Environment 

Poetry environments download all dependencies from PyPI, so you can use pip to do the actual download for you. But if you want to list all the dependencies in your Poetry environment to ensure you capture them all, you’ll need to use specific Poetry commands and files.

Download Poetry Package and Dependencies

As mentioned, Poetry installs dependencies from PyPI, so you can use the pip command to download a package and all of its dependencies. For example, to download the requests package and all its dependencies to the current directory without installing them, do the following:

(current directory) $ pip download requests

The output should look similar to:

    certifi-2020.4.5.1-py2.py3-none-any.whl
    chardet-3.0.4-py2.py3-none-any.whl
    idna-2.9-py2.py3-none-any.whl
    requests-2.23.0-py2.py3-none-any.whl
    urllib3-1.25.9-py2.py3-none-any.whl 

Download Poetry Environment Dependencies Only

You can identify just the dependencies for a Poetry environment in one of two ways: 

  • View the pyproject.toml file`s dependencies section
  • Using poetry show command to list all dependencies in a visual tree

You can then copy these dependencies into a requirements.txt file and use the usual pip command to download them.

Get Dependencies with Pyproject.toml

Poetry environment dependencies are specified in pyproject.toml files under the [tool.poetry.dev-dependencies] section. For example:

[tool.poetry]
...
[tool.poetry.dev-dependencies]
pytest = "^5.2"
...

Copy the dependencies listed in the dev-dependencies section into a requirements.txt file. See step #3 below.

Get Dependencies with Poetry Show

Alternatively, you can use the poetry show command to list Poetry project dependencies in a visual tree that may be helpful in understanding the overall dependency structure. To use poetry show, do the following: 

  1. Cd into the project and enter:
(poetry project) $  poetry show -t 

The output should look similar to:

   pytest 5.4.2 pytest: simple powerful testing with Python
  |-- atomicwrites >=1.0
  |-- attrs >=17.4.0
  |-- colorama *
   ...

2. Copy the dependencies list into a requirements.txt file

3. Now that we have a requirements.txt file with our dependencies, we can download them. Note that because Poetry downloads packages from PyPI by default, you can actually use Pip to download the dependencies for your Poetry environment and save them to a specific location. To do so cd into your Poetry project, and enter:

(poetry project) $ pip download -r requirements.txt -d <location>

Auto-resolve dependencies with the ActiveState Platform

Dependencies. They’re the best, but also the worst. You can count on PyPI (the Python Package Index) or CPAN (the Comprehensive Perl Archive Network) to provide packages for so many things, from date and time handling to logging to web service clients. This can be a huge time saver, but dependency resolution can also be a huge time sink.

Dependency resolution is at the core of the ActiveState Platform. When you create a project and start adding requirements, we tell you what dependencies those requirements have. Sometimes we need to tell you that your requirements are impossible because of dependency conflicts.

The ActiveState Platform aims to handle every dependency for every language. That means handling libraries down to the C/C++ level, external tools, and all the conditional dependencies that exist. To take things even further, our ultimate goal is to support multi-language projects. That means that you can create a project using both Python and Perl packages, and we’ll make sure that both languages are using the same (up to date) OpenSSL version.

Need to build a runtime environment for your Python project? The ActiveState Platform gives you access to a pre-configured build environment:

  • Eliminate Dependency Hell – dependencies are automatically resolved for you, where possible
  • Automated Packaging – minimal language/OS expertise required
  • No Tooling – no compiler to source/set up
  • Automated Setup – install your runtime into a virtual environment with a single command using the State Tool.

Watch how the platform helps developers kick-off their Python projects faster

Remi M