Using Python's pip to Manage Your Projects' Dependencies

Finding pip on Your System

The Python 3 installer gives you the option to install pip when installing Python on your system. In fact, the option to install pip with Python is checked by default, so pip should be ready for you to use after installing Python.

You can verify that pip is available by looking for the pip3 executable on your system. Select your operating system below and use your platform-specific command accordingly:

• Windows
• Linux + macOS

C:\> where pip3

$ which pip3

On Windows and Unix systems, pip3 may be found in more than one location. This can happen when you have multiple Python versions installed. If you can’t find pip in any location on your system, then you may consider reinstalling pip.

Instead of running your system pip directly, you can also run it as a Python module. In the next section, you’ll learn how.

Running pip as a Module

When you run your system pip directly, the command itself doesn’t reveal which Python version pip belongs to. This unfortunately means that you could use pip to install a package into the site-packages of an old Python version without noticing. To prevent this from happening, you can run pip as a Python module:

$ python3 -m pip

Notice that you use python3 -m to run pip. The -m switch tells Python to run a module as an executable of the python3 interpreter. This way, you can ensure that your system default Python 3 version runs the pip command. If you want to learn more about this way of running pip, then you can read Brett Cannon’s insightful article about the advantages of using python3 -m pip.

Sometimes you may want to be more explicit and limit packages to a specific project. In situations like this, you should run pip inside a virtual environment.

Using pip in a Python Virtual Environment

To avoid installing packages directly into your system Python installation, you can use a virtual environment. A virtual environment provides an isolated Python interpreter for your project. Any packages that you use inside this environment will be independent of your system interpreter. This means that you can keep your project’s dependencies separate from other projects and the system at large.

Using pip inside a virtual environment has three main advantages. You can:

1. Be sure that you’re using the right Python version for the project at hand
2. Be confident that you’re referring to the correct pip instance when running pip or pip3
3. Use a specific package version for your project without affecting other projects

Python 3 has the built-in venv module for creating virtual environments. This module helps you create virtual environments with an isolated Python installation. Once you’ve activated the virtual environment, then you can install packages into this environment. The packages that you install into one virtual environment are isolated from all other environments on your system.

You can follow these steps to create a virtual environment and verify that you’re using the pip module inside the newly created environment:

• Windows
• Linux + macOS

C:\> python -m venv venv
C:\> venv\Scripts\activate.bat
(venv) C:\>  pip3 --version
pip 21.2.3 from ...\lib\site-packages\pip (python 3.10)
(venv) C:\>  pip --version
pip 21.2.3 from ...\lib\site-packages\pip (python 3.10)

$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip3 --version
pip 21.2.3 from .../python3.10/site-packages/pip (python 3.10)
(venv) $ pip --version
pip 21.2.3 from .../python3.10/site-packages/pip (python 3.10)

Here you create a virtual environment named venv by using Python’s built-in venv module. Then you activate it with the source command. The parentheses (()) surrounding your venv name indicate that you successfully activated the virtual environment.

Finally, you check the version of the pip3 and pip executables inside your activated virtual environment. Both point to the same pip module, so once your virtual environment is activated, you can use either pip or pip3.

Reinstalling pip When Errors Occur

When you run the pip command, you may get an error in some cases. Your specific error message will depend on your operating system:

Error messages like these indicate that something went wrong with the installation of pip.

Getting errors like the ones shown above can be frustrating because pip is vital for installing and managing external packages. Some common problems with pip are related to how this tool was installed on your system.

Although the error messages for various systems differ, they all point to the same problem: Your system can’t find pip in the locations listed in your PATH variable. On Windows, PATH is part of the system variables. On macOS and Linux, PATH is part of the environment variables. You can check the contents of your PATH variable with this command:

• Windows
• Linux + macOS

C:\> echo %PATH%

$ echo $PATH

The output of this command will show a list of locations (directories) on your disk where the operating system looks for executable programs. Depending on your system, locations can be separated by a colon (:) or a semicolon (;).

By default, the directory that contains the pip executable should be present in PATH after you install Python or create a virtual environment. However, missing pip is a common issue. Two supported methods can help you install pip again and add it to your PATH:

1. The ensurepip module
2. The get-pip.py script

The ensurepip module has been part of the standard library since Python 3.4. It was added to provide a straightforward way for you to reinstall pip if, for example, you skipped it when installing Python or you uninstalled pip at some point. Select your operating system below and run ensurepip accordingly:

• Windows
• Linux + macOS

C:\> python -m ensurepip --upgrade

$ python3 -m ensurepip --upgrade

If pip isn’t installed yet, then this command installs it in your current Python environment. If you’re in an active virtual environment, then the command installs pip into that environment. Otherwise, it installs pip globally on your system. The --upgrade option ensures that the pip version is the same as the one declared in ensurepip.

Another way to fix your pip installation is to use the get-pip.py script. The get-pip.py file contains a full copy of pip as an encoded ZIP file. You can download get-pip.py directly from the PyPA bootstrap page. Once you have the script on your machine, then you run the Python script like this:

• Windows
• Linux + macOS

C:\> python get-pip.py

$ python3 get-pip.py

This script will install the latest version of pip, setuptools, and wheel in your current Python environment. If you only want to install pip, then you can add the --no-setuptools and --no-wheel options to your command.

If none of the methods above work, then it might be worth trying to download the latest Python version for your current platform. You can follow the Python 3 Installation & Setup Guide to make sure that pip is appropriately installed and works without errors.

Using the Python Package Index (PyPI)

One of the many packages that PyPI hosts is called requests. The requests library helps you to interact with web services by abstracting the complexities of HTTP requests. You can learn all about requests on its official documentation site.

When you want to use the requests package in your project, you must first install it into your environment. If you don’t want to install it in your system Python site-packages, then you can create a virtual environment first, as shown above.

Once you’ve created the virtual environment and activated it, then your command-line prompt shows the name of the virtual environment inside the parentheses. Any pip commands that you perform from now on will happen inside your virtual environment.

To install packages, pip provides an install command. You can run it to install the requests package:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install requests

(venv) $ python3 -m pip install requests

In this example, you run pip with the install command followed by the name of the package that you want to install. The pip command looks for the package in PyPI, resolves its dependencies, and installs everything in your current Python environment to ensure that requests will work.

The pip install <package> command always looks for the latest version of the package and installs it. It also searches for dependencies listed in the package metadata and installs them to ensure that the package has all the requirements that it needs.

It’s also possible to install multiple packages in a single command:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install rptree codetiming

(venv) $ python3 -m pip install rptree codetiming

By chaining the packages rptree and codetiming in the pip install command, you install both packages at once. You can add as many packages as you want to the pip install command. In cases like this, a requirements.txt file can come in handy. Later in this tutorial, you’ll learn how to use a requirements.txt file to install many packages at once.

You can use the list command to display the packages installed in your environment, along with their version numbers:

• Windows
• Linux + macOS

(venv) C:\> python -m pip list
Package            Version
------------------ ---------
certifi            x.y.z
charset-normalizer x.y.z
codetiming         x.y.z
idna               x.y.z
pip                x.y.z
requests           x.y.z
rptree             x.y.z
setuptools         x.y.z
urllib3            x.y.z

(venv) $ python3 -m pip list
Package            Version
------------------ ---------
certifi            x.y.z
charset-normalizer x.y.z
idna               x.y.z
pip                x.y.z
requests           x.y.z
setuptools         x.y.z
urllib3            x.y.z

The pip list command renders a table that shows all installed packages in your current environment. The output above shows the version of the packages using an x.y.z placeholder format. When you run the pip list command in your environment, pip displays the specific version number that you’ve installed for each package.

To get more information about a specific package, you can look at the package’s metadata by using the show command in pip:

• Windows
• Linux + macOS

(venv) C:\> python -m pip show requests
Name: requests
Version: x.y.z
Summary: Python HTTP for Humans.
    ...
Requires: certifi, idna, charset-normalizer, urllib3
Required-by:

(venv) $ python3 -m pip show requests
Name: requests
Version: x.y.z
Summary: Python HTTP for Humans.
    ...
Requires: certifi, idna, charset-normalizer, urllib3
Required-by:

The output of this command on your system will list the package’s metadata. The Requires line lists packages, such as certifi, idna, charset-normalizer, and urllib3. These were installed because requests depends on them to work correctly.

Now that you’ve installed requests and its dependencies, you can import it just like any other regular package in your Python code. Start the interactive Python interpreter and import the requests package:

>>> import requests
>>> requests.__version__
"x.y.z"

After starting the interactive Python interpreter, you imported the requests module. By calling requests.__version__, you verified that you were using the requests module within your virtual environment.

Using a Custom Package Index

By default, pip uses PyPI to look for packages. But pip also gives you the option to define a custom package index.

Using pip with a custom index can be helpful when the PyPI domain is blocked on your network or if you want to work with packages that aren’t publicly available. Sometimes system administrators also create their own internal package index to better control which package versions are available to pip users on the company’s network.

A custom package index must comply with PEP 503 – Simple Repository API to work with pip. You can get an impression of how such an API (Application Programming Interface) looks by visiting the PyPI Simple Index—but be aware that this is a large page with a lot of hard-to-parse content. Any custom index that follows the same API can be targeted with the --index-url option. Instead of typing --index-url, you can also use the -i shorthand.

For example, to install the rptree tool from the TestPyPI package index, you can run the following command:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install -i https://test.pypi.org/simple/ rptree

(venv) $ python3 -m pip install -i https://test.pypi.org/simple/ rptree

With the -i option, you tell pip to look at a different package index instead of PyPI, the default one. Here, you’re installing rptree from TestPyPI rather than from PyPI. You can use TestPyPI to fine-tune the publishing process for your Python packages without cluttering the production package index on PyPI.

If you need to use an alternative index permanently, then you can set the index-url option in the pip configuration file. This file is called pip.conf, and you can find its location by running the following command:

• Windows
• Linux + macOS

(venv) C:\> python -m pip config list -vv

(venv) $ python3 -m pip config list -vv

With the pip config list command, you can list the active configuration. This command only outputs something when you have custom configurations set. Otherwise, the output is empty. That’s when the additive --verbose, or -vv, option can be helpful. When you add -vv, pip shows you where it looks for the different configuration levels.

If you want to add a pip.conf file, then you can choose one of the locations that pip config list -vv listed. A pip.conf file with a custom package index looks like this:

# pip.conf

[global]
index-url = https://test.pypi.org/simple/

When you have a pip.conf file like this, pip will use the defined index-url to look for packages. With this configuration, you don’t need to use the --index-url option in your pip install command to specify that you only want packages that can be found in the Simple API of TestPyPI.

Installing Packages From Your GitHub Repositories

You’re not limited to packages hosted on PyPI or other package indexes. pip also provides the option to install packages from a GitHub repository. But even when a package is hosted on PyPI, like the Real Python directory tree generator, you can opt to install it from its Git repository:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install git+https://github.com/realpython/rptree

(venv) $ python3 -m pip install git+https://github.com/realpython/rptree

With the git+https scheme, you can point to a Git repository that contains an installable package. You can verify that you installed the package correctly by running an interactive Python interpreter and importing rptree:

>>> import rptree
>>> rptree.__version__
"x.y.z"

After starting the interactive Python interpreter, you import the rptree module. By calling rptree.__version__, you verify that you’re using the rptree module that’s based in your virtual environment.

Installing packages from a Git repository can be helpful if the package isn’t hosted on PyPI but has a remote Git repository. The remote repository you point pip to can even be hosted on an internal Git server on your company’s intranet. This can be useful when you’re behind a firewall or have other restrictions for your Python projects.

Installing Packages in Editable Mode to Ease Development

When working on your own package, installing it in an editable mode can make sense. By doing this, you can work on the source code while still using your command line like you would in any other package. A typical workflow is to first clone the repository and then use pip to install it as an editable package in your environment:

• Windows
• Linux + macOS

 1C:\> git clone https://github.com/realpython/rptree
 2C:\> cd rptree
 3C:\rptree> python3 -m venv venv
 4C:\rptree> venv\Scripts\activate.bat
 5(venv) C:\rptree> python -m pip install -e .

 1$ git clone https://github.com/realpython/rptree
 2$ cd rptree
 3$ python3 -m venv venv
 4$ source venv/bin/activate
 5(venv) $ python3 -m pip install -e .

With the commands above, you installed the rptree package as an editable module. Here’s a step-by-step breakdown of the actions you just performed:

1. Line 1 cloned the Git repository of the rptree package.
2. Line 2 changed the working directory to rptree/.
3. Lines 3 and 4 created and activated a virtual environment.
4. Line 5 installed the content of the current directory as an editable package.

The -e option is shorthand for the --editable option. When you use the -e option with pip install, you tell pip that you want to install the package in editable mode. Instead of using a package name, you use a dot (.) to point pip to the current directory.

If you hadn’t used the -e flag, pip would’ve installed the package normally into your environment’s site-packages/ folder. When you install a package in editable mode, you’re creating a link in the site-packages to the local project path:

 ~/rptree/venv/lib/python3.10/site-packages/rptree.egg-link

Using the pip install command with the -e flag is just one of many options that pip install offers. You can check out pip install examples in the pip documentation. There you’ll learn how to install specific versions of a package or point pip to a different index that’s not PyPI.

In the next section, you’ll learn how requirements files can help with your pip workflows.

Pinning Requirements

When you share your Python project with other developers, you may want them to use the same versions of external packages that you’re using. Maybe a specific version of a package contains a new feature that you rely on, or the version of a package that you’re using is incompatible with former versions.

These external dependencies are also called requirements. You’ll often find Python projects that pin their requirements in a file called requirements.txt or similar. The requirements file format allows you to specify precisely which packages and versions should be installed.

Running pip help shows that there’s a freeze command that outputs the installed packages in requirements format. You can use this command, redirecting the output to a file to generate a requirements file:

• Windows
• Linux + macOS

(venv) C:\> python -m pip freeze > requirements.txt

(venv) $ python3 -m pip freeze > requirements.txt

This command creates a requirements.txt file in your working directory with the following content:

certifi==x.y.z
charset-normalizer==x.y.z
idna==x.y.z
requests==x.y.z
urllib3==x.y.z

Remember that x.y.z displayed above is a placeholder format for the package versions. Your requirements.txt file will contain real version numbers.

The freeze command dumps the name and version of the currently installed packages to standard output. You can redirect the output to a file that you can later use to install your exact requirements into another system. You can name the requirements file whatever you want. However, a widely adopted convention is to name it requirements.txt.

When you want to replicate the environment in another system, you can run pip install, using the -r switch to specify the requirements file:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install -r requirements.txt

(venv) $ python3 -m pip install -r requirements.txt

In the command above, you tell pip to install the packages listed in requirements.txt into your current environment. The package versions will match the version constraints that the requirements.txt file contains. You can run pip list to display the packages you just installed, with their version numbers:

• Windows
• Linux + macOS

(venv) C:\> python -m pip list

Package            Version
------------------ ---------
certifi            x.y.z
charset-normalizer x.y.z
idna               x.y.z
pip                x.y.z
requests           x.y.z
setuptools         x.y.z
urllib3            x.y.z

(venv) $ python3 -m pip list

Package            Version
------------------ ---------
certifi            x.y.z
charset-normalizer x.y.z
idna               x.y.z
pip                x.y.z
requests           x.y.z
setuptools         x.y.z
urllib3            x.y.z

Now you’re ready to share your project! You can submit requirements.txt into a version control system like Git and use it to replicate the same environment on other machines. But wait, what happens if new updates are released for these packages?

Fine-Tuning Requirements

The problem with hardcoding your packages’ versions and dependencies is that packages are updated frequently with bug and security fixes. You probably want to leverage those updates as soon as they’re published.

The requirements file format allows you to specify dependency versions using comparison operators that give you some flexibility to ensure packages are updated while still defining the base version of a package.

Open requirements.txt in your favorite editor and turn the equality operators (==) into greater than or equal to operators (>=), like in the example below:

# requirements.txt

certifi>=x.y.z
charset-normalizer>=x.y.z
idna>=x.y.z
requests>=x.y.z
urllib3>=x.y.z

You can change the comparison operator to >= to tell pip to install an exact or greater version that has been published. When you set a new environment by using the requirements.txt file, pip looks for the latest version that satisfies the requirement and installs it.

Next, you can upgrade the packages in your requirements file by running the install command with the --upgrade switch or the -U shorthand:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install -U -r requirements.txt

(venv) $ python3 -m pip install -U -r requirements.txt

If a new version is available for a listed package, then the package will be upgraded.

In an ideal world, new versions of packages would be backward compatible and would never introduce new bugs. Unfortunately, new versions can introduce changes that’ll break your application. To fine-tune your requirements, the requirements file syntax supports additional version specifiers.

Imagine that a new version, 3.0, of requests is published but introduces an incompatible change that breaks your application. You can modify the requirements file to prevent 3.0 or higher from being installed:

# requirements.txt

certifi==x.y.z
charset-normalizer==x.y.z
idna==x.y.z
requests>=x.y.z, <3.0
urllib3==x.y.z

Changing the version specifier for the requests package ensures that any version greater than or equal to 3.0 doesn’t get installed. The pip documentation provides extensive information about the requirements file format, and you can consult it to learn more.

Separating Production and Development Dependencies

Not all packages that you install during the development of your applications will be production dependencies. For example, you’ll probably want to test your application, so you need a test framework. A popular framework for testing is pytest. You want to install it in your development environment, but you don’t want it in your production environment, because it isn’t a production dependency.

You create a second requirements file, requirements_dev.txt, to list additional tools to set up a development environment:

# requirements_dev.txt

pytest>=x.y.z

Having two requirements files will demand that you use pip to install both of them, requirements.txt and requirements_dev.txt. Fortunately, pip allows you to specify additional parameters within a requirements file, so you can modify requirements_dev.txt to also install the requirements from the production requirements.txt file:

# requirements_dev.txt

-r requirements.txt
pytest>=x.y.z

Notice that you use the same -r switch to install the production requirements.txt file. Now, in your development environment, you only have to run this single command to install all requirements:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install -r requirements_dev.txt

(venv) $ python3 -m pip install -r requirements_dev.txt

Because requirements_dev.txt contains the -r requirements.txt line, you’ll install not only pytest but also the pinned requirements of requirements.txt. In a production environment, it’s sufficient to install the production requirements only:

• Windows
• Linux + macOS

(venv) C:\> python -m pip install -r requirements.txt

(venv) $ python3 -m pip install -r requirements.txt

With this command, you install the requirements listed in requirements.txt. In contrast to your development environment, your production environment won’t have pytest installed.

Freezing Requirements for Production

You created the production and development requirement files and added them to source control. These files use flexible version specifiers to ensure that you leverage bug fixes published by your dependencies. You’ve also tested your application and are now ready to deploy it to production.

You know that all the tests pass and the application works with the dependencies that you used in your development process, so you probably want to ensure that you deploy identical versions of dependencies to production.

The current version specifiers don’t guarantee that the identical versions will be deployed to production, so you want to freeze the production requirements before releasing your project.

After you’ve finished development with your current requirements, a workflow to create a new release of your current project can look like this:

With a workflow like this, the requirements_lock.txt file will contain exact version specifiers and can be used to replicate your environment. You’ve ensured that when your users install the packages listed in requirements_lock.txt into their own environments, they’ll be using the versions that you intend them to use.

Freezing your requirements is an important step to ensure that your Python project works the same way for your users in their environments as it did in yours.