Introduction: PyPI and friends

The Python ecosystem thrives on the sharing of open-source libraries and applications. Python software is commonly organized as projects that provide one or more packages. A package is a directory containing modules (Python files) and sub-packages.

There is an official and centralized package index called PyPI (Python Package Index). Anyone can upload projects to PyPI. While there are popular and reputable Python projects that serve as pillars of the ecosystem (such as NumPy), being on PyPI is not a guarantee of quality, reliability, or even good intentions.

Historically, Python packaging and dependency handling was problematic. One major issue was the tendency to install everything into the same Python environment and abuse PYTHONPATH. In this training, we present modern installation methods and contemporary Python packaging practices.

Different situations for package users¶

We will consider different methods adapted to various use cases:

• Installing Python applications

• Installing Python libraries for direct Python usage (for example in IPython or notebooks)

• Installing Python libraries for a Python script

• Installing Python libraries for package development. In this case, we can distinguish between different types of dependencies:

  • Build dependencies

  • Runtime dependencies

  • Development dependencies

Before presenting practical methods for each of these cases, we need to establish the context.

PyPI and different types of Python projects¶

The website pypi.org (PyPI) is the official package index for Python. It is also known as the “Cheese Shop”, in reference to a Monty Python sketch.

PyPI hosts a vast number of projects and handles a huge volume of requests. This infrastructure is provided by the PSF (Python Software Foundation).

Project maintainers typically upload two different types of distributions:

• The sdist (source distribution): an archive containing the project’s source code

• Wheels for different operating systems, architectures, and potentially Python versions. Wheels are archives containing compiled projects, potentially including binaries (Python extensions and executables) and Python code files. Wheels are platform-specific and ready to be installed.

The wheel format is quite versatile, allowing for different types of packages: applications or libraries.

Examples of wheel types¶

Pure Python library (e.g., requests):

requests-2.31.0-py3-none-any.whl

The py3-none-any tag indicates it works with any Python 3 version, any ABI, and any platform.

Pure Python application (e.g., pdm):

pdm-2.12.3-py3-none-any.whl

Similar naming convention, but contains console scripts/entry points for command-line usage.

Python wrapper of a native library (e.g., PySide6):

PySide6-6.6.1-cp39-abi3-manylinux_2_28_x86_64.whl
PySide6-6.6.1-cp39-abi3-macosx_11_0_universal2.whl
PySide6-6.6.1-cp39-abi3-win_amd64.whl

Python library with extensions (e.g., numpy):

numpy-1.26.3-cp312-cp312-manylinux_2_17_x86_64.whl
numpy-1.26.3-cp312-cp312-macosx_11_0_arm64.whl
numpy-1.26.3-cp312-cp312-win_amd64.whl

These contain compiled C extensions, so separate wheels are needed for each Python version and platform.

Pip, the official Python package installer¶

Pip is the official Python package installer, maintained by the Python Packaging Authority (PyPA). It is not designed to install non Python dependencies. pip installs by default from the Python Package Index (PyPI).

# Install a package in the current environment
pip install numpy

# List the packages installed in the current environment
pip list

If a compatible wheel is available, pip uses it. Otherwise, a wheel must be built locally, which can involve compilation (a potentially lengthy and resource-intensive process that requires compilers and libraries).

Virtual environments¶

For different projects, you may need to use different versions of the same package. Or you may be facing a situation where a package you want to use needs a specific version of another package (an older version than the one you already have installed). To get around potentially messy install, you should use virtual environments.

Virtual environments are central to modern Python installation workflows. A virtual environment is an isolated Python environment with its own installation directories that doesn’t share packages with other virtual environments or the system Python installation. This isolation prevents dependency conflicts between different projects.

The standard library includes a module for creating virtual environments:

python -m venv name-of-the-venv

The virtual environment can then be “activated”. On Unix systems:

. name-of-the-venv/bin/activate

On Windows (PowerShell):

name-of-the-venv\Scripts\Activate

Other tools for creating virtual environments¶

virtualenv: A more feature-rich alternative to the standard library venv module, offering faster environment creation and additional configuration options.

Next generation tools: Project managers like PDM, UV, Poetry, and Hatch integrate virtual environment management with dependency resolution and package building, providing a more comprehensive development workflow.

There are many tools available. Let’s explore why this diversity exists and how it’s possible.

PyPA¶

The Python Packaging Authority (PyPA) is a working group responsible for the evolution of the packaging ecosystem.

It hosts the development of packaging tools (Pip, Build, setuptools-scm, and others).

The PyPA oversees PEPs related to packaging.

The PyPA maintains an official guide at https://packaging.python.org about installing and creating Python packages.

Recent developments related to PEPs 517/518¶

The current state of Python packaging reflects significant changes introduced by two important PEPs: PEP 518 (pyproject.toml) and PEP 517 (frontends/backends and isolated builds).

PEP 518: pyproject.toml¶

PEP 518 introduced pyproject.toml as the standard way to describe Python projects and their build requirements. This file uses the TOML format and eliminates the need for setup.py in many cases, providing a cleaner and more declarative approach to project configuration.

PEP 517: Build frontends and backends¶

PEP 517 introduced a clear separation between:

• Frontends: build and installation tools used by developers and users (Pip, Build, PDM, UV, and others)

• Backends: build systems that create sdists and wheels (setuptools, Flit, Meson, pdm-backend, Hatchling, and others)

This separation enables isolated builds, where build dependencies are installed in a temporary environment, ensuring reproducible builds without polluting the user’s environment.

A diverse ecosystem of tools, both established and emerging¶

The standardization introduced by PEPs 517/518 has enabled a diverse ecosystem with many choices for PEP 517 frontends (Pip, Build, PDM, UV, and others) and PEP 517 backends (setuptools, Flit, Meson, pdm-backend, and others).

A new generation of tools has emerged: Python project managers (PDM, UV, Hatch, Poetry, and others) inspired by tooling from the NodeJS and Rust ecosystems. These tools integrate multiple aspects of the development workflow: environment management, dependency resolution, building, and publishing.

These modern tools introduce concepts such as declared requirements, freeze files, and lock files (results of dependency resolution). PEP 751 standardizes lock files with the pyproject.lock format.