The standard library

The standard library#

Todo

Check standard library

  • Small examples from (or just mention) sys, os/shutils, copy, pathlib/glob argparse, math, re, dataclasses, itertools, turtle, functools, unittest, collections, datetime

  • https://docs.python.org/3/tutorial/stdlib.html

  • Mention good alternatives outside of the standard library (pytest, click, requests/urllib3, typing-extensions, python-dateutil/pendulum, fsspec, certifi, idna, …)

The standard library + presentation of few very common packages#

The Python standard library (see also this tuto) is a quite impressive set of packages useful for many many things. These packages are included natively in Python. They are very stable (difficult to find bugs). Here is a small list:

  • math - Mathematical functions

  • sys - System-specific parameters and functions (a lot about the Python system)

  • copy - Shallow and deep copy operations

  • os - Miscellaneous operating system interfaces

  • glob - Unix style pathname pattern expansion (like ls)

  • shutil - High-level file operations

  • pdb - activate the python debugger

  • subprocess

  • datetime

  • pickle - Python object serialization

  • re - Regular expressions

  • argparse - Parser for command-line options, arguments and sub-commands

  • unittest - Unit testing framework

  • logging - Event logging system

  • platform - Access to underlying platform’s identifying data

  • threading - Higher-level threading interface

  • multiprocessing - Process-based “threading” interface

math - Mathematical functions#

For example to use \(\pi\) in an environment where Numpy might not be installed:

import math

print(type(math))

<class 'module'>

from math import cos

print("pi is approximately equal to     ", math.pi)
print("cos(pi) is approximately equal to", cos(math.pi))

pi is approximately equal to      3.141592653589793
cos(pi) is approximately equal to -1.0

pprint - Pretty-print lists, tuples, & dictionaries recursively#

from pprint import pprint

sys - System-specific parameters and functions (a lot about the Python system)#

If you want to know where Python looks for module during the import statements, you can do

import sys

pprint(sys.path)

['/usr/local/lib/python313.zip',
 '/usr/local/lib/python3.13',
 '/usr/local/lib/python3.13/lib-dynload',
 '',
 '/builds/py-edu-fr/py-edu-fr/books/science/.venv/lib/python3.13/site-packages']

os: Miscellaneous operating system interfaces#

os is a very important module.

import os

os.getcwd()

'/builds/py-edu-fr/py-edu-fr/src/en/structure-reuse-code'

There is in particular the os.path module, which you use each time you work with paths towards files and directories. It can be used to build paths in the most robust manner:

# Building a path to a file to read...
directory_path = "./files/"
file_name = "file_to_read.txt"
# String concatenation works but is not very robust
full_path = directory_path + file_name
print(full_path)
# Better to do
full_path = os.path.join(directory_path, file_name)
print(full_path)

./files/file_to_read.txt
./files/file_to_read.txt

For example, we can create the string for a new path in a cross-platform way like this

# Method to get cross-platform home directory ($HOME)
home_dir = os.path.expanduser("~")
os.path.join(home_dir, "opt", "miniconda3", "lib/python3.6")

'/home/appuser/opt/miniconda3/lib/python3.6'

To make a new directory if it does not exist:

path_tmp = "/tmp/tmp_dir_example_python"

import shutil
shutil.rmtree(path_tmp, ignore_errors=True)

if not os.path.exists(path_tmp):
    os.mkdir(path_tmp)
pprint(os.listdir("../common/examples/"))

['example0', 'example1', 'helloworld.py', 'wrong.py']

To scan the content of a directory:

def list_dir_files():
    for base, path_dir, path_files in os.walk("../common/examples"):
        if base.startswith("__"):
            continue
        print(
            (
                f"In the directory {base}:\n"
                f"\tdirectories: {path_dir}\n\tfiles {path_files}."
            )
        )


list_dir_files()
print(os.path.exists(path_tmp))
os.rmdir(path_tmp)
print(os.path.exists(path_tmp))
list_dir_files()

In the directory ../common/examples:
	directories: ['example0', 'example1']
	files ['helloworld.py', 'wrong.py'].
In the directory ../common/examples/example0:
	directories: ['__pycache__']
	files ['prog.py', 'util.py'].
In the directory ../common/examples/example0/__pycache__:
	directories: []
	files ['util.cpython-313.pyc'].
In the directory ../common/examples/example1:
	directories: []
	files ['prog.py', 'util.py'].
True
False
In the directory ../common/examples:
	directories: ['example0', 'example1']
	files ['helloworld.py', 'wrong.py'].
In the directory ../common/examples/example0:
	directories: ['__pycache__']
	files ['prog.py', 'util.py'].
In the directory ../common/examples/example0/__pycache__:
	directories: []
	files ['util.cpython-313.pyc'].
In the directory ../common/examples/example1:
	directories: []
	files ['prog.py', 'util.py'].

Other handy functions of os.path:

  • os.path.basename: returns the basename of a path (last member of a path)

  • os.path.isfile: returns True if the path points to a file

  • …

See https://docs.python.org/3.7/library/os.path.html

glob - Unix style pathname pattern expansion#

The equivalent of the Unix “ls” is in the glob module:

from glob import glob

l = glob("*")
print("list unsorted:", l)
print("list sorted:  ", sorted(l))

list unsorted: ['index.md', 'classes.md', 'functions-again.md', 'functions.md', 'tests.md', 'modules-packages.md', 'standard-library.md']
list sorted:   ['classes.md', 'functions-again.md', 'functions.md', 'index.md', 'modules-packages.md', 'standard-library.md', 'tests.md']

pathlib: Object-oriented filesystem paths#

A modern (Python 3) and nicer method to manipulate file paths.

from pathlib import Path

path_tmp = Path("/tmp/tmp_dir_example_python")
print(path_tmp.exists())
path_tmp.mkdir(exist_ok=True)

for i in range(4):
    (path_tmp / f"tmp{i}.xml").touch()
    (path_tmp / f"tmp{i}.txt").touch()

pprint(sorted(path_tmp.glob("*.txt")))

False
[PosixPath('/tmp/tmp_dir_example_python/tmp0.txt'),
 PosixPath('/tmp/tmp_dir_example_python/tmp1.txt'),
 PosixPath('/tmp/tmp_dir_example_python/tmp2.txt'),
 PosixPath('/tmp/tmp_dir_example_python/tmp3.txt')]

Note

Code using libpath.Path is often nicer than the equivalent using os.path and glob.

shutil - High-level file operations#

Copy of files and directories can be done with shutil, in particular with shutil.copytree.

pdb: useful to debug code#

On a script:

  1. import pdb

  2. write pdb.set_trace() to set up a breakpoint

  3. run the script

At execution time, the script will stop at the first line containing pdb.set_trace() and gives the user access to the interpreter.

Remarks:

  • even nicer: ipdb (but not part of the standard library).

  • even nicer: breakpoint() built-in function in Python 3.7.

subprocess#

subprocess is very important since it is the simple way to launch other programs and bash commands from Python. For example, in order to run bash (and not sh) commands, you can do

import subprocess


def call_bash(commands):
    return subprocess.call(["/bin/bash", "-c", commands])


ret = call_bash(
    """
echo Hello; ls /tmp
"""
)
if ret == 0:
    print("command succeed")
else:
    print(f"command failed with return code {ret}")

Hello
tmp_dir_example_python
tmpngxtwri7.json
uv-f2bf9ef577c418d2.lock
zoo.txt
command succeed

argparse - Parser for command-line options, arguments and sub-commands#

argparse is the right tool to develop a command line script with options and help. Example from the tutorial at https://docs.python.org/3/howto/argparse.html :

# File prog.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo", help="echo the string you use here")
args = parser.parse_args()
print(args.echo)

Usage :#

$ python3 prog.py
usage: prog.py [-h] echo
prog.py: error: the following arguments are required: echo
$ python3 prog.py --help
usage: prog.py [-h] echo

positional arguments:
  echo        echo the string you use here

optional arguments:
  -h, --help  show this help message and exit
$ python3 prog.py foo
foo

logging - Event logging system#

logging allows the programmer to print (or not) different levels of messages.

import logging

log_level = logging.INFO  # to get information messages
# log_level = logging.WARNING  # no information messages
logging.basicConfig(
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", level=log_level
)

thing = "beer"
logging.info('Would you like to have a "%s"?', thing)

2025-10-29 22:48:49,852 - root - INFO - Would you like to have a "beer"?
Contents