Python's __all__: Packages, Modules, and Wildcard Imports

Explicit Imports

In Python, when you need to get a specific object from a module or a particular module from a package, you can use an explicit import statement. This type of statement allows you to bring the target object to your current namespace so that you can use the object in your code.

To import a module by its name, you can use the following syntax:

import module [as name]

This statement allows you to import a module by its name. The module must be listed in Python’s import path, which is a list of locations where the path based finder searches when you run an import.

The part of the syntax that’s enclosed in square brackets is optional and allows you to create an alias of the imported name. This practice can help you avoid name collisions in your code.

As an example, say that you have the following module:

def add(a, b):
    return float(a + b)

def subtract(a, b):
    return float(a - b)

def multiply(a, b):
    return float(a * b)

def divide(a, b):
    return float(a / b)

This sample module provides functions that allow you to perform basic calculations. The containing module is called calculations.py. To import this module and use the functions in your code, go ahead and start a REPL session in the same directory where you saved the file.

Then run the following code:

>>> import calculations

>>> calculations.add(2, 4)
6.0
>>> calculations.subtract(8, 4)
4.0
>>> calculations.multiply(5, 2)
10.0
>>> calculations.divide(12, 2)
6.0

The import statement at the beginning of this code snippet brings the module name to your current namespace. To use the functions or any other object from calculations, you need to use fully qualified names with the dot notation.

import calculations as calc

import numpy as np
import pandas as pd

You can also use a similar syntax to import a Python package:

import package [as name]

In this case, Python loads the content of your package’s __init__.py file into your current namespace. If that file exports objects, then those objects will be available to you.

Finally, if you want to be more specific in what you import into your current namespace, then you can use the following syntax:

from module import name [as name]

With this import statement, you can import specific names from a given module. This approach is recommended when you only need a few names from a long module that defines many objects or when you don’t expect name collisions in your code.

To continue with the calculations module, you can import the needed function only:

>>> from calculations import add

>>> add(2, 4)
6.0

In this example, you only use the add() function. The from module import name syntax lets you import the target name explicitly. In this case, the rest of the functions and the module itself won’t be accessible in your namespace or scope.

Wildcard Imports on Modules

When you’re working with Python modules, a wildcard import is a type of import that allows you to get all the public names from a module in one go. This type of import has the following syntax:

from module import *

The name wildcard import derives from the asterisk at the end of the statement, which denotes that you want to import all the objects from module.

Go back to your terminal window and restart your REPL session. Then, run the following code:

>>> from calculations import *

>>> dir()
[
    ...
    'add',
    'divide',
    'multiply',
    'subtract'
]

In this code snippet, you first run a wildcard import. This import makes available all the names from the calculations modules and brings them to your current namespace. The built-in dir() function allows you to see what names are available in the current namespace. As you can confirm from the output, all the functions that live in calculations are now available.

When you’re completely sure that you need all the objects that a given module defines, using a wildcard import is a quick solution. In practice, this situation is rare, and you just end up cluttering your namespace with unneeded objects and names.

Using wildcard imports is explicitly discouraged in PEP 8 when they say:

Wildcard imports (from <module> import *) should be avoided, as they make it unclear which names are present in the namespace, confusing both readers and many automated tools. There is one defensible use case for a wildcard import, which is to republish an internal interface as part of a public API (for example, overwriting a pure Python implementation of an interface with the definitions from an optional accelerator module and exactly which definitions will be overwritten isn’t known in advance). (Source)

The main drawback of wildcard import is that you don’t have control over the imported objects. You can’t be specific. Therefore, you can confuse the users of your code and clutter their namespace with unnecessary objects.

Even though wildcard imports are discouraged, some libraries and tools use them. For example, if you search for applications built with Tkinter, then you’ll realize that many of the examples use the form:

from tkinter import *

This import gives you access to all the objects defined in the tkinter module, which is pretty convenient if you’re starting to learn how to use this tool.

You may find many other tools and third-party libraries that use wildcard imports for code examples in their documentation, and that’s okay. However, in real-world projects, you should avoid this type of import.

In practice, you can’t control how the users of your code will manage their imports. So, you better prepare your code for wildcard imports. You’ll learn how to do this in the upcoming sections. First, you’ll learn about using wildcard imports on packages.

Wildcard Import and Non-Public Names

Python has a well-established naming convention that allows you to tell the users of your code when a given name in a module is for internal or external use.

If an object’s name starts with a single leading underscore, then that name is considered non-public, so it’s for internal use only. In contrast, if a name starts with a lowercase or uppercase letter, then that name is public and, therefore, is part of the module’s public API.

When you have non-public names in a given module, you should know that wildcard imports won’t import those names. Say that you have the following module:

from math import pi as _pi

class Circle:
    def __init__(self, radius):
        self.radius = _validate(radius)

    def area(self):
        return _pi * self.radius**2

class Square:
    def __init__(self, side):
        self.side = _validate(side)

    def area(self):
        return self.side**2

def _validate(value):
    if not isinstance(value, int | float) or value <= 0:
        raise ValueError("positive number expected")
    return value

In this module, you have two non-public objects _pi and _validate(). You know this because they have a leading underscore in their names. If someone runs a wildcard import on this module, then the non-public names won’t be imported:

>>> from shapes import *

>>> dir()
[
    'Circle',
    'Square',
    ...
]

If you take a look at the output of dir(), then you’ll note that only the Circle and Square classes are available in your current namespace. The non-public objects, _pi and _validate(), aren’t available. So, wildcard imports won’t import non-public names.

Wildcard Import on Packages

Up to this point, you know how wildcard imports work with modules. You can also use this type of import with packages. In that case, the syntax is the same, but you need to use a package name rather than a module name:

from package import *

Now, what happens when you run this type of import? You may expect that this import causes Python to search the file system, find the modules and subpackages that are present in the package, and import them.

However, doing this file system search could take a long time. Additionally, importing modules might have unwanted side effects, because when you import a module, all the executable code in that module runs.

Because of these potential issues, Python has the __all__ special variable, which will allow you to explicitly define the list of modules that you want to expose to wildcard import in a given package. You’ll explore the details in the next section.

Names From a Module

You can use the __all__ variable to explicitly control what names a module exposes to wildcard imports. In this sense, __all__ allows you to establish a module’s public interface or API. This technique is also a way to explicitly communicate what the module’s API is.

If you have a large module with many public names, then you can use __all__ to create a list of exportable names so that wildcard imports don’t pollute the namespace of your code’s users.

In general, modules can have a few different types of names:

• Public names are part of the module’s public interface.
• Non-public names are for internal use only.
• Imported names are names that the module imports as public or non-public names.

As you already know, public names are those that start with a lowercase or uppercase letter. Non-public names are those that start with a single leading underscore.

Finally, imported names are those that you import as public names in a module. These names are also exported from that module. So, that’s why you’ll see imports like the following in many codebases:

import sys as _sys

In this example, you import the sys module as _sys. The as specifier lets you create an alias for the imported object. In this case, the alias is a non-public name. With this tiny addition to your import statement, you prevent sys from being exported when someone uses a wildcard import on the module.

So, if you don’t want to export imported objects from a module, then use the as specifier and a non-public alias for the imported objects.

Ideally, the __all__ list should only contain public names that are defined in the containing module. As an example, say that you have the following module containing functions and classes that allow you to make HTTP requests:

import requests

__all__ = ["get_page_content", "WebPage"]

BASE_URL = "http://example.com"

def get_page_content(page):
    return _fetch_page(page).text

def _fetch_page(page):
    url = f"{BASE_URL}/{page}"
    return requests.get(url)

class WebPage:
    def __init__(self, page):
        self.response = _fetch_page(page)

    def get_content(self):
        return self.response.text

In this sample module, you import the requests library. Next, you define the __all__ variable. In this example, __all__ includes the get_page_content() function and the WebPage class, which are public names.

Note that the helper function _fetch_page() is for internal use only. So, you don’t want to expose it to wildcard imports. Additionally, you don’t want the BASE_URL constant or the imported requests module to be exposed to wildcard imports.

Here’s how the module responds to a wildcard import:

>>> from webreader import *

>>> dir()
[
    'WebPage',
    ...
    'get_page_content'
]

When you run a wildcard import on the webreader module, only the names listed in __all__ are imported. Now go ahead and comment out the line where you define __all__, restart your REPL session, and run the import again:

>>> from webreader import *

>>> dir()
[
    'BASE_URL',
    'WebPage',
    ...
    'get_page_content',
    'requests'
]

A quick look at the output of dir() shows that now your module exports all the public names, including BASE_URL and even the imported requests library.

The __all__ variable lets you have full control over what a module exposes to wildcard imports. However, note that __all__ doesn’t prevent you from importing specific names from a module using an explicit import:

>>> from webreader import _fetch_page

>>> dir()
[
    ...
    '_fetch_page'
]

Note that you can use explicit import to bring any name from a given module, even non-public names as _fetch_page() in the example above.

Names From a Package

In the previous section, you learned how to use __all__ to define which objects are exposed to wildcard imports. Sometimes, you want to do something similar but at the package level. If you want to control the objects and names that a package exposes to wildcard imports, then you can do something like the following in the package’s __init__.py file:

from module_0 import name_0, name_1, name_2, name_3
from module_1 import name_4, name_5, name_6

__all__ = [
    "name_0",
    "name_1",
    "name_2",
    "name_3",
    "name_4",
    "name_5",
    "name_6",
]

The import statements tell Python to grab the names from each module in the package. Then, in __all__, you list the imported names as strings. This technique is great for those cases where you have a package with many modules, and you want to provide a direct path for imports.

As an example of how this technique works in practice, get back to the shapes package and update the __init__.py file as in the code below:

from shapes.circle import Circle
from shapes.square import Square

__all__ = ["Circle", "Square"]

In this update, you’ve added two explicit imports to get the Circle and Square classes from their respective module. Then, you add the class names as strings to the __all__ variable.

Here’s how the package responds to wildcard imports now:

>>> from shapes import *

>>> dir()
[
    'Circle',
    'Square',
    ...
]

Your shapes package exposes the Circle and Square classes to wildcard imports. These classes are what you’ve defined as the public interface of your package. Note how this technique facilitates direct access to names that otherwise you would have to import through qualified names.

Iterating Over a Package’s Interface

Because __all__ is typically a list object, you can use it to iterate over the objects that make up a module’s interface. The advantage of using __all__ over dir() is that the package author has explicitly defined which names they consider to be part of the public interface of their package. If you iterate over __all__, you won’t need to filter out non-public names as you’d have to when you iterate over dir(module).

For example, say that you have a module with a few similar classes that share the same interface. Here’s a toy example:

__all__ = ["Car", "Truck"]

class Car:
    def start(self):
        print("The car is starting")

    def drive(self):
        print("The car is driving")

    def stop(self):
        print("The car is stopping")

class Truck:
    def start(self):
        print("The truck is starting")

    def drive(self):
        print("The truck is driving")

    def stop(self):
        print("The truck is stopping")

In this module, you have two classes that represent vehicles. They share the same interface, so you can use them in similar places. You’ve also defined the __all__ variable, listing the two classes as strings.

Now say that you want to use these classes in a loop. How can you do this? You can use __all__ as in the code below:

>>> import vehicles

>>> for v in vehicles.__all__:
...     vehicle = getattr(vehicles, v)()
...     vehicle.start()
...     vehicle.drive()
...     vehicle.stop()
...
The car is starting
The car is driving
The car is stopping
The truck is starting
The truck is driving
The truck is stopping

In this example, you first import the vehicles module. Then, you start a for loop over the __all__ variable. Because __all__ is a list of strings, you can use the built-in getattr() function to access the specified objects from vehicles. This way, you’ve iterated over the classes that make up the module’s public API.

Accessing Non-Public and Dunder Names

When you’re writing modules and packages, sometimes you use module-level names that start and end with double underscores. These names are typically called dunder names. There are a few dunder constants, such as __version__ and __author__, that you may need to expose to wildcard imports.

Remember that the default behavior is that these names aren’t imported because they start with a leading underscore. To work around this issue, you can explicitly list these names in your __all__ variable.

To illustrate this practice, get back your webreader.py file and update it as in the code below:

import requests

__version__ = "1.0.0"
__author__ = "Real Python"

__all__ = ["get_page_content", "WebPage", "__version__", "__author__"]

BASE_URL = "http://example.com"

def get_page_content(page):
    return _fetch_page(page).text

# ...

In this update, you define two module-level constants that use dunder names. The first constant provides information about the module’s version, and the second constant holds the author’s name.

Here’s how a wildcard import works on this module:

>>> from webreader import *

>>> dir()
[
    'WebPage',
    ...
    '__author__',
    ...
    '__version__',
    'get_page_content'
]

Now, when someone uses a wildcard import on the webreader module, they get the dunder variables imported into their namespace.