In Python, a protocol specifies the methods and attributes that a class must implement to be considered of a given type. Protocols are important in Python’s type hint system, which allows for static type checking through external tools, such as mypy, Pyright, and Pyre.
Before there were protocols, these tools could only check for nominal subtyping based on inheritance. There was no way to check for structural subtyping, which relies on the internal structure of classes. This limitation affected Python’s duck typing system, which allows you to use objects without considering their nominal types. Protocols overcome this limitation, making static duck typing possible.
In this tutorial, you’ll:
- Gain clarity around the use of the term protocol in Python
- Learn how type hints facilitate static type checking
- Learn how protocols allow static duck typing
- Create custom protocols with the
Protocolclass - Understand the differences between protocols and abstract base classes
To get the most out of this tutorial, you’ll need to know the basics of object-oriented programming in Python, including concepts such as classes and inheritance. You should also know about type checking and duck typing in Python.
Get Your Code: Click here to download the free sample code that shows you how to leverage structural subtyping with Python protocols
The Meaning of “Protocol” in Python
During Python’s evolution, the term protocol became overloaded with two subtly different meanings. The first meaning refers to internal protocols, such as the iterator, context manager, and descriptor protocols.
These protocols are widely understood in the community and consist of special methods that make up a given protocol. For example, the .__iter__() and .__next__() methods define the iterator protocol.
Python 3.8 introduced a second, slightly different type of protocol. These protocols specify the methods and attributes that a class must implement to be considered of a given type. So, these protocols also have to do with a class’s internal structure.
With this kind of protocol, you can define interchangeable classes as long as they share a common internal structure. This feature allows you to enforce a relationship between types or classes without the burden of inheritance. This relationship is known as structural subtyping or static duck typing.
In this tutorial, you’ll focus on this second meaning of the term protocol. First, you’ll have a look at how Python manages types.
Dynamic and Static Typing in Python
Python is a dynamically typed language, which means that the Python interpreter checks an object’s type when the code runs. It also means that while a variable can only reference one object at a time, the type of that object can change during the variable’s lifetime.
For example, you can have a variable that starts as a string and changes into an integer number:
>>> value = "One hundred"
>>> value
'One hundred'
>>> value = 100
>>> value
100
In this example, you have a variable that starts as a string. Later in your code, you change the variable’s value to an integer.
Because of its dynamic nature, Python has embraced a flexible typing system that’s known as duck typing.
Duck Typing
Duck typing is a type system in which an object is considered compatible with a given type if it has all the methods and attributes that the type requires. This typing system supports the ability to use objects of independent and decoupled classes in a specific context as long as they adhere to some common interface.
Note: To dive deeper into duck typing, check out the Duck Typing in Python: Writing Flexible and Decoupled Code tutorial.
As an example of duck typing, you can consider built-in container data types, such as lists, tuples, strings, dictionaries, and sets. All of these data types support iteration:
>>> numbers = [1, 2, 3]
>>> person = ("Jane", 25, "Python Dev")
>>> letters = "abc"
>>> ordinals = {"one": "first", "two": "second", "three": "third"}
>>> even_digits = {2, 4, 6, 8}
>>> containers = [numbers, person, letters, ordinals, even_digits]
>>> for container in containers:
... for element in container:
... print(element, end=" ")
... print()
...
1 2 3
Jane 25 Python Dev
a b c
one two three
8 2 4 6
In this code snippet, you define a few variables using different built-in types. Then, you start a for loop over the collections and iterate over each of them to print their elements to the screen. Even though the built-in types are significantly different from one another, they all support iteration.
The duck typing system allows you to create code that can work with different objects, provided that they share a common interface. This system allows you to set relationships between classes that don’t rely on inheritance, which produces flexible and decoupled code.
Type Hints and Type Checking
Even though Python is a dynamically typed language that widely relies on duck typing, not knowing the type of arguments, return values, and attributes can be a source of errors. This is especially true in large codebases where functions and classes are spread through several modules and packages.
To overcome the potential issues, Python 3.5 introduced type hints or optional static typing. Type hints let you optionally specify types of arguments, return values, and attributes in your functions and classes. Then, you can check these types with a static type checker, such as mypy, and get useful information that can help you debug and improve your code, making it more robust.
Here’s a quick overview of some great benefits of using type hints in your code:
- Improved code readability: You can explicitly state the type of arguments, return values, and attributes, which enhances code readability for developers, making the interface of functions and classes easier to understand.
- Support for static type checking: You can provide type hints in your code and then use tools like mypy, Pyright, and Pyre to perform static type checking. These tools will allow you to find type inconsistencies and mismatches early in the development process so that you can fix them and make your code more robust.
- Better development environments: You can use type hints and take advantage of a modern code editor or IDE’s enhanced code completion, function signature information, and inline documentation, which improve your life quality as a Python developer. These features can also speed up development and reduce bugs caused by incorrect type usage.
- Aids in refactoring: You can make refactors that involve changing the type of arguments, return values, and attributes of a given function or class. In this situation, the type checkers will help you find all the parts of the code that you must update to align with the new definitions.
- Improved documentation: You can take advantage of type hints when writing your code’s documentation with tools like MkDocs. This way, the type information in the documentation will stay in sync with the code.
Python doesn’t do anything with type hints at runtime. They are stored in an .__annotations__ attribute and otherwise ignored. So, apart from the improved code readability, you need external tools like a static type checker or a modern code editor or IDE to enjoy these benefits.
Note: To learn more about type checking in Python, check out the Python Type Checking (Guide) tutorial.
As an example of using type hints, consider the following function that adds two numbers:
calculations.py
1from typing import Union
2
3def add(a: Union[float, int], b: Union[float, int]) -> float:
4 return float(a + b)
5
6print(add(2, 4))
7
8print(add("2", "4"))
The typing module plays a crucial role in defining type hints. It defines classes like Union that you can use to specify multiple possible types of variables, function arguments, and return values. In this example, you add type hints for the function’s arguments, a and b, and for the return value.
The arguments to add() can be floating-point or integer numbers. To set up these type hints, you use the Union class with the float and int types. Alternatively, you can use the pipe operator to express the same type hint. So, for example, you can write something like a: float | int.
Then, you use the arrow syntax (-> float) to state that the function’s return value will be a floating-point number. Finally, you have a couple of calls to add(). Here’s what you get when you run this script:
$ python calculations.py
6.0
24.0
The first result is correct because you used two numbers as arguments. Unfortunately, the second result is incorrect, but your code didn’t break. It ran without apparent problems. How would you avoid this type of issue? You can run some static type checking in your code.
To perform static type checking on the above code, you need an external tool. For example, you can use mypy, but first, you need to install it with the following command:
$ python -m pip install mypy
This command will install the mypy static type checker from PyPI. Now you can run the following command against your calculations.py file:
$ mypy calculations.py
calculations.py:8: error: Argument 1 to "add" has incompatible type "str";
expected "float | int" [arg-type]
calculations.py:8: error: Argument 2 to "add" has incompatible type "str";
expected "float | int" [arg-type]
Found 2 errors in 1 file (checked 1 source file)
With this command, you run a static type analysis on your code. As a result, you learn that your code has two errors in line 8. Those errors warn you about type inconsistencies in your code. With this information, you can go to your code and fix the inconsistencies so that the code works correctly.
Duck Typing vs Type Hints
As you’ve learned, duck typing allows for flexible and dynamic code. With duck typing, you can use different and unrelated objects in a given context if those objects have the expected methods and attributes. You don’t have to ensure that the objects share a common parent type through inheritance.
When you add type hints to a piece of code that relies on duck typing, you can encounter some challenges. Here’s a toy example that illustrates how type hints and duck typing can collide:
birds_v1.py
1class Duck:
2 def quack(self):
3 return "The duck is quacking!"
4
5def make_it_quack(duck: Duck) -> str:
6 return duck.quack()
7
8class Person:
9 def quack(self):
10 return "The person is imitating a duck quacking!"
11
12print(make_it_quack(Duck()))
13
14print(make_it_quack(Person()))
In this code, you define a Duck class with a .quack() method. Then, you define the make_it_quack() function that takes an argument of type Duck. Next, you create a Person class to use in a duck typing context. In this example, the calls to print() will work. However, in the second call, you have a type inconsistency because an instance of Person isn’t of type Duck.
Here’s what you get when you run mypy on this file:
$ mypy birds_v1.py
birds_v1.py:14: error: Argument 1 to "make_it_quack" has incompatible type
"Person"; expected "Duck" [arg-type]
Found 1 error in 1 file (checked 1 source file)
The type checker displays an error telling you that the call to print() in line 14 is using an object with an incompatible type.
One way you can fix this issue is to use inheritance:
birds_v2.py
class QuackingThing:
def quack(self):
raise NotImplementedError(
"Subclasses must implement this method"
)
class Duck(QuackingThing):
def quack(self):
return "The duck is quacking!"
class Person(QuackingThing):
def quack(self):
return "The person is imitating a duck quacking!"
def make_it_quack(duck: QuackingThing) -> str:
return duck.quack()
print(make_it_quack(Duck()))
print(make_it_quack(Person()))
In this new implementation, you create a new class called QuackingThing and use it as the parent class for Duck and Person. Now, the type checker passes:
$ mypy birds_v2.py
Success: no issues found in 1 source file
The type checker’s output is clean. However, you traded duck typing for inheritance. Now, your classes are tightly coupled, even if they don’t have a clear inheritance relationship. How can you fix this collision between duck typing and type hints? This is where protocols come into the scene.
Structural Subtyping and Protocols
In Python’s type system, you’ll find two ways to decide whether two objects are compatible as types:
- Nominal subtyping is strictly based on inheritance. A class that inherits from a parent class is a subtype of its parent.
- Structural subtyping is based on the internal structure of classes. Two classes with the same methods and attributes are structural subtypes of one another.
To understand the first subtyping strategy, say you have a class hierarchy to represent animals:
animals_v1.py
class Animal:
def __init__(self, name):
self.name = name
def eat(self):
print(f"{self.name} is eating.")
def drink(self):
print(f"{self.name} is drinking.")
class Dog(Animal):
def bark(self):
print(f"{self.name} is barking.")
class Cat(Animal):
def meow(self):
print(f"{self.name} is meowing.")
In this code, Dog and Cat are nominal subtypes of Animal. This means that you can use instances of Dog or Cat when Animal instances are expected. This form of subtyping is quick to understand and matches how the built-in isinstance() works:
>>> from animals_v1 import Animal, Cat, Dog
>>> pluto = Dog("Pluto")
>>> isinstance(pluto, Animal)
True
>>> tom = Cat("Tom")
>>> isinstance(tom, Animal)
True
The built-in isinstance() function allows you to check whether a given object is an instance of a class. This function considers subtypes when running the check. That’s why you get True in both examples above.
Checking types explicitly isn’t a popular practice in Python. Instead, the language favors duck typing, where you use the objects without considering their types but only the expected operations. Here’s where structural subtyping comes in handy.
For example, you can reimplement the animal classes without setting a formal relationship between them:
animals_v2.py
class Dog:
def __init__(self, name):
self.name = name
def eat(self):
print(f"{self.name} is eating.")
def drink(self):
print(f"{self.name} is drinking.")
def make_sound(self):
print(f"{self.name} is barking.")
class Cat:
def __init__(self, name):
self.name = name
def eat(self):
print(f"{self.name} is eating.")
def drink(self):
print(f"{self.name} is drinking.")
def make_sound(self):
print(f"{self.name} is meowing.")
Now Dog and Cat don’t have a strict inheritance relationship. They’re completely decoupled and independent classes.
However, they implement the same public interface. In other words, they have the same internal structure, including methods and attributes. Because of this characteristic, they’re structural subtypes, and you can use them in a duck typing context:
>>> from animals_v2 import Cat, Dog
>>> for animal in [Cat("Tom"), Dog("Pluto")]:
... animal.eat()
... animal.drink()
... animal.make_sound()
...
Tom is eating.
Tom is drinking.
Tom is meowing.
Pluto is eating.
Pluto is drinking.
Pluto is barking.
These classes work okay in a duck typing context. However, up to this point, you don’t have a formal way to indicate that Cat and Dog are subtypes. You only know they are because the code tells you they have the same internal structure. To formalize this subtype relationship, you can use a protocol.
As you’ve already learned, protocols allow you to specify the expected methods and attributes that a class should have to support a given feature without requiring explicit inheritance. So, protocols are explicit sets of methods and attributes.
In practice, a class can support multiple protocols. For example, you can informally say that your Dog and Cat classes have a living protocol consisting of the .eat() and .drink() methods, which are the operations required to support life. You can also say that the classes have a sounding protocol comprised of the .make_sound() method.
Note: In Python’s built-in classes, you’ll find many examples of classes that support multiple protocols. For example, lists, tuples, and strings support the sequence and iterable protocols.
With this in mind, you can create other classes that support any of these protocols:
person.py
class Person:
def __init__(self, name):
self.name = name
def eat(self):
print(f"{self.name} is eating.")
def drink(self):
print(f"{self.name} is drinking.")
def talk(self):
print(f"{self.name} is talking.")
This Person class supports the living protocol because it implements the required behaviors, .eat() and .drink(). However, the class doesn’t implement the .make_sound() method, so it doesn’t support the sounding protocol.
Protocols are particularly useful when it’s impractical to modify the inheritance structure of classes. With protocols, you can focus on defining the desired behavior and characteristics without having to design complex inheritance relationships.
Static Duck Typing With Protocols
As you’ve already learned, duck typing and type hints collide, imposing restrictions on Python coders who want to use both techniques. Python 3.8 introduced a way to create formal protocols in the typing system.
From this version on, the typing module defines a base class called Protocol that you can use to create custom protocols. This new class provides a formal way to support structural subtyping in the type hint system.
Since Python 3.8, the static type checkers started to support structural subtyping, which means that they check whether objects are of a specific nominal type and also whether they meet the requirement to be of a given structural type.
Consider the following toy example:
adder_v1.py
from typing import Protocol
class Adder(Protocol):
def add(self, x, y): ...
class IntAdder:
def add(self, x, y):
return x + y
class FloatAdder:
def add(self, x, y):
return x + y
def add(adder: Adder) -> None:
print(adder.add(2, 3))
add(IntAdder())
add(FloatAdder())
In this code, you define a class called Adder by inheriting from typing.Protocol. Adder implements an .add() method, which defines the Adder protocol itself. Note that protocol methods don’t have a body, which you typically indicate with the ellipsis (...) syntax.
Then, you define two classes, IntAdder and FloatAdder. These classes implement the Adder protocol because they have an .add() method. Therefore, you can use objects of either class as arguments to the add() function, which takes an Adder object as an argument.
The static type checker will be happy with your type hints on the add() function:
$ mypy adder_v1.py
Success: no issues found in 1 source file
The static type check passes because both IntAdder and FloatAdder support the Adder protocol. In this case, mypy considers the object’s internal structure rather than its nominal type.
Note that you haven’t made these classes inherit from Adder. They’re completely independent and decoupled classes that you can use in a duck typing context, such as the add() function.
Custom Protocols in Python
There’s much more to learn about creating custom protocols in Python. For example, you can create protocols through single or multiple inheritance. You can have protocols with different types of members. You can even create generic protocols.
In the following sections, you’ll learn about these topics and how they can help you improve your custom protocols and make them meet your typing requirements.
Creating Custom Protocols
You’ve already learned that you can define custom protocols using the Protocol class from the typing module. Adding type hints to the equation will make your type checking process more strict.
Consider the following example:
adder_v2.py
from typing import Protocol
class Adder(Protocol):
def add(self, x: float, y: float) -> float:
...
class IntAdder:
def add(self, x, y):
return x + y
def add(adder: Adder) -> None:
print(adder.add(2, 3))
add(IntAdder()) # mypy: Success: no issues found in 1 source file
Because the IntAdder class doesn’t use explicit type hints, mypy considers the method’s arguments and return value to be of type Any, which makes the type check succeed regardless of the types used.
Now, go ahead and add type hints for the arguments to .add() and its return value in the IntAdder class to enable mypy to perform type checking:
adder_v2.py
# ...
class IntAdder:
def add(self, x: int, y: int) -> int:
return x + y
def add(adder: Adder) -> None:
print(adder.add(2, 3))
add(IntAdder())
In this update, you’ve stated that the .add() method will take integer arguments and return an integer value. Go ahead and run mypy on this new version of adder_v2.py:
$ mypy adder_v2.py
adder_v2.py:17: error: Argument 1 to "add" has incompatible type "IntAdder";
expected "Adder" [arg-type]
adder_v2.py:17: note: Following member(s) of "IntAdder" have conflicts:
adder_v2.py:17: note: Expected:
adder_v2.py:17: note: def add(self, x: float, y: float) -> float
adder_v2.py:17: note: Got:
adder_v2.py:17: note: def add(self, x: int, y: int) -> int
Found 1 error in 1 file (checked 1 source file)
This output tells you that your IntAdder class is incompatible with the Adder protocol because .add() takes and returns incompatible types. How do you make the check pass? You could update the Adder protocol to accept either integer or floating-point numbers:
adder_v3.py
from typing import Protocol
class Adder(Protocol):
def add(self, x: int | float, y: int | float) -> int | float:
...
# ...
In this case, the .add() method can take either integer or float arguments. It can also return arguments of either type.
Go ahead and run mypy again:
$ mypy adder_v3.py
adder_v2.py:17: error: Argument 1 to "add" has incompatible type "IntAdder";
expected "Adder" [arg-type]
adder_v2.py:17: note: Following member(s) of "IntAdder" have conflicts:
adder_v2.py:17: note: Expected:
adder_v2.py:17: note: def add(self, x: int | float, y: int | float)
-> int | float
adder_v2.py:17: note: Got:
adder_v2.py:17: note: def add(self, x: int, y: int) -> int
Found 1 error in 1 file (checked 1 source file)
This update doesn’t solve the issue. You need to take another approach, and that’s when generic protocols come in handy.
Building Generic Protocols
Protocol classes can be generic. To create a generic protocol, you can use the typing.TypeVar class. The syntax for a generic protocol is like the following:
from typing import Protocol, TypeVar
T = TypeVar("T")
class GenericProtocol(Protocol[T]):
def method(self, arg: T) -> T:
...
In this code snippet, you first define a generic type, T, using the TypeVar class. Then, you create a generic protocol using Protocol[T] as the parent class. You can also use generic types in the arguments and return values of the protocol’s methods.
You can use a generic protocol to overcome the issue you found in the previous section:
adder_v4.py
from typing import Protocol, TypeVar
T = TypeVar("T", bound=int | float)
class Adder(Protocol[T]):
def add(self, x: T, y: T) -> T:
...
class IntAdder:
def add(self, x: int, y: int) -> int:
return x + y
class FloatAdder:
def add(self, x: float, y: float) -> float:
return x + y
def add(adder: Adder) -> None:
print(adder.add(2, 3))
add(IntAdder())
add(FloatAdder())
In this example, you first define a generic type for your protocol. You use the bound argument to state that the generic type can be an int or float object. Then, you have your concrete adders. In this case, you have IntAdder and FloatAdder to sum numbers.
If you’re using Python 3.12, then you can use a simplified syntax:
from typing import Protocol
class Adder(Protocol):
def add[T: int | float](self, x: T, y: T) -> T:
...
# ...
With this syntax, you don’t have to import TypeVar and create the generic type beforehand. You just have to use square brackets after the method’s name, use the generic T type, and optionally bound some existing types.
Exploring Possible Protocol Members
Up to this point, you’ve written protocols with regular instance methods. However, protocols can have different types of members, including the following:
- Class attributes
- Instance attributes
- Instance methods
- Class methods
- Static methods
- Properties
- Abstract methods
You should use the ClassVar class to distinguish between class attributes and instance attributes. Here’s a demo protocol class that defines all the above members:
members.py
from abc import abstractmethod
from typing import ClassVar, Protocol
class ProtocolMembersDemo(Protocol):
class_attribute: ClassVar[int]
instance_attribute: str = ""
def instance_method(self, arg: int) -> str:
...
@classmethod
def class_method(cls) -> str:
...
@staticmethod
def static_method(arg: int) -> str:
...
@property
def property_name(self) -> str:
...
@property_name.setter
def property_name(self, value: str) -> None:
...
@abstractmethod
def abstract_method(self) -> str:
...
You use ClassVar to define a class attribute. Then, you have an instance attribute with a default value, which is optional.
It’s important to note that instance attributes must be declared at the class level for the type checker to consider them a part of the protocol. Otherwise, if you define instance attributes inside instance methods, which is a common practice in Python, then you’ll get an error from your type checker.
Next up, you have different types of methods. In this example, none of the methods have a defined implementation. However, you can have protocol methods with default implementations. Finally, note that all the members of a protocol class can have type hints.
Creating Subprotocols
You can take advantage of existing protocols to create new ones using inheritance. For example, say that you’re creating a platform to publish blog posts and video posts. You want to use proper type hints in all your classes and functions, so you write the following protocols:
contents.py
from typing import Protocol
class ContentCreator(Protocol):
def create_content(self) -> str:
...
class Blogger(ContentCreator, Protocol):
posts: list[str]
def add_post(self, title: str, content: str) -> None:
...
class Vlogger(ContentCreator, Protocol):
videos: list[str]
def add_video(self, title: str, path: str) -> None:
...
In this code snippet, you have a base protocol called ContentCreator that defines a .create_content() method. Then, you create two derived protocols, Blogger and Vlogger. These classes inherit from the base protocol, ContentCreator, which makes them subprotocols.
The classes also inherit from Protocol. Note that this isn’t mandatory. However, it’s a best practice to ensure that the semantics and intentions are clear.
Next, you define two concrete classes:
contents.py
# ...
class Blog:
def __init__(self):
self.blog_posts = []
def create_content(self) -> str:
return "Creating a post."
def add_post(self, title: str, content: str) -> None:
self.blog_posts.append(f"{title}: {content}")
print(f"Post added: {title}")
class Vlog:
def __init__(self):
self.videos = []
def create_content(self) -> str:
return "Recording a video."
def add_video(self, title: str, path: str) -> None:
self.videos.append(f"{title}: {path}")
print(f"Video added: {title}")
The Blog class meets the Blogger protocol while the Vlog class meets the Vlogger protocol. Now, you can have client code to use these classes:
contents.py
# ...
def produce_content(creator: ContentCreator):
print(creator.create_content())
def add_post(blogger: Blogger, title: str, content: str):
blogger.add_post(title, content)
def add_video(vlogger: Vlogger, title: str, path: str):
vlogger.add_video(title, path)
The first function allows you to create a new piece of content. It can take any object that satisfies the ContentCreator protocol. The other two functions allow you to add a new blog post and video. They can take objects that satisfy the Blogger and Vlogger protocols, respectively.
Now, you can use mypy to check if your type hints work:
$ mypy contents.py
Success: no issues found in 1 source file
Great! The output is clean. Your subprotocol classes work as expected!
Recursive Protocols
You can also define recursive protocols, which are protocols that reference themselves in their definition. To reference a protocol, you must provide its name as strings.
Recursive protocols are useful for representing self-referential data structures like linked lists. Consider the following example:
linked_list.py
from typing import Optional, Protocol
class LinkedListNode(Protocol):
value: int
next_node: Optional["LinkedListNode"]
def __str__(self) -> str:
return f"{self.value} -> {self.next_node}"
In this code snippet, you define a protocol called LinkedListNode to represent a node in a linked list. The class has two instance attributes: .value to hold the current node’s value and .next_node to hold the next node in the list.
Note: To reference a protocol within its definition, you must include its name as a string literal to avoid errors. That’s because you can’t refer to a type that isn’t fully defined yet. While this limitation will change in the future, for now, you can use a future import as an alternative:
from __future__ import annotations
from typing import Optional, Protocol
class LinkedListNode(Protocol):
value: int
next_node: Optional[LinkedListNode]
def __str__(self) -> str:
return f"{self.value} -> {self.next_node}"
By importing annotations from __future__, you ensure that the annotations or type hints are treated as strings, allowing you to reference LinkedListNode even before it’s fully defined.
The .next_node attribute should also be a node object, so you use a recursive reference to the LinkedListNode as a string. In this example, you use the Optional class to express that .next_node can also be None.
Now, go ahead and create a concrete node and some client code:
linked_list.py
# ...
class Node:
def __init__(
self,
value: int,
next_node: Optional["LinkedListNode"] = None,
):
self.value = value
self.next_node = next_node
def __str__(self) -> str:
return f"{self.value} -> {self.next_node}"
def print_linked_list(start_node: LinkedListNode):
print(start_node)
node3 = Node(3)
node2 = Node(2, node3)
node1 = Node(1, node2)
print_linked_list(node1)
The Node class satisfies the LinkedListNode protocol. Next, you have the print_linked_list() function, which takes a LinkedListNode object as an argument and prints the linked list to the screen. Finally, you have code to create a few nodes and call print_linked_list().
You can run mypy on this file now:
$ mypy linked_list.py
Success: no issues found in 1 source file
The output from mypy is clean, so your recursive protocol works as expected. You’ve written the correct type hints for your classes, which is great!
Predefined Protocols in Python
Python has several predefined protocols. Iterable and Iterator are good examples. A few of them are in the typing module under the Protocols section. However, most of the currently available protocols live in the collections.abc module because they’re implemented as abstract base classes.
Some of the most common predefined protocols include the following:
| Class | Methods |
|---|---|
Container |
.__contains__() |
Hashable |
.__hash__() |
Iterable |
.__iter__() |
Iterator |
.__next__() and .__iter__() |
Reversible |
.__reversed__() |
Generator |
.send(), .throw(), .close(), .__iter__(), and .__next__() |
Sized |
.__len__() |
Callable |
.__call__() |
Collection |
.__contains__(), .__iter__(), and .__len__() |
Sequence |
.__getitem__(), .__len__(), .__contains__(), .__iter__(), .__reversed__(), .index(), and .count() |
MutableSequence |
.__getitem__(), .__setitem__(), .__delitem__(), .__len__(), .insert(), .append(), .clear(), .reverse(), .extend(), .pop(), .remove(), and .__iadd__() |
ByteString |
.__getitem__() and .__len__() |
Set |
.__contains__(), .__iter__(), .__len__(), .__le__(), .__lt__(), .__eq__(), .__ne__(), .__gt__(), .__ge__(), .__and__(), .__or__(), .__sub__(), .__xor__(), and .isdisjoint() |
MutableSet |
.__contains__(), .__iter__(), .__len__(), .add(), .discard(), .clear(), .pop(), .remove(), .__ior__(), .__iand__(), .__ixor__(), and .__isub__() |
Mapping |
.__getitem__(), .__iter__(), .__len__(), .__contains__(), .keys(), .items(), .values(), .get(), .__eq__(), and .__ne__() |
MutableMapping |
.__getitem__(), .__setitem__(), .__delitem__(), .__iter__(), .__len__(), .pop(), .popitem(), .clear(), .update(), and .setdefault() |
AsyncIterable |
.__aiter__() |
AsyncIterator |
.__anext__() and .__aiter__() |
AsyncGenerator |
.asend(), .athrow(), .aclose(), .__aiter__(), and .__anext__() |
Buffer |
.__buffer__() |
Even though these classes are abstract base classes (ABC) rather than formal protocols, you can use them as protocols in your type hints. The static type checkers should be able to process them as expected.
In some situations, you probably won’t need to use these classes because you can just use the concrete built-in type. For example, instead of using the Set ABC to type hint an argument or a return value, you can use the built-in set class.
However, there are situations where using the concrete type won’t meet your needs. For example, say that you want to code a function that takes some integer values and filters the even numbers, returning a list. Here’s the function without type hints:
even_v1.py
def filter_even_numbers(numbers):
return [number for number in numbers if number % 2 == 0]
print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))
This function works with any iterable object. In other words, you can use a list, tuple, set, or any other iterable object as an argument to filter_even_numbers().
How would you type hint the numbers argument in this function? You can think of doing something like the following:
even_v2.py
def filter_even_numbers(numbers: list[int]) -> list[int]:
return [number for number in numbers if number % 2 == 0]
print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))
The list[int] type hint for numbers doesn’t work well. It causes numbers to only accept list objects, which makes the function less generic. Your static type checker will fail with tuples, sets, or any other iterable:
$ mypy even_v2.py
even_v2.py:5: error: Argument 1 to "filter_even_numbers" has incompatible
type "tuple[int, int, int, int, int, int, int, int, int, int]";
expected "list[int]" [arg-type]
even_v2.py:6: error: Argument 1 to "filter_even_numbers" has incompatible
type "set[int]"; expected "list[int]" [arg-type]
Found 2 errors in 1 file (checked 1 source file)
This output shows that the two final calls to print() cause issues related to the data types of the input values for the numbers argument.
To fix the issues and provide a generic type hint that allows your function to take different iterables of numbers, you can use the Iterable ABC as in the code below:
even_v3.py
from collections.abc import Iterable
def filter_even_numbers(numbers: Iterable[int]) -> list[int]:
return [number for number in numbers if number % 2 == 0]
print(filter_even_numbers([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
print(filter_even_numbers((1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
print(filter_even_numbers({1, 2, 3, 4, 5, 6, 7, 8, 9, 10}))
In this update, you import the Iterable class from collections.abc. This class implements the .__iter__() method, satisfying the iterable protocol. Being an iterable of integer values, this method is exactly what your filter_even_numbers() function needs to work properly.
Now, go ahead and run mypy again:
$ mypy even_v3.py
Success: no issues found in 1 source file
The output is now clean. This means that the type hints for the function work correctly. You can use this function with any input iterable of integer values.
Protocols vs Abstract Base Classes
An abstract base class (ABC) is designed to be subclassed but not instantiated. This type of class defines a specific public interface (API) and enforces that interface in its subclasses. To create an abstract base class, you can use the ABC class from the abc module.
To illustrate how you’ll typically use ABCs, consider the following example:
shapes_v1.py
from abc import ABC, abstractmethod
from math import pi
class Shape(ABC):
@abstractmethod
def get_area(self) -> float:
pass
@abstractmethod
def get_perimeter(self) -> float:
pass
class Circle(Shape):
def __init__(self, radius) -> None:
self.radius = radius
def get_area(self) -> float:
return pi * self.radius**2
def get_perimeter(self) -> float:
return 2 * pi * self.radius
class Square(Shape):
def __init__(self, side) -> None:
self.side = side
def get_area(self) -> float:
return self.side**2
def get_perimeter(self) -> float:
return 4 * self.side
In this example, the Shape class is an abstract base class. With this class, you’re stating that any class that you create as a Shape subclass must have the .get_area() and .get_perimeter() methods in their public interface. That’s what the Circle and Square classes do. If you fail to implement one of the required methods, then you’ll get an error.
You can use the Shape class to provide type hints to your client code. Go ahead and add the following code to your shapes_v1.py file:
shapes_v1.py
# ...
def print_shape_info(shape: Shape):
print(f"Area: {shape.get_area()}")
print(f"Perimeter: {shape.get_perimeter()}")
circle = Circle(10)
square = Square(5)
print_shape_info(circle)
print_shape_info(square)
The type hint for the shape argument will work correctly because Circle and Square are subclasses of Shape.
In this example, you’ve used an ABC and inheritance to enforce a specific interface in a group of classes. However, sometimes it’s desirable to have a similar result without inheritance. That’s when you can use a protocol:
shapes_v2.py
from math import pi
from typing import Protocol
class Shape(Protocol):
def get_area(self) -> float:
...
def get_perimeter(self) -> float:
...
class Circle:
def __init__(self, radius) -> None:
self.radius = radius
def get_area(self) -> float:
return pi * self.radius**2
def get_perimeter(self) -> float:
return 2 * pi * self.radius
class Square:
def __init__(self, side) -> None:
self.side = side
def get_area(self) -> float:
return self.side**2
def get_perimeter(self) -> float:
return 4 * self.side
def print_shape_info(shape: Shape):
print(f"Area: {shape.get_area()}")
print(f"Perimeter: {shape.get_perimeter()}")
circle = Circle(10)
square = Square(5)
print_shape_info(circle)
print_shape_info(square)
In this new implementation, you use a protocol instead of an ABC. Now, you don’t rely on inheritance for your type hints to work correctly. Your classes are decoupled from each other. Their only point of coincidence is that they have a piece of interface in common.
In short, the main difference between an abstract base class and a protocol is that the former works through a formal inheritance relationship, while the latter doesn’t need this relationship. Note that this difference doesn’t make ABCs better than protocols or vice versa. They have different use cases and purposes.
ABCs are suitable when you have control over the class hierarchy and want to define a consistent interface across subclasses. Meanwhile, protocols are useful in scenarios where modifying class hierarchies is impractical or when there’s no clear inheritance relationship between classes.
Considering Potential Downsides of Protocols
It’s worthwhile to mention a downside of protocols that ABCs don’t exhibit. For example, protocols can make the type checker accept a completely unrelated type, which happens to satisfy the protocol completely by accident but is otherwise inappropriate.
Here’s a relevant example:
from typing import Protocol
class Message(Protocol):
def encode(self) -> bytes:
...
def send(message: Message) -> None:
...
send("Hello, World!") # Passes the type checker
In this example, strings happen to have the .encode() method, which returns bytes and satisfies the protocol. However, you may want the send() function to only accept instances of custom classes that implement the Message protocol. This behavior creates a loophole in the type checking system.
Another potential downside of protocols is that isinstance() will raise an exception when used with them. Consider the following code that takes advantage of the shapes example from the previous section:
>>> from shapes_v2 import circle, Shape
>>> isinstance(circle, Shape)
Traceback (most recent call last):
...
TypeError: Instance and class checks can only be used with
@runtime_checkable protocols
This check could be unreliable because protocols don’t set an inheritance relationship. If you need your classes to work with isinstance() even if they’re not subclasses, then you can use the @runtime_checkable decorator in the class definition:
shapes_v3.py
from typing import Protocol, runtime_checkable
@runtime_checkable
class Shape(Protocol):
...
The @runtime_checkable decorator marks a protocol class as a runtime protocol so that you can use it with isinstance() and issubclass().
Here’s how the previous example works now:
>>> from shapes_v3 import circle, Shape
>>> isinstance(circle, Shape)
True
After decorating your Shape protocol class with @runtime_checkable, the built-in isinstance() function works as you wanted.
Conclusion
Now you know how to use protocols in Python. Protocols let you define a type relationship between objects without the burden of inheritance. This relationship is based on the internal structure of classes.
With protocols, you can perform structural subtyping or static duck typing using Python’s type hint system and external static type checkers, like mypy, Pyright, and Pyre.
In this tutorial, you’ve:
- Gained clarity about the different uses of the term protocol in Python
- Learned how type hints facilitate static type checking
- Learned how protocols provide support for static duck typing
- Created custom protocols with the
Protocolclass - Understood the differences between protocols and ABCs
With this knowledge, you’re ready to get the most out of using Python’s type hint system and static type checkers.
Get Your Code: Click here to download the free sample code that shows you how to leverage structural subtyping with Python protocols
