Chapter 15. More About Type Hints

This is a new chapter in this edition of the book.

What’s New in This Chapter #

Overloaded Signatures #

it’s the signatures that we are overloading, not the function.
remember that python doesn’t allow function overloading!

implementation:

the actual function will ned no type hints, because the overloads will take care of it

can be implemented within the same module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
        import functools
        import operator
        from collections.abc import Iterable
        from typing import overload, Union, TypeVar

        T = TypeVar('T')
        S = TypeVar('S')  # <1> for the second overload

        @overload
        def sum(it: Iterable[T]) -> Union[T, int]: ...  # <2>
        @overload
        def sum(it: Iterable[T], /, start: S) -> Union[T, S]: ...  # <3>
        def sum(it, /, start=0):  # <4>
            return functools.reduce(operator.add, it, start)

Max Overload #

pythonic apis are hard to annotate. this is because they strongly leverage the powerful dynamic features of python
this section demonstrates what it takes to annotate the max function.

Takeaways from Overloading max #

the expressiveness of annotation markings is very limited, compared to that of python

TypedDict #

gotcha: remember for json objs we’ll need to do runtime checking. the pydantic package is great for this.
Static type checking is unable to prevent errors with code that is inherently dynamic, such as json.loads()
objective: we want to be able to define the structure of a container type (heterogeneous)
we should be able to provide a type specific to a key
TypedDict have no runtime effect, only for static analysis
Gives:
1. Class-like syntax to annotate a dict with type hints for the value of each “field.”
2. A constructor that tells the type checker to expect a dict with the keys and values as specified.
```
       from typing import TypedDict

       class BookDict(TypedDict):
               isbn: str
               title: str
               authors: list[str]
               pagecount: int
```
  looks very similar to a dataclass builder like a typing.NamedTuple but it isn’t.
@ runtime, the constructor just ends up creating a plain dict. No instance attributes, no init functions for the class, no method definitions.
none of the types will be enforced, “illegal” assignments can happen

Type Casting #

type casting is for type checkers to get assisted by us

typing.cast() special function provides one way to handle type checking malfunctions or incorrect type hints in code we can’t fix.

Casts are used to silence spurious type checker warnings and give the type checker a little help when it can’t quite understand what is going on.

Does absolutely nothing @ runtime

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  # tag::CAST[]
  from typing import cast

  def find_first_str(a: list[object]) -> str:
      index = next(i for i, x in enumerate(a) if isinstance(x, str))
      # We only get here if there's at least one string
      return cast(str, a[index])
  # end::CAST[]


  from typing import TYPE_CHECKING

  l1 = [10, 20, 'thirty', 40]
  if TYPE_CHECKING:
      reveal_type(l1)

  print(find_first_str(l1))

  l2 = [0, ()]
  try:
      find_first_str(l2)
  except StopIteration as e:
      print(repr(e))

too many uses of cast is likely a code-smell; Mypy is not that useless!
why casts still have some purpose:
1. the other workarounds are worse:
  - # type: ignore is less informative
  - Any is contagious, it will have cascading effects through type inference and undermine the type checker’s ability to detect errors in other parts of the code

Reading Type Hints at Runtime #

within the __annotations__ attribute, it’s a dict that has the names and their types
the return type has the key "return"
annotations are evaluated by the interpreter at import time, just like param default values

Problems with Annotations at Runtime #

extra CPU and memory load when importing
types not yet defined are strings instead of actual types \(\implies\) the forward-reference-problem
we can use introspection helpers for this
e.g. inspect.get_type_hints
this is the recommended way to read type hints at runtime

Dealing with the Problem #

just keep an eye out on how to handle this, it’s likely to change from 3.10 onwards

Implementing a Generic Class #

have to concretise the generic type by giving a type parameter: machine = LottoBlower[int](range(1, 11))

here’s a generic LottoBlower:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
  import random

  from collections.abc import Iterable
  from typing import TypeVar, Generic

  from tombola import Tombola

  T = TypeVar('T')

  class LottoBlower(Tombola, Generic[T]):  # <1> have to subclass Generic to declare the formal type params

      def __init__(self, items: Iterable[T]) -> None:  # <2>
          self._balls = list[T](items)

      def load(self, items: Iterable[T]) -> None:  # <3>
          self._balls.extend(items)

      def pick(self) -> T:  # <4>
          try:
              position = random.randrange(len(self._balls))
          except ValueError:
              raise LookupError('pick from empty LottoBlower')
          return self._balls.pop(position)

      def loaded(self) -> bool:  # <5>
          return bool(self._balls)

      def inspect(self) -> tuple[T, ...]:  # <6>
          return tuple(self._balls)

Basic Jargon for Generic Types #

Generic type: type with 1 or more type vars
Formal Type Parameter: the generic type var used to define a generic type
Parameterized type: type declared with actual type parameters (resolved)
Actual type param: the actual types given as params when a param type is declared

Variance #

useful to know if we want to support generic container types or provide callback-based APIs.
Practically speaking, most cases supported if we just support the invariant containers

the following sections use a concrete analogy to drive the point:

Imagine that a school cafeteria has a rule that only juice dispensers can be installed. General beverage dispensers are not allowed because they may serve sodas, which are banned by the school board.

code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
  from typing import TypeVar, Generic


  class Beverage:
      """Any beverage."""


  class Juice(Beverage):
      """Any fruit juice."""


  class OrangeJuice(Juice):
      """Delicious juice from Brazilian oranges."""


  T_co = TypeVar('T_co', covariant=True)


  class BeverageDispenser(Generic[T_co]):
      def __init__(self, beverage: T_co) -> None:
          self.beverage = beverage

      def dispense(self) -> T_co:
          return self.beverage


  class Garbage:
      """Any garbage."""


  class Biodegradable(Garbage):
      """Biodegradable garbage."""


  class Compostable(Biodegradable):
      """Compostable garbage."""


  T_contra = TypeVar('T_contra', contravariant=True)


  class TrashCan(Generic[T_contra]):
      def put(self, trash: T_contra) -> None:
          """Store trash until dumped."""


  class Cafeteria:
      def __init__(
          self,
          dispenser: BeverageDispenser[Juice],
          trash_can: TrashCan[Biodegradable],
      ):
          """Initialize..."""


  ################################################ exact types

  juice_dispenser = BeverageDispenser(Juice())
  bio_can: TrashCan[Biodegradable] = TrashCan()

  arnold_hall = Cafeteria(juice_dispenser, bio_can)


  ################################################ covariant dispenser

  orange_juice_dispenser = BeverageDispenser(OrangeJuice())

  arnold_hall = Cafeteria(orange_juice_dispenser, bio_can)


  ################################################ non-covariant dispenser

  beverage_dispenser = BeverageDispenser(Beverage())

  ## Argument 1 to "Cafeteria" has
  ## incompatible type "BeverageDispenser[Beverage]"
  ##          expected "BeverageDispenser[Juice]"
  # arnold_hall = Cafeteria(beverage_dispenser, bio_can)


  ################################################ contravariant trash

  trash_can: TrashCan[Garbage] = TrashCan()

  arnold_hall = Cafeteria(juice_dispenser, trash_can)


  ################################################ non-contravariant trash

  compost_can: TrashCan[Compostable] = TrashCan()

  ## Argument 2 to "Cafeteria" has
  ## incompatible type "TrashCan[Compostable]"
  ##          expected "TrashCan[Biodegradable]"
  # arnold_hall = Cafeteria(juice_dispenser, compost_can)

An Invariant Dispenser #

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# tag::BEVERAGE_TYPES[]
from typing import TypeVar, Generic

class Beverage:  # <1> we establish a type hierarchy
    """Any beverage."""

class Juice(Beverage):
    """Any fruit juice."""

class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""

T = TypeVar('T')  # <2> simple typevar

class BeverageDispenser(Generic[T]):  # <3> Parameterised on the beverage type
    """A dispenser parameterized on the beverage type."""
    def __init__(self, beverage: T) -> None:
        self.beverage = beverage

    def dispense(self) -> T:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:  # <4> module-global function
    """Install a fruit juice dispenser."""
# end::BEVERAGE_TYPES[]

################################################ exact type

# tag::INSTALL_JUICE_DISPENSER[]
juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)
# end::INSTALL_JUICE_DISPENSER[]


################################################ variant dispenser

# tag::INSTALL_BEVERAGE_DISPENSER[]
beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
##          expected "BeverageDispenser[Juice]"
# end::INSTALL_BEVERAGE_DISPENSER[]


################################################ variant dispenser

# tag::INSTALL_ORANGE_JUICE_DISPENSER[]
orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[OrangeJuice]"
##          expected "BeverageDispenser[Juice]"
# end::INSTALL_ORANGE_JUICE_DISPENSER[]

BeverageDispenser(Generic[T]) is invariant when BeverageDispenser[OrangeJuice] is not compatible with BeverageDispenser[Juice] — despite the fact that OrangeJuice is a subtype-of Juice.
It depends on how we have defined the typevar
In this case, the function was defined with an actual type var: def install(dispenser: BeverageDispenser[Juice]) -> None:

A Covariant Dispenser #

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
from typing import TypeVar, Generic


class Beverage:
    """Any beverage."""


class Juice(Beverage):
    """Any fruit juice."""


class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""


# tag::BEVERAGE_TYPES[]
T_co = TypeVar('T_co', covariant=True)  # <1> convention to suffix it like that.


class BeverageDispenser(Generic[T_co]):  # <2> we use the typevar as the param for the generic class
    def __init__(self, beverage: T_co) -> None:
        self.beverage = beverage

    def dispense(self) -> T_co:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:  # <3>
    """Install a fruit juice dispenser."""
# end::BEVERAGE_TYPES[]

################################################ covariant dispenser

# tag::INSTALL_JUICE_DISPENSERS[]

# both Juice and OrangeJuice aer valid in a covariant BeverageDispenser:
juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
# end::INSTALL_JUICE_DISPENSERS[]

################################################ more general dispenser not acceptable

# tag::INSTALL_BEVERAGE_DISPENSER[]
beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
##          expected "BeverageDispenser[Juice]"
# end::INSTALL_BEVERAGE_DISPENSER[]

covariance: the subtype relationship of the parameterized dispensers varies in the same direction as the subtype relationship of the type parameters.
- two type of types: A: type vars B: dispenser type vars
  The question is whether the we allow the variance in the same direction (co-variant).
Supports Generic type and ALSO its subtypes
Implementation notes:
- by convention, the typevar should be suffixed with _co
- just need to set covariant=True when we declare the typevar

A Contravariant Trash Can #

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# tag::TRASH_TYPES[]
from typing import TypeVar, Generic

class Refuse:  # <1>
    """Any refuse."""

class Biodegradable(Refuse):
    """Biodegradable refuse."""

class Compostable(Biodegradable):
    """Compostable refuse."""

T_contra = TypeVar('T_contra', contravariant=True)  # <2>

class TrashCan(Generic[T_contra]):  # <3>
    def put(self, refuse: T_contra) -> None:
        """Store trash until dumped."""

def deploy(trash_can: TrashCan[Biodegradable]):
    """Deploy a trash can for biodegradable refuse."""
# end::TRASH_TYPES[]


################################################ contravariant trash can


# tag::DEPLOY_TRASH_CANS[]
bio_can: TrashCan[Biodegradable] = TrashCan()
deploy(bio_can)

trash_can: TrashCan[Refuse] = TrashCan()
deploy(trash_can)
# end::DEPLOY_TRASH_CANS[]


################################################ more specific trash can

# tag::DEPLOY_NOT_VALID[]
compost_can: TrashCan[Compostable] = TrashCan()
deploy(compost_can)
## mypy: Argument 1 to "deploy" has
## incompatible type "TrashCan[Compostable]"
##          expected "TrashCan[Biodegradable]"
# end::DEPLOY_NOT_VALID[]

implementation:
- use _contra suffix for the contravariant typevar
in our example, TrashCan is contravariant on the type of refuse.

Variance Review #

Invariant Types #

if L (generic type) is invariant, then L[A] (parameterised type) is not a supertype or a subtype of L[B]
This is regardless of the relationships between A and B (the actual types)
examples: mutable collections in python are invariant so list[int] is not consistent-with list[float and vice-versa
if a formal type param (T) appears in the type hints of the method args AS WELL AS the return types, then the parameter must be invariant
this ensures type safety
by default, TypeVar creates invariant types

Covariant Types #

nomenclature:
- X :> Y: means that X is supertype of OR same as Y and vice versa
Covariant generic types follow the subtype relationship of the actual type parameters.
if A :> B (type B is a subclass of type A) and a we consider type C (generic type). Iff C is contravariant then C[A] :> C[B].
Here, A and B are the actual type params.
examples:
- Frozen set
  float :> int and frozenset[float] :> frozenset[int]
  SAME DIRECTION
- Iterators
  Any code expecting an abc.Iterator[float] yielding floats can safely use an abc.Iterator[int] yielding integers.

Callable types are covariant on the return type
this is so that the subclass will also work

Contravariant Types #

A :> B, a generic type K is contravariant if K[A] <: K[B]
reverses the subtype relationship of the actual type parameters (opposite direction)

A contravariant container is usually a write-only data structure, aka a sink
Callable[[ParamType, …], ReturnType] is contravariant on the parameter types, but covariant on the ReturnType
examples:
1. Refuse :> Biodegradable so TrashCan[Refuse] <: TrashCan[Biodegradable]
2. for callbacks,
  - contravariant formal parameter defines the type of args used to invoke / send data to this object \(\implies\) the callback object is therefore the sink
  - covariant formal params define the types of outputs produced by the object (yield type or return type) \(\implies\) callback object acts as a source the gives outputs.

Rules of Thumb #

producing:
If a formal type parameter defines a type for data that comes out of the object, it can be covariant. So we can produce more specific (narrower) results than strictly required.
consuming:
If a formal type parameter defines a type for data that goes into the object after its initial construction, it can be contravariant. So we can accept/consume more specific (narrower) results than strictly required.
consuming == producing \(\implies\) invariant
If a formal type parameter defines a type for data that comes out of the object and the same parameter defines a type for data that goes into the object, it must be invariant.
To err on the safe side, make formal type parameters invariant.

Callbacks are special because they accept inputs and return output

Python’s typing system with Callable is directly designed to reflect this.

See this elaboration:

You're absolutely on the right track! **Callbacks** (like callables or function-type objects) in Python's typing system **are indeed special** because **they play both roles:**
- They **accept inputs** (they are a "sink" for arguments), and
- They **return outputs** (they are a "source" for results)

This duality is exactly why variance in callables is *split*:
- **Parameter types are contravariant** (inputs; the callback can accept _more general_ arguments than strictly required)
- **Return types are covariant** (outputs; the callback can produce _more specific_ results than strictly required)

Let me show this with an example.

---
### Example: Callbacks, Variance, and Typing

Imagine you define this type:

```python
from typing import Callable

# A callback that takes a Biodegradable and returns a Juice
CallbackType = Callable[[Biodegradable], Juice]
```

This means:
- You need something that can accept a `Biodegradable` (or *any* base/parent of Biodegradable, i.e., `Refuse`)
- And it must return a `Juice` (or any *subtype*, e.g., `OrangeJuice`)

#### Function Matching the Signature

```python
def process_refuse(refuse: Refuse) -> OrangeJuice:
    """Accepts any refuse, returns orange juice (a subtype of Juice)."""
    return OrangeJuice()
```

This function:
- Accepts a `Refuse` (broader than just `Biodegradable`) ⇒ **contravariant**
- Returns an `OrangeJuice` (narrower than `Juice`) ⇒ **covariant**

**We can use process_refuse wherever CallbackType is required, thanks to variance:**

```python
def handle_event(cb: CallbackType) -> None:
    refuse = Biodegradable()
    juice = cb(refuse)
    print(f"{juice!r} dispensed.")

handle_event(process_refuse)  # This typechecks!
```

---
### Why?

- **Contravariant parameter**:
  A function that can handle any `Refuse` is good enough anywhere that needs just `Biodegradable` — it won't ever receive a non-biodegradable!
- **Covariant return**:
  If someone expects a `Juice`, it's fine to give them something more specific (like `OrangeJuice`).

---
## Summary Table

| Callback signature                      | Description               | Variance      |
|----------------------------------------- |---------------------------|-------------- |
| `Callable[[ParamType], ReturnType]`      | callback taking input(s), returning output | Parameter: **Contravariant**Return: **Covariant** |

- Think of input parameters as arguments being "sent in" (*sink*, contravariant)
- Think of return values as outputs "sent out" (*source*, covariant)

---
## Intuitive Analogy

- **Input arguments**: You want callbacks that can accept *at least* what you have (so a broader parameter is better).
- **Output/Return**: You want callbacks to promise to return *at least* what you asked for (so a narrower/specific return type is better).

---
### Expanded Example

#### Class hierarchy

```python
class Refuse: pass
class Biodegradable(Refuse): pass
class Compostable(Biodegradable): pass

class Beverage: pass
class Juice(Beverage): pass
class OrangeJuice(Juice): pass
```

#### Callbacks

```python
from typing import Callable

# Expected callback: takes a Biodegradable, returns a Juice
CallbackType = Callable[[Biodegradable], Juice]

def cb1(arg: Refuse) -> OrangeJuice:
    ... # Accepts any Refuse, returns a specific Juice

def cb2(arg: Compostable) -> Beverage:
    ... # Accepts only Compostable, returns a general Beverage

def cb3(arg: Biodegradable) -> Juice:
    ... # Accepts Biodegradable, returns Juice

f: CallbackType

f = cb1  # OK! Wider input, narrower output
f = cb2  # mypy: error (input too narrow, output too broad)
f = cb3  # OK! exact types
```

---
## Conclusion
So your summary is **correct**:
- **Callbacks _are both_ a sink (inputs) and a source (outputs)**.
    - **Input arguments:** Contravariant (wider is OK)
    - **Return type:** Covariant (narrower is OK)

**Python’s typing system with Callable is directly designed to reflect this.**

Implementing a Generic Static Protocol #

There’s some examples here for reference, nothing special though

Chapter Summary #

We have covered:

type @overload, including for the max function
typing.TypedDict, which is not a class builder,
this is useful for defining the type of a dict (keys and values) when a dict is used as a record, often with the handling of JSON data
can give a false sense of security though, since it has no runtime effect
typing.cast as a way to handle some issues with type checkers. If overdone, it’s a code smell.
Runtime type hint access including name-forwarding approaches
GENERICS!!
back to typed world
Generic Static Protocol
- allows us to be specific in the original protocol form