Chapter 8. Type Hints in Functions

Table of Contents

What’s New in This Chapter #

About Gradual Typing #

what it means by a gradual type system #

“type hints are optional at all levels”

:NOTER_PAGE: (285 0.308300395256917 . 0.14267185473411154)

it has an interplay between duck typing and nominal typing
a type system that is optional
- by optional it means that we need to be able to silence it
- we can silence it at varying levels of abstraction
a type system that doesn’t catch type errors @ runtime
- only for static analysis
doesn’t enhance performance

Type hints are optional at all levels: you can have entire packages with no type hints, you can silence the type checker when you import one of those packages into a mod‐ ule where you use type hints, and you can add special comments to make the type checker ignore specific lines in your code.

Gradual Typing in Practice #

Starting with Mypy #

Making Mypy More Strict #

GOTCHA: accidentally using `=` instead of `:` for type hints #

so this is legal but also a typo: def hex2rgb(color=str) -> tuple[int,int,int]

wherein we accidentally wrote = instead of :.

Just have to be careful for these things because the static analyser won’t point it out (since it’s legal).

Good Style: #

No space between the parameter name and the :; one space after the :
Spaces on both sides of the = that precedes a default parameter value

Use blue instead of black for the static typechecking, it’s more aligned with python’s idioms.

A Default Parameter Value #

EXTRA NOTE: python prefers single quotes by default for strings #

``using single quotes''

Using None as a Default \(\implies\) use `Optional` #

the idea here is that None can be a better default value to use. So the type hinting should use an Optional

It still needs a default value (of None) because typehints are meaningless at runtime.

BTW, it’s not the annotation that makes the param optional, it’s the provisioning of a default value for that param.

Types Are Defined by Supported Operations #

what’s a “type”?
in a practical sense, see it as the set of supported operations
a supported operation here refers to whether the data object has the associated operator function defined or not.
So the example given is abc.Sequence and it does not have the __mul__ implemented, so if the function is this then the type checker will complain
```
  from collections import abc

  def double(x: abc.Sequence):
          return x * 2
```

Gradual Type System: an interplay b/w duck typing and nominal typing #

``have the interplay of two different views of types:''

the key idea is, when do we want to detect typing errors: if @ runtime, then it’s more aligned with duck typing. if @ compile time, then it’s aligned with nominal typing.

duck typing (implicitly, structural typing):
- focuses on “behaviour”, only enforced at runtime
- objects are “typed” but variables aren’t
- what really matters is what operations are supported \(\implies\) that’s why it’s duck typing.
  “if it quacks like a duck” means if it has an implementation like that and the implementation supports the arguments provided
- naturally this type checking is done at runtime
nominal typing:
- focuses on “type identity”
- “nominal” because it depends on the name, referring to the declaration that was made (like a label)
- compatibility of type depends on what the explicitly-defined type is
NOTE: a static checker may complain about type errors even if the code will actually work and execute without issues.

there’s a duality to be balanced here #

This little experiment shows that duck typing is easier to get started and is more flexi‐ ble, but allows unsupported operations to cause errors at runtime. Nominal typing detects errors before runtime, but sometimes can reject code that actually runs—such as the call alert_bird(daffy)

Types Usable in Annotations #

The Any Type #

the purpose of defining an `any` type #

more general types \(\implies\) narrower interfaces in the sense that they support fewer operations.
need for a special wildcard type: so you’d want to have something that can accept values of every type but not end up having a narrow interface \(\rightarrow\) that’s why we have any
so, Any is a magic type that sits at the bottom and at the top of the type hierarchy (from the POV of the typechecker).

More general types have narrower interfaces, i.e., they support fewer operations. The object class implements fewer operations than abc.Sequence, which implements fewer operations than abc.MutableSequence, which implements fewer operations than list. But Any is a magic type that sits at the top and the bottom of the type hierarchy. It’s simultaneously the most general type—so that an argument n: Any accepts values of every type—and the most specialized type, supporting every possible operation. At least, that’s how the type checker understands Any

Contrasting `subtype-of` vs `consistent-with` relations #

In a gradual type-system there are elements of behavioural sub-typing (the classic one that adheres to LSP principle) as well as a more flexible compatibility notion in the form of consistent sub-typing.

subtype-of relationship: behavioural sub-typing adheres to LSP
- LSP was actually defined in the context of supported operations:
  If an object of T2 substitutes an object of type T1 and the program still behaves correctly, then T2 is a subtype-of T1.
T2 is expected. This focus on supported operations is reflected in the name behavioral subtyping,

consistent-with relationship: that’s what the any is for
this is the part where Any is consistence with both up and down the heirarchy.

Simple Types and Classes #

Can just directly use them for type-hinting.
for classes, consistent-with is defined like subtype-of: a subclass is consistent with all its superclasses.
exception: int is Consisten-With complex
- all the numeric types are directly subclassed from object.
- int has a superset of functions but it’s not really a subclass of complex but it is still consistent-with complex!

Optional and Union Types #

even the optional type is just syntax sugar for Union[myType , None].
the latest syntax allows us to use A | B instead of Union[A, B].
NOTE: we can actually define return types that are Unions, but this makes it ugly because the caller of this function now needs to handle the type checking at runtime.
Union is more useful with types that are not consistent among themselves.
For example:
Union[int, float] is redundant because int is consistent-with float.
If you just use float to annotate the parameter, it will accept int values as well.

syntactic sugar for optional and union type: `|` #

Better Syntax for Optional and Union in Python 3.10 We can write str | bytes instead of Union[str, bytes] since Python 3.10. It’s less typing, and there’s no need to import Optional or Union from typing. Contrast the old and new syntax for the type hint of the plural parameter of show_count: plural: Optional[str] = None plural: str | None = None
The | operator also works with isinstance and issubclass to build the second argument: isinstance(x, int | str). For more, see PEP 604—Complementary syntax for Union[].

try not to define return values with union types #

it means the responsibility of doing type checking on the return values is on the consumer of the function \(\rightarrow\) bad pattern

Generic Collections (defining types for collections like `list[str]`) #

python collections (container classes) are generally heterogeneous
Generic types can be declared with type parameters to specify the type of the items they can handle.
the simplest form of generic type hints is container[item] where container is any container type; examples being:
- list
- set
- abc.MutableSet
references:
- see the official docs on GenericAlias

Situations that python’s type annotations won’t be able to handle: #

unsupported 1 - can’t type check array.array typecode for python v 3.10

unsupported 2 - when collection defined with typecode, overflow is not checked for
yet another reminder that these numerics in python are not fixed-width
constructor argument, which determines whether integers or floats are stored in the array. An even harder problem is how to type check integer ranges to prevent OverflowError at runtime when adding elements to arrays. For example, an array with typecode=‘B’ can only hold int values from 0 to 255. Cur‐ rently, Python’s static type system is not up to this challenge.

Tuple Types #

There are 3 ways we can annotate tuple types:

annotating them as records
annotating them as records with named fields
annotating them as immutable sequences

tuples as records #

Just use the builtin like e.g. def geohash(lat_lon: tuple[float,float]) -> str:

for tuples being used as records with named fields \(\implies\) using `NamedTuple` #

can “alias” it using a named tuple – follows the consistent-with relationship

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
  from typing import NamedTuple
  from geolib import geohash as gh

          PRECISION = 9

          class Coordinate(NamedTuple):
              lat: float
              lon: float
              # NOTE this wrapper prevents static checkers from complaining that the geohash lib does not have typehints.
          def geohash(lat_lon:Coordinate) -> str:
                  return gh.encode(*lat_lon, PRECISION)

So here, Coordinate is consistent-with tuple[float,float]
because of this consistency, if a fn signature was def display(lat_lon: tuple[float, float]) -> str:, then Coordinate NamedTuple will still work

None

for tuples to be used as immutable sequences #

Objective here is to annotate tuples of unspecified length that are used as immutable lists

We specify a single type, followed by a comma and ...

This ellipsis is useful to us.

e.g. tuple[int, ...] is a tuple with int items.

note: we can’t tie down a particular length though

Here’s a consolidated example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from collections.abc import Sequence
        def columnize(
                # 1: this is a sequence of strings
                sequence: Sequence[str], num_columns: int = 0
                # 2: return type below means it's a list of tuples and the tuples only contain strings.
        ) -> list[tuple[str, ...]]:
                if num_columns == 0:
                        num_columns = round(len(sequence) ** 0.5)

                num_rows, reminder = divmod(len(sequence), num_columns)
                num_rows += bool(reminder)
                return [tuple(sequence[i::num_rows]) for i in range(num_rows)]

Generic Mappings #

the syntax is just MappingType[KeyType, ValueType]
we can annotate local variables! e.g. index: dict[str, set[str]] = {}

interesting example of an inverted index #

``returning an inverted index’'

There’s a whole bunch of literature on inverted indexes. This does a value -> key mapping.

see wikipedia article on this

good example code #

In the example below:

the local variable of index is annotated because the typechecker will complain otherwise

the walrus operator := is used to define a name for an expression-output

TO_HABIT: this is useful and I already use it for while loops, but I can potentially use it for other expressions as well.

in the example for charidx.py:

   # we name the expression for the if-predicate so that we can use it thereafter
   if name := unicodedata.name(char, ''):
           for word in tokenize(name):
                   index.setdefault(word, set()).add(char)

More information about the walrus operator:

   The **walrus operator** (`:=`), introduced in Python 3.8, allows for assignment expressions, enabling you to assign a value to a variable as part of an expression. This feature can enhance code readability and efficiency in certain contexts. Here’s a detailed overview of its functionality, use cases, and implications.

   ### What is the Walrus Operator?

   - **Syntax**: The walrus operator is used as follows:
     ```python
     variable := expression
     ```
     This assigns the result of `expression` to `variable` and returns that value.

   - **Purpose**: The primary purpose of the walrus operator is to allow assignments to be made within expressions, reducing redundancy and improving code conciseness.

   ### Key Use Cases

   1. **Reducing Redundant Calculations**:
      - The walrus operator can be particularly useful when you want to use a value multiple times without recalculating it.
      ```python
      # Without walrus operator
      result = [func(x) for x in data if func(x)]

      # With walrus operator
      result = [y for x in data if (y := func(x))]
      ```
      In this example, `func(x)` is called only once per iteration instead of twice.

   2. **Cleaner Loop Constructs**:
      - It simplifies loops where the loop condition depends on user input or other calculations.
      ```python
      # Using walrus operator
      while (data := input("Enter your data: ")) != "":
          print("You entered:", data)
      ```
      This eliminates the need for an initial assignment before entering the loop.

   3. **Conditional Assignments**:
      - You can assign a value within an if statement, making the code more readable.
      ```python
      if (match := re.match(pattern, s)):
          print("Match found:", match.group())
      ```

   4. **Accumulate Data In-Place**:
      - The walrus operator can also be used to accumulate values while iterating.
      ```python
      c = 0
      print([(c := c + x) for x in [5, 4, 3, 2]])  # Output: [5, 9, 12, 14]
      ```

   ### Advantages

   - **Conciseness**: It reduces boilerplate code by allowing assignments within expressions.
   - **Performance**: It can improve performance by avoiding repeated function calls or calculations.
   - **Readability**: In certain contexts, it makes the code clearer by showing intent directly where values are being assigned and used.

   ### Considerations

   - **Readability vs. Complexity**: While it can enhance readability, excessive or inappropriate use may lead to complex and hard-to-read code. It's important to balance conciseness with clarity.
   - **Avoiding Nested Expressions**: Using nested walrus operators can make code difficult to understand and maintain.

   ### Etymology of "Walrus"

   The term "walrus operator" is informal and comes from the resemblance of the `:=` symbol to a walrus's eyes and tusks. The playful name was popularized in discussions about its introduction and has since become widely accepted in the Python community.

   ### Mental Model

   To conceptualize the walrus operator:
   - Think of it as a way to "capture" a value while simultaneously using it in an expression.
   - Visualize it as a tool that allows you to hold onto something (the value) while you continue working with it immediately (the expression).

   ### Summary

   The walrus operator (`:=`) in Python provides a powerful way to assign values within expressions, enhancing code conciseness and performance in specific scenarios. While it offers significant advantages, careful consideration should be given to its use to maintain code clarity and avoid unnecessary complexity.

   Citations:
   [1] https://www.geeksforgeeks.org/walrus-operator-in-python-3-8/
   [2] https://martinheinz.dev/blog/79
   [3] https://www.kdnuggets.com/how-not-to-use-pythons-walrus-operator
   [4] https://realpython.com/python-walrus-operator/
   [5] https://www.reddit.com/r/Python/comments/jmnant/walrus_operator_good_or_bad/
   [6] https://stackoverflow.com/questions/73644898/why-is-python-walrus-operator-needed-instead-of-just-using-the-normal-assig
   [7] https://realpython.com/python-operator-module/
   [8] https://www.digitalocean.com/community/tutorials/how-to-use-args-and-kwargs-in-python-3

the tokenize function is a generator. KIV for chapter 17 for a deep dive into this.

Example 8-14. charindex.py


import re
import unicodedata

from collections.abc import Iterator

RE_WORD = re.compile(r"\w+")
STOP_CODE = sys.maxunicode + 1


def tokenize(text: str) -> Iterator[str]:
    """
    return iterable of uppercased words
    """
    for match in RE_WORD.finditer(text):
        yield match.group().upper()


def name_index(start: int = 32, end: int = STOP_CODE) -> dict[str, set[str]]:
    index: dict[str, set[str]] = {}

    for char in (chr(i) for i in range(start, end)):
        if name := unicodedata.name(char, ""):
            for word in tokenize(name):
                index.setdefault(word, set()).add(char)
    return index

Abstract Base Classes #

PRINCIPLE: Robustness Principle / Postel’s Law:
“Be conservative in what you send, be liberal in what you accept.”
it makes sense to define a generic type hint (of abstract classes) so that we can support many concrete implementations of it.

rule of thumb - better to use abc.Mapping or abc.MutableMapping instead of dict #

Because it will support more mapping types

Therefore, in general it’s better to use abc.Mapping or abc.MutableMapping in parameter type hints, instead of dict (or typing.Dict in legacy code).

fall of the “numeric tower” of numeric class-hierarchy #

there used to be a bunch of ABCs for numeric types, but now it’s not useful because numeric types are special.
they are directly subclassed from Object type and are consistent-with each other.
this numeric tower is a linear hierarchy of ABCs with Number at the top
1. Number
2. Complex
3. Real
4. Rational
5. Integral

Point being that the static type checking of things within the numeric tower doesn’t work well – have to use the explicit types, KIV the solution for it, comes in a later chapter

Those ABCs work perfectly well for runtime type checking, but they are not sup‐ ported for static type checking. The “Numeric Tower” section of PEP 484 rejects the numbers ABCs and dictates that the built-in types complex, float, and int should be treated as special cases, as explained in “int Is Consistent-With complex” on page

3 options to type-annotate numeric things #

use a concrete type instead e.g. int, float, complex
declare a union type Union[float, Decimal, Fraction]
Use numeric protocols e.g. SupportsFloat
kiv numeric protocols for chapter 13

In practice, if you want to annotate numeric arguments for static type checking, you have a few options:
Use one of the concrete types int, float, or complex—as recommended by PEP
Declare a union type like Union[float, Decimal, Fraction].
If you want to avoid hardcoding concrete types, use numeric protocols like Sup
portsFloat, covered in “Runtime Checkable Static Protocols” on page 468. The upcoming section “Static Protocols” on page 286 is a prerequisite for understand‐ ing the numeric protocols. Meanwhile, let’s get to one of the most useful ABCs for type hints: Iterable.

Generic Iterables #

Python Typeshed Project #

Not that important.

Just for compatibility initially.

It is a way to provide “headers” with type annotations.

This is how the type annotations are retrofit in existing stdlibs because the stdlib fucntions have no annotations.

It relies on a .pyi file that’s basically like a c-header file.

:NOTER_PAGE: (310 0.6666666666666667 . 0.2703549060542797)

``Stub Files and the Typeshed Project''

Explicit Type Aliases are supported, they improve readability #

⚠️ Danger of unbounded iterables on memory requirements #

GOTCHA: iterable arguments need to be completely consumed. This poses a risk if we have infinite iterables (e.g. cyclic generators).
this is something to keep in mind about.
however, the value of this is that it allows flexibility and the ability to inject in generators instead of prebuilt sequences

return a result. Given an endless iterable such as the itertools.cycle generator as input, these functions would consume all memory and crash the Python process. Despite this potential danger, it is fairly common in modern Python to offer functions that accept an Iterable input even if they must process it completely to return a result.

Parameterized Generics and TypeVar #

for us to refer to a generic type, we have to use TypeVars
KIV the fact that TypeVar also allows us to define covariants and contravariants in addition to bounds.

type var bound @ point of usage, is a reflection on the result type #

where T is a type vari‐ able that will be bound to a specific type with each usage. This allows a parameter type to be reflected on the result type.

why TypeVar is needed (and unique to python) #

TypeVar is a construct that is unique to the python language
introduces the variable name in the current namespace as opposed to getting that variable declared beforehand
it’s unique because languages like C, Java, Typescript don’t needt he name of type variables to be declared beforehand, so they don’t need such a construct

mental model: it’s a variable representing a type instead of being a type by itself

see more on typevar:

  The concept of **TypeVar** in Python is a unique construct primarily used for creating generic types, allowing developers to write functions and classes that can operate on any data type while maintaining type safety. This feature is particularly useful in statically typed languages, but it has specific implications and uses in Python, which is dynamically typed. Here’s a breakdown of why TypeVar is significant in Python and how it differs from similar concepts in languages like JavaScript.

  ### Understanding TypeVar in Python

  1. **Generic Programming**:
     - **TypeVar** allows you to define a placeholder for a type that can be specified later when the function or class is instantiated. This enables generic programming, where you can write code that works with any data type.
     - Example:
       ```python
       from typing import TypeVar, Generic

       T = TypeVar('T')

       class Wrapper(Generic[T]):
           def __init__(self, value: T):
               self.value = value

       int_wrapper = Wrapper(10)  # T is inferred as int
       str_wrapper = Wrapper("Hello")  # T is inferred as str
       ```

  2. **Type Safety**:
     - TypeVar enhances type safety by ensuring that the operations performed on the generic type are valid for the specific type passed during instantiation. This helps catch errors at development time rather than runtime.

  3. **Flexibility**:
     - It allows for more flexible and reusable code. You can create functions and classes that can handle multiple types without duplicating code for each specific type.

  ### Comparison with JavaScript

  JavaScript does not have a direct equivalent to Python's TypeVar due to its dynamic typing system. Here are some key differences:

  1. **Dynamic vs. Static Typing**:
     - JavaScript is dynamically typed, meaning types are determined at runtime and variables can hold values of any type without explicit declarations.
     - In contrast, Python’s TypeVar allows for static type checking when using tools like `mypy`, enabling developers to specify expected types while still maintaining flexibility.

  2. **Lack of Generics**:
     - While JavaScript supports some level of generics through its type systems (like TypeScript), it does not have built-in constructs like TypeVar that are part of the core language syntax.
     - In TypeScript (a superset of JavaScript), generics are defined differently, using angle brackets (`<T>`), but they do not use a construct like `TypeVar` to define a variable type that can be reused across multiple functions or classes.

  3. **Type Inference**:
     - Python's TypeVar allows for type inference based on context, which can help with readability and maintainability of code. JavaScript's dynamic nature means that developers often rely on documentation or comments to convey expected types.

  ### Etymology of "TypeVar" and Mental Model

  The term **TypeVar** combines "Type" (referring to data types) and "Var" (short for variable). This naming emphasizes that it acts as a variable representing a type rather than being a concrete type itself.

  #### Mental Model:
  - Think of **TypeVar** as a placeholder or a template for a data type:
    - Imagine it as an empty box labeled "T" where you can put different items (data types) later. When you define a function or class using TypeVar, you’re saying, “This box can hold anything; just tell me what it will hold when you use it.”
  - This concept aligns with generic programming principles found in other languages but is uniquely adapted to Python's dynamic typing environment.

  ### Summary

  - **TypeVar** is a powerful construct in Python that enables generic programming by allowing developers to create flexible and reusable code while maintaining type safety.
  - Unlike JavaScript, which lacks direct support for generics in its core syntax, Python provides TypeVar as part of its typing module, facilitating static type checking.
  - The term "TypeVar" reflects its role as a variable representing types, allowing developers to think in terms of templates or placeholders when designing their functions and classes.

  Citations:
  [1] https://stackoverflow.com/questions/55345608/instantiate-a-type-that-is-a-typevar
  [2] https://discuss.python.org/t/non-uniqueness-of-typevar-on-python-versions-3-12-causes-resolution-issues/37350
  [3] https://guicommits.com/python-generic-type-function-class/
  [4] https://typing.readthedocs.io/en/latest/spec/generics.html
  [5] https://www.reddit.com/r/learnpython/comments/1adbgfp/should_i_use_a_typevar/
  [6] https://dagster.io/blog/python-type-hinting
  [7] https://docs.python.org/es/3.13/library/typing.html
  [8] https://www.typescriptlang.org/play/typescript/language-extensions/nominal-typing.ts.html

make the [] operator work on classes like Sequence[T]. But the name of the T variable inside the brackets must be defined somewhere—otherwise the Python interpreter would need deep changes to support generic type notation as special use of []. That’s why the typing.TypeVar constructor is needed: to introduce the variable name in the cur‐ rent namespace. Languages such as Java, C#, and TypeScript don’t require the name of type variable to be declared beforehand,

Restricting/Bounding the TypeVar #

there might be a need to explicitly restrict using a whilelist of types instead of letting the consistent-with subtyping do its job.
Without the restriction, anything that is consistent with T will work, but that’s unideal because the function that’s consuming the type most likely needs this to be restricted
we have 2 ways to restrict the possible types assigned to T:

[1] restricted TypeVar – references a whitelist
This is a fixed whitelist.
Problem is that, it may not be easy to maintain if numerous items in the list.
that’s where bounding can be done.
``NumberT = TypeVar(‘NumberT’, float, Decimal, Fraction)’'

[2] bounded TypeVar – defines an upper bound on the type, works on anything that is consistent-with
- sets an upper boundary for the acceptable types.
  e.g. HashableT = TypeVar('HashableT', bound=Hashable) then the variable could be Hashable or any of its subtypes
- careful not to get confused with the use of the word “bound” for that named param to TypeVar. It’s just
- this becomes the same generics construct as in Java
The solution is another optional parameter of TypeVar: the bound keyword parame‐ ter. It sets an upper boundary for the acceptable types. In Example 8-18, we have bound=Hashable, which means the type parameter may be Hashable or any subtype- of it.14

Predefined TypeVars #

AnyStr is an example of such a predefined type var, supports both bytes and str .

Static Protocols via `typing.Protocols` #

A protocol in the historical sense is an informal interface. KIV proper introduction to Protocols till Chapter 13.

In the context of type hints,

A protocol is really all about structural typing. Types match if the behaviours are consistent-with each other.

This feature is also known as “static duck typing”. It’s because we make duck typing explicit for static type checkers.

the solution to annotate the series parameter of top was to say “The nominal type of series doesn’t matter, as long as it implements the __lt__ method.” Python’s duck typing always allowed us to say that implicitly, leaving static type checkers clueless. That’s the contrast with implicit duck typing that we have been seeing all along

protocol definition vs implementation #

protocol can be defined by subclassing typing.Protocol
it’s a class of its own,
here’s an example:
1 2 3 4 5
from typing import Protocol, Any class SupportsLessThan(Protocol): def __lt__(self, other: Any) -> bool: ...
and then we can use this protocol to define a TypeVar: LT = TypeVar('LT', bound=SupportsLessThan)
NOTE:
1. it subclasses typing.Protocol
2. class body has one or more methods. the methods have ... in their bodies.
  this is sufficient to define the type signature for the protocol, and that’s what matters / is used to determine if something adheres to a protocol.
the implementer of a protocol doesn’t need to inherit, register or declare any relationship with the class that defines the protocol

a protocol type is defined by specifying one or more methods, and the type checker verifies that those methods are imple‐ mented where that protocol type is required. In Python, a protocol definition is written as a typing.Protocol subclass. However, classes that implement a protocol don’t need to inherit, register, or declare any rela‐ tionship with the class that defines the protocol. It’s up to the type checker to find the available protocol types and enforce their usage.

use case: when we can’t just define a boundary, but we want to define a protocol for the functions it supports #

were able to use typing.Hashable as the upper bound for the type parameter. But now there is no suitable type in typing or abc to use, so we need to create it.

T just needs to be checked if it’s consistent-with the protocol #

disambiguation b/w protocols and abstract classes (as interfaces) in OOP (see perplexity thread)

  To clarify the concepts of **protocols** and **interfaces** in Python, and to compare them with Java's virtual interfaces, let's break down the definitions and functionalities.

  ### Disambiguation: Protocols vs. Interfaces in Python

  1. **Protocols**:
     - **Definition**: A protocol in Python, introduced in PEP 484, defines a set of methods and properties that a class should implement, but it does not require explicit inheritance. Protocols allow for structural typing, meaning that if a class has the required methods and properties, it can be considered to implement the protocol.
     - **Usage**: Protocols are defined using the `typing.Protocol` class. They are useful for static type checking with tools like `mypy` and allow for "duck typing" — if an object behaves like a certain type, it can be used as that type regardless of its actual class.
     - **Example**:
       ```python
       from typing import Protocol

       class Drawable(Protocol):
           def draw(self) -> None:
               ...

       class Circle:
           def draw(self) -> None:
               print("Drawing a circle")

       def render(shape: Drawable) -> None:
           shape.draw()

       circle = Circle()
       render(circle)  # This works because Circle has a draw method
       ```

  2. **Interfaces**:
     - **Definition**: In Python, interfaces are often implemented using abstract base classes (ABCs) defined in the `abc` module. An interface specifies a contract that classes must adhere to by explicitly declaring methods that must be implemented.
     - **Usage**: Interfaces require explicit inheritance from an ABC. This means that if a class claims to implement an interface, it must inherit from the ABC and implement all its abstract methods.
     - **Example**:
       ```python
       from abc import ABC, abstractmethod

       class Drawable(ABC):
           @abstractmethod
           def draw(self) -> None:
               pass

       class Circle(Drawable):
           def draw(self) -> None:
               print("Drawing a circle")

       def render(shape: Drawable) -> None:
           shape.draw()

       circle = Circle()
       render(circle)  # This works because Circle implements Drawable
       ```

  ### Comparison with Java's Virtual Interfaces

  - In Java, interfaces are explicitly defined contracts that classes must implement. A class can implement multiple interfaces, providing flexibility similar to Python's protocols.
  - Java's interfaces support default methods (methods with an implementation), which is not directly available in Python's protocols or ABCs without additional workarounds.
  - Both Java interfaces and Python protocols allow for polymorphism; however, Python's protocols enable duck typing without requiring explicit declarations of interface implementation.

  ### Summary of Differences

  | Feature               | Protocols (Python)                  | Interfaces (Python via ABC)               | Java Interfaces                       |
  |-----------------------|-------------------------------------|-------------------------------------------|---------------------------------------|
  | Definition            | Structural typing                   | Explicit method contracts                  | Explicit method contracts              |
  | Inheritance           | No explicit inheritance required    | Requires explicit inheritance              | Requires explicit implementation       |
  | Flexibility           | More flexible (duck typing)        | Less flexible (strict adherence needed)   | Supports multiple interface implementation |
  | Default Methods       | No default methods                  | No default methods                        | Supports default methods               |
  | Use Case              | Type checking with mypy            | Enforcing method implementation            | Defining contracts for behavior       |

  ### Conclusion

  In summary, while both protocols and interfaces serve to define expected behaviors in Python, they do so in different ways. Protocols offer more flexibility through structural typing, while interfaces enforce strict adherence through inheritance. Understanding these distinctions helps in designing systems that leverage polymorphism effectively.

  If you have further questions or need more clarification on any specific points regarding protocols, interfaces, or their comparisons with other languages like Java, feel free to ask!

  Citations:
  [1] https://stackoverflow.com/questions/67233702/python-terminology-interface-vs-protocol
  [2] https://blog.glyph.im/2021/03/interfaces-and-protocols.html
  [3] https://sinavski.com/post/1_abc_vs_protocols/
  [4] https://testdriven.io/tips/9f452585-e673-4617-8f35-ac85ab413e14/
  [5] https://www.reddit.com/r/Python/comments/10ikape/interfaces_with_protocols_why_not_ditch_abc_for/
  [6] https://news.ycombinator.com/item?id=26488156
  [7] https://andrewbrookins.com/technology/building-implicit-interfaces-in-python-with-protocol-classes/
  [8] https://discuss.python.org/t/add-built-in-flatmap-function-to-functools/21137

protocol has one or more method definitions, with … in their bodies. A type T is consistent-with a protocol P if T implements all the methods defined in P, with matching type signatures.

example #

the examples below use MyPy’s debugging facilities, take note.

typing.TYPE_CHECK can be used to guard against runtime function calls
- things like reveal_type() is a Mypy debugging facility, not a regural function.
typing.TYPE_CHECKING constant is always False at runtime, but type check‐ ers pretend it is True when they are type checking.

reveal_type() is a pseudofunction, a mypy debugging facility
``reveal_type() pseudofunction call, showing the inferred type of the argument.''

Callables via `typing.Callable` #

allows us to hint the type of Higher Order Functions that are taking in callables
parameterized like so: Callable[[ParamType1, ParamType2], ReturnType]
The params list can have zero or more types.
if we need a type hint to match a function with a flexible signature, replace the whole parameter list with a ...
Callable[..., ReturnType]
other than that, there’s NO syntax to annotate optional or kwargs

Variance in Callable Types #

With generic type params, we now have to deal with type hierarchies and so we have to deal with type variance.

KIV variance on Chapter 15

covariance
example: Callable[[], int] is a subtype-of Callable[[], float] because int is a subtype of float
\(\implies\) Callable is covariant on the return type because the subtype-of relationships of the types int and float is in the same direction as the relationship of the Callable types that use them as return types
most parameterized generic types are invariant

NoReturn via `typing.NoReturn` #

for functions that never return

actually used for no returns like exception throws in the case of sys.exit() that raises SystemExit

extra: typeshed-like stub files don’t define default values, so they use `...` instead #

``Stub files don’t spell out the default values, they use … instead.''

:NOTER_PAGE: file:///Users/rtshkmr/org/future_vyapari/books/Luciano Ramalho - Fluent Python_ Clear, Concise, and Effective Programming-O’Reilly Media (2022).pdf :ID: ./Luciano Ramalho - Fluent Python_ Clear, Concise, and Effective Programming-O’Reilly Media (2022).pdf-annot-325-10

extra: the use of ellipsis operator `...` is context-dependent #

The `...` operator in Python, known as the **ellipsis**, is a built-in singleton object of type `ellipsis`. Its role varies depending on context, and it’s often used where meaning can be ambiguous because Python itself does not mandate one specific use. Here are its main uses:

1. **Placeholder for Incomplete Code**
   You can use `...` inside functions, classes, or other blocks to indicate "code not yet implemented" or "to be done later," similar to `pass`. For example:
   ```python
   def my_function():
       ...
   ```
   This works as a no-op placeholder, and some developers prefer it stylistically over `pass`.

2. **Extended Slicing in Multi-Dimensional Arrays**
   In libraries like **NumPy**, `...` is very useful for slicing arrays with many dimensions. It stands for "expand all missing dimensions" in slice notation, allowing you to avoid explicitly typing many colons:
   ```python
   import numpy as np
   arr = np.arange(27).reshape(3, 3, 3)
   result = arr[..., 0]  # Equivalent to arr[:, :, 0]
   ```
   This slices all elements along all earlier dimensions and selects index 0 of the last dimension.

3. **Unique Default or Sentinel Value**
   Sometimes `...` is used as a unique sentinel or default parameter value to distinguish "no argument provided" from `None` or other possible values:
   ```python
   def greet(name, greeting=...):
       if greeting is ...:
           greeting = "Hello"
       print(f"{greeting}, {name}!")
   ```

4. **Representation of Recursive or Circular References**
   When you print objects with circular references (e.g., a list containing itself), Python shows `[...]` in the output to avoid infinite recursion. This is a display convention, not an actual replacement of the object.

5. **Type Hinting Uses**
   The ellipsis appears in some advanced type hinting scenarios, for example, to indicate variable-length tuples (`Tuple[int, ...]`) or unspecified arguments in callables.

### Summary Table of Ellipsis Roles

| Role                                | Description & Example                         |
|------------------------------------|-----------------------------------------------|
| Placeholder for unimplemented code | Use `...` inside functions/classes instead of `pass` |
| Extended slicing in multidimensional arrays | `arr[..., 0]` to slice across many dimensions |
| Unique default/sentinel value      | Default function arguments different from None |
| Circular reference representation  | Shown as `[...]` in printed recursive data to avoid infinite loops |
| Type hinting                       | Used in tuple or callable annotations to indicate variable length |

In essence, `...` is a versatile built-in object intended to serve as a **syntactic convenience and semantic marker** that gains meaning from context, especially in libraries like NumPy and in type hinting.

If you are seeing it "everywhere" ambiguously, it’s because its meaning tends to be **context-dependent**, and it’s often used as a compact placeholder or slicing shorthand.

Let me know if you want detailed examples or specific usage in any particular context like NumPy, type hints, or function placeholders.

---
References:
1. Scaler: Uses of Ellipsis in Python[1]
2. Real Python: When to Use Ellipsis[2]
3. Python Land: Python Ellipsis Explained[4]
4. Explaining Ellipsis in Function Defaults and NumPy Slicing[3]

[1] https://www.scaler.com/topics/python-ellipsis/
[2] https://realpython.com/python-ellipsis/
[3] https://www.kdnuggets.com/exploring-pythons-ellipsis-more-than-just-syntax-sugar
[4] https://python.land/python-ellipsis
[5] https://stackoverflow.com/questions/772124/what-does-the-ellipsis-object-do
[6] https://www.reddit.com/r/learnpython/comments/12pqfz5/til_about_ellipses_in_python/
[7] https://gist.github.com/promto-c/f51cc2c0eb8742ce5cc3e65601df2deb
[8] https://www.geeksforgeeks.org/python/what-is-three-dots-or-ellipsis-in-python3/
[9] https://mbizsoftware.com/to-what-purpose-does-a-python-ellipsis-perform/

Annotating Positional Only and Variadic Parameters #

Consider this example:

1
2
3
4
5
6
7
8
9
from typing import Optional

def tag(
        name: str,
        /, # this marks the preceding arguments as positional only
        *content: str, # for the arbitratry positional params
        class_: Optional[str] = None,
        **attrs: str, # this is for the variadic named kwargs
      ) -> str:

So what we see here is that:

for the arbitrary positional params, it’s all fixed to str
for the kwargs, it’s **atrs: <mytype> where mytype would be the type of the value and the key will be str

Imperfect Typing and Strong Testing #

Some limitations to the type hinting capabilities:

unsupported: useful things like argument unpacking #

handy features can’t be statically checked; for example, argument unpack‐ ing like config(**settings).

unsupported: advanced features like properties, descriptors, meta things #

properties, descriptors, metaclasses, and metaprogram‐ ming in general are poorly supported or beyond comprehension for type checkers.

since can’t hint data constraints, type hinting doesn’t help with correctness of business logic #

Common data constraints cannot be expressed in the type system—even simple ones. For example, type hints are unable to ensure “quantity must be an integer > 0” or “label must be a string with 6 to 12 ASCII letters.” In general, type hints are not help‐ ful to catch errors in business logic.

conclusion - robustness of python codes comes mainly from quality unit-testing #

concluded: “If a Python program has adequate unit tests, it can be as robust as a C++, Java, or C# program with adequate unit tests (although the tests in Python will be faster to write).”

Chapter Summary #

Protocol and how it enables static duck typing is useful because it’s very ‘pythonic’ #

in many ways, type hinting is very unPythonic, but typing.Protocol fits nicely..
protocols should be seen as the bridge between python’s duck-typed core and the nominal typing that allows static type checkers to catch bugs.

3.8, Protocol is not widely used yet—but it is hugely important. Protocol enables static duck typing: the essential bridge between Python’s duck-typed core and the nominal typing that allows static type checkers to catch bugs.

using the term “generics” in python is kinda funny #

because it’s actually doing the opposite by tying-down exactly what the types can be instead of keeping it more “generic”

Generics or Specifics? From a Python perspective, the typing usage of the term “generic” is backward. Com‐ mon meanings of “generic” are “applicable to an entire class or group” or “without a brand name.” Consider list versus list[str]. The first is generic: it accepts any object. The sec‐ ond is specific: it only accepts str. The term makes sense in Java, though. Before Java 1.5, all Java collections (except the magic array) were “specific”: they could only hold Object references, so we had to cast the items that came out of a collection to

for a deeper dive into variance #

see the PEP for it

we can do runtime type-checking in python #

for advanced versions of runtime typechecking (i.e. anything beyond doing guards for isinstance) we have to rely on some libraries

See this

  Yes, you can perform runtime type checking in Python, and there are several ways to achieve this. Here’s a detailed overview based on the search results:

  ### Runtime Type Checking in Python

  1. **Dynamic Typing**: Python is a dynamically typed language, meaning that types are determined at runtime. This allows for flexibility but can lead to type-related errors if not managed properly.

  2. **Type Checking Tools**:
     - **`isinstance()`**: The built-in function `isinstance()` can be used to check if an object is an instance of a specific class or a tuple of classes. However, it does not support checking against complex type annotations defined in the `typing` module.
       ```python
       x = 5
       if isinstance(x, int):
           print("x is an integer")
       ```

  3. **TypeGuard Library**:
     - Libraries like **TypeGuard** provide runtime type checking capabilities that can validate function arguments and return types based on type annotations.
     - You can use the `@typechecked` decorator to automatically check types at runtime.
       ```python
       from typeguard import typechecked

       @typechecked
       def add(a: int, b: int) -> int:
           return a + b

       add(1, 2)  # Works fine
       add(1, "2")  # Raises TypeError at runtime
       ```

  4. **Other Libraries**:
     - **`runtime-type-checker`**: This library allows you to perform type checks at runtime using decorators or direct function calls like `check_type()`.
       ```python
       from runtime_type_checker import check_type

       check_type("hello", str)  # OK
       check_type(123, str)      # Raises TypeError
       ```

  5. **Pydantic and Marshmallow**: These libraries are often used for data validation and serialization, providing runtime checks based on type annotations.
     - **Pydantic Example**:
       ```python
       from pydantic import BaseModel

       class User(BaseModel):
           name: str
           age: int

       user = User(name="Alice", age=30)  # Valid
       user = User(name="Alice", age="30")  # Raises validation error
       ```

  ### Summary of Key Points

  - **Type Annotations**: While Python supports type hints (introduced in PEP 484), they are primarily for static analysis tools like `mypy`. By default, Python does not enforce these types at runtime.
  - **Runtime Type Checking Libraries**: To enforce type checks during execution, you can use libraries such as TypeGuard, runtime-type-checker, Pydantic, or Marshmallow.
  - **Flexibility vs. Safety**: The dynamic nature of Python provides flexibility but may lead to errors that can be caught with proper type checking mechanisms.

  ### Conclusion

  In summary, while Python does not enforce type hints at runtime by default, you can implement runtime type checking using various libraries designed for this purpose. These tools help ensure that your code adheres to the expected types defined in your annotations, providing an additional layer of safety in your applications.

  Citations:
  [1] https://stackoverflow.com/questions/43646823/python-3-5-checking-type-annotation-at-runtime
  [2] https://github.com/agronholm/typeguard
  [3] https://pypi.org/project/runtime-type-checker/
  [4] https://testdriven.io/blog/python-type-checking/
  [5] https://discuss.python.org/t/proposal-for-enhancing-runtime-type-checking-in-python-using-pattern-matching-and-type-hints-in-function-definitions/59391
  [6] https://www.datacamp.com/tutorial/type-checking-in-python
  [7] https://www.infoworld.com/article/2268917/get-started-with-python-type-hints.html
  [8] https://realpython.com/python-type-checking/

``covers runtime type checking validation.''