Notes on "Fluent Python"

My unedited org-noter notes from the classic book “Fluent Python – Clear, Concise, and Effective Programming” by Luciano Ramalho.

The notes for this are messy, sorry about that. There are some chapters I could not get time to finish so they are left as TODOs for now.

Outline and Notes

Each chapter’s summary page is really useful. We should always start with the summary if we were to review these topics in the future, having already read them at least once before.

The things that are useful and I want to create a habit for, I write a coment with the word: “TO_HABIT” so that we can search it easily.

Part I. Data Structures

Chapter 1. The Python Data Model

Seeing python as a “framework”

This gives us some use cases / purpose for implementing special methods to interface w python as a “framework”

the special methods are dunder methods

We implement special methods when we want our objects to support and interact with fundamental language constructs such as: • Collections • Attribute access • Iteration (including asynchronous iteration using async for) • Operator overloading • Function and method invocation • String representation and formatting • Asynchronous programming using await • Object creation and destruction • Managed contexts using the with or async with statements

What’s New in This Chapter

A Pythonic Card Deck

this is a demonstrative example on how we can adapt to the “interface” for the “framework” that is python.

Class Composition and how Delegation pattern in the data model helps

our getitem delegates to the [] operator of self._cards, our deck automatically supports slicing. Here’s

• The use of base classes allows OOP benefits for us such as being able to delegate functionality.
• Delegation is different from forwarding
• this python example is closer to the concept of “forwarding” actually

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

Yes, the concept of "delegation" referenced in *Fluent Python* when discussing how `__getitem__` delegates to the `[]` operator of `self._cards` is directly related to a general programming language principle—specifically, the **delegation pattern** in object-oriented programming.

## Delegation in Programming

**Delegation** is a software design pattern where an object (the *delegator*) passes responsibility for a particular task to another object (the *delegate*) that is better suited to handle it. The delegator maintains the context and control, but leverages the delegate's specialized capabilities to fulfill certain operations[1][2][3].

In the context of Python's data model, when you write:

```python
def __getitem__(self, position):
    return self._cards[position]
```

You are *delegating* the item access operation to the underlying list (`self._cards`), which already knows how to handle slicing and indexing. This is a classic example of the delegation pattern: your class does not implement slicing logic itself, but instead relies on a delegate (the list) to do so.

## Delegation vs. Forwarding

- **Delegation**: The delegator object maintains the context and can pass itself (or relevant context) to the delegate. In some languages, this means that `self` (or `this`) inside the delegate's methods refers back to the original object, not the delegate[3][2].
- **Forwarding**: The delegator simply passes the request to the delegate, and the delegate operates in its own context. The delegate does not know about the original sender[3][2].

In Python, the example above is technically closer to *forwarding* because the list (`self._cards`) does not receive any special context from the delegator. However, in the broader sense, this is often referred to as delegation in Python literature.

## Delegation Pattern in Object-Oriented Programming

The delegation pattern allows composition to achieve the same code reuse as inheritance. Instead of inheriting behavior from a parent class, you compose your class with instances of other classes (the delegates) and forward certain operations to them[2][1]. This makes your code more modular and flexible.

## Language Support

Some languages have built-in support for delegation (e.g., Kotlin's `by` keyword), but in Python, delegation is typically implemented manually by calling methods on composed objects[2].

## Summary Table

| Concept      | Description                                                                 | Example in Python                |
|--------------|-----------------------------------------------------------------------------|----------------------------------|
| Delegation   | Object passes responsibility to another, often with context preservation    | `self._cards[position]` (loosely)|
| Forwarding   | Object passes request to another, delegate operates in its own context      | `self._cards[position]` (strict) |

## Key Takeaway

Delegation is a general programming pattern for code reuse and modularity, where one object relies on another to perform certain tasks. In Python, this is often seen when a class delegates an operation to an internal object, as in the `__getitem__` example from *Fluent Python*[1][2][3].

[1] https://dev.to/kalashin1/delegation-4h29
[2] https://handwiki.org/wiki/Delegation_pattern
[3] https://en.wikipedia.org/wiki/Delegation_(object-oriented_programming)
[4] https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/delegates/
[5] https://en.wikipedia.org/wiki/Delegation_(computing)
[6] https://www.cs.unc.edu/~dewan/comp401/current/Class%20Notes/22_Delegation_Notes.pdf
[7] https://developer.apple.com/library/archive/documentation/General/Conceptual/CocoaEncyclopedia/DelegatesandDataSources/DelegatesandDataSources.html
[8] https://langdev.stackexchange.com/questions/3586/are-delegates-still-relevant-in-modern-c
[9] https://www.reddit.com/r/Kotlin/comments/1928bzd/can_someone_explain_me_the_concept_of_delegations/
[10] https://perso.limsi.fr/pointal/python:delegator

How Special Methods Are Used

• NOTE: built-ins that are variable sized under the hood have an ob_size attribute that holds the size of that collection. This makes it faster to call len(my_object) since it’s not really a function call, the interpreter just reads off the pointer.

• 1. Emulating Numeric Types
  • it’s all about implementing the number-class related dunder methods, then anything can behave like a number

• 2. String Representation __repr__
  • repr is different from string in the sense that it’s supposed to be a visual representation of the creation of that object. Therefore, it should be unambiguous, and if possible, match source code necessary to recreate the represented object
  • repr is not really for display purposes, that’s what str builtin is for
  • implement the special function repr first then str

• 3. Boolean Value of a Custom Type

  • default: forward to bool() elif len()

    By default, instances of user-defined classes are considered truthy, unless either bool or len is implemented. Basically, bool(x) calls x._bool_() and uses the result. If bool is not implemented, Python tries to invoke x._len_(), and if that returns zero, bool returns False. Otherwise bool returns True.

• 4. Collection API

  The collections api is new, and it unifies the 3 following interfaces: • Iterable to support for, unpacking, and other forms of iteration • Sized to support the len built-in function • Container to support the in operator

  There’s no need to inherit from these ABCs specifically, as long as the dunder methods are implemented then it’s considered as satisfying the ABC

  • Specialisations of Collection

    Three very important specializations of Collection are: • Sequence, formalizing the interface of built-ins like list and str • Mapping, implemented by dict, collections.defaultdict, etc. • Set, the interface of the set and frozenset built-in types

    I want to use the vocabulary here when describing what primitives I want to use.

  • python dicts are “ordered” in the sense that the insertion order is preserved

    • there’s nothing else we can do about the ordering property (e.g. manipulating[rearranging] the order and such)

Overview of Special Methods

• there’s a bunch, the latest ones are more on the async support side, they will be covered throughout the book

Why len Is Not a Method

“Practicality beats purity”.

• there’s no method call for len(x) when x is a CPython built-in because it’s a direct read of a C-struct
• for custom objects, we can implement the dunder method __len__
• it kinda looks like a functional style (since len is a fn) in a OOP-styled language. To reconcile this, we can think of abs and len as unary functions!

Chapter Summary

Further Reading

Python’s DataModel can be seen as a MetaObject Protocol

Metaobjects The Art of the Metaobject Protocol (AMOP) is my favorite computer book title. But I mention it because the term metaobject protocol is useful to think about the Python Data Model and similar features in other languages. The metaobject part refers to the objects that are the building blocks of the language itself. In this context, protocol is a synonym of interface. So a metaobject protocol is a fancy synonym for object model: an API for core language constructs.

Chapter 2. An Array of Sequences

What’s New in This Chapter

Overview of Built-in Sequences

• two factors to group sequences by:
  1. by container (heterogeneous) / flat (homogeneous) sequences
     1. Container sequences: can be heterogeneous
        • holds references (“pointers”)
     2. Flat sequences: are homogeneous
        • holds values
  2. by mutability / immutability
• things like generators can be seen in the context of sequences themselves “To fill up sequences of any type”

• Mem Representation for Python Objects: have a header (with metadata) and value

  example of meta fields (using float as a reference):

  1. refcount
  2. type
  3. value

  Every Python object in memory has a header with metadata. The simplest Python object, a float, has a value field and two metadata fields: • ob_refcnt: the object’s reference count • ob_type: a pointer to the object’s type • ob_fval: a C double holding the value of the float On a 64-bit Python build, each of those fields takes 8 bytes. That’s why an array of floats is much more compact than a tuple of floats: the array is a single object holding the raw values of the floats, while the tuple consists of several objects—the tuple itself and each float object contained in it.

List Comprehensions and Generator Expressions

• List Comprehensions and Readability

  • a loop has generic purpose, but a listcomp’s purpose is always singular: to build a list
  • we should stick to this purpose and not introduce abuse mechanisms like adding in side-effects from listcomp evaluations
  • List comprehensions build lists from sequences or any other iterable type by filtering and transforming items.

  • Scope: listcomps have a local scope, use walrus operator to expand the scope to its outer frame

    ``Local Scope Within Comprehensions and Generator Expressions''

    if that name is modified using global or nonlocal, then the scope is accordingly set

    defines the scope of the target of := as the enclos‐ ing function, unless there is a global or nonlocal declaration for that target.

• Listcomps Versus map and filter

• Cartesian Products

  This is the part where we have more than one iterable within the listcomp

• Generator Expressions

Tuples Are Not Just Immutable Lists

The immutable list part is definitely one of the main features.

It should also be seen as a nameless record.

• Tuples as Records

  • some examples of tuple unpacking:
    1. the loop constructs automatically support unpacking, we can assign vars even for each iteration of the loop
    2. the % formatting operator will also unpack values within the tuple when doing string formats

• Tuples as Immutable Lists

  2 benefits:

  1. clarity: the length of tuple is fixed thanks to its immutability
  2. performance: memory use is a little better, also allows for some optimisations

  • Warning: the immutability is w.r.t references contained within the tuple, not values

    So tuples containing mutable items can be a source of bugs Also, unhashable tuple => can’t be inserted as a dict key or set

  • Tuple’s Performance Efficiency Reasons

    Tuples are more efficient because:

    1. bytecode: tuple has simpler bytecode required: Python compiler generates bytecode for a tuple constant in one operation; but for a list literal, the generated bytecode pushes each element as a separate constant to the data stack, and then builds the list.
    2. constructor: tuple construction from existing doesn’t need any copying, it’s the same reference:
       • the list constructor returns a copy of a given list if its list(l)
       • tuple constructor returns a reference to the same t if we do tuple(t) (because they’re immutable anyway so why not same reference)
    3. amortisation: tuple, since fixed size, doesn’t need to account for future size changes by amortising that operation
    4. no extra layer of indirection The references to the items in a tuple are stored in an array in the tuple struct,while a list holds a pointer to an array of references stored elsewhere. The indirection is necessary because when a list grows beyond the space currently allocated, Python needs to reallocate the array of references to make room. The extra indirection makes CPU caches less effective.

• Comparing Tuple and List Methods

Unpacking Sequences and Iterables

• safer extraction of elements from sequences
• works with any iterable object as the datasource, including iterators.
  • for the iterable case, as long as the iterable yields exactly one item per variable in the receiving end (or * is used to do a glob capture)

• Parallel assignment

  This is the multi-name assignments that we do, and how involves sequence unpacking

  most visible form of unpacking is parallel assignment; that is, assigning items from an iterable to a tuple of variables, as you can see in this example: >>> lax_coordinates = (33.9425, -118.408056) >>> latitude, longitude = lax_coordinates # unpacking >>> latitude

• Using * to Grab Excess Items

  • classic case is the use of the grabbing part for varargs
  • context of parallel assignment, the * prefix can be applied to exactly one variable, but it can appear in any position

• Unpacking with * in Function Calls and Sequence Literals

  • the use of the unpacking operator is context-dependent, so in the context of function calls and the creation of sequences, they can be used multiple times. In the context of parallel asisgnment, it’s a singular use (else there’s going to be ambiguity on how to partition values in the sequence)

• Nested Unpacking

• GOTCHA: single-item tuple syntax may have silent bugs if used improperly

  Both of these could be written with tuples, but don’t forget the syntax quirk that single-item tuples must be written with a trailing comma. So the first target would be (record,) and the second ((field,),). In both cases you get a silent bug if you forget a comma.

Pattern Matching with Sequences

Pattern matching is an example of declarative programming: the code describes “what” you want to match, instead of “how” to match it. The shape of the code follows the shape of the data, as Table 2-2 illustrates.

• here’s the OG writeup for structural pattern matching. Some points from it:
  • Therefore, an important exception is that patterns don’t match iterators. Also, to prevent a common mistake, sequence patterns don’t match strings.
  • the matching primitives allow us to use guards on the match conditions (see here)
  • there’s support for defining sub-patterns like so:

    1	case (Point(x1, y1), Point(x2, y2) as p2): ...

• here’s a more comprehensive tutorial PEP 636 - Structural Pattern Matching

• Pattern-matching is declarative

  Pattern matching is an example of declarative programming: the code describes “what” you want to match, instead of “how” to match it. The shape of the code fol‐ lows the shape of the data, as Table 2-2 illustrates.

• python’s match goes beyond just being a switch statement because it supports destructuring similar to elixir

  • random thought: this features is really useful if we were to write out a toy interpreter for some source code. Here’s lis.py

  On the surface, match/case may look like the switch/case statement from the C lan‐ guage—but that’s only half the story.4 One key improvement of match over switch is destructuring—a more advanced form of unpacking. Destructuring is a new word in the Python vocabulary, but it is commonly used in the documentation of languages that support pattern matching—like Scala and Elixir. As a first example of destructuring, Example 2-10 shows part of Example 2-8 rewrit‐ ten with match/case.

• class-patterns gift us the ability to do runtime type checks

  1	case [str(name), _, _, (float(lat), float(lon))]:

  • the constructor-like syntax is not a constructor, it’s a runtime check

  • the names (name, lat, lon) are binded here and are available for referencing thereafter within the codeblock

  • this is really interesting, it’s in the context of patterns that the syntax does runtime type checking and the code does

  The expressions str(name) and float(lat) look like constructor calls, which we’d use to convert name and lat to str and float. But in the context of a pattern, that syntax performs a runtime type check: the preceding pattern will match a four-item sequence in which item 0 must be a str, and item 3 must be a pair of floats. Additionally, the str in item 0 will be bound to the name variable, and the floats in item 3 will be bound to lat and lon, respectively. So, although str(name) borrows the syntax of a constructor call, the semantics are completely different in the context of a pattern. Using arbitrary classes in patterns is covered in “Pattern Matching Class Instances” on page 192.

• Pattern Matching Sequences in an Interpreter

  • it’s interesting how the python 2 code was described as “a fan of pattern matching” because it matches on the first element and then the tree of control flow paths does their job, so it’s really like a switch

  • this switch-like pattern-matching style is something abstract even more so than in it’s concrete programming language implementation that we have been discussing so far

  • the catch-all is used for error-handling purposes here. In general there should always be a fallthrough case instead of going for no-ops which will end up being more silent

Slicing

• Why Slices and Ranges Exclude the Last Item

  this refers to the fact that one end of the range is closed (inclusive) and the other is open (exclusive).

  • easy to calculate lengths
  • easy to split / partition without creating overlaps

• Slice Objects

  • useful to know this because it lets you assign names to slices, like spreadsheets allow the naming of cell-ranges

• Multidimensional Slicing and Ellipsis

  This is more useful in the context of numpy lib, the book doens’t include examples here for the python stdlib

  • built-ins are single dim, except for memoryview Except for memoryview, the built-in sequence types in Python are one-dimensional, so they support only one index or slice, and not a tuple of them.
  • Multiple indexes or slices get passed in as tuples a[i,j] is evaluated as a.__getitem__((i,j)) e.g. numpy multi-dim array accesses
  • ellipsis class is a singleton, the sole object being Elipsis
    • a similar case is bool class and True, False
  • so in numpy, if x is a four-dimensional array, x[i, ...] is a shortcut for x[i, :, :, :,]

• Assigning to Slices

  Applies to mutable sequences.

  • Gotcha: when LHS of assignment is slice, the RHS must be iterable

    In the example below, we’re trying to graft some sequence to another. With that intent, we can only graft an iterable onto another sequence, not a single element. Hence, the requirement that the RHS must be iterable.

    1 2 3 4 5 6 7 8 9 10 11 12

    l = list(range(10)) try: # so this is wrong: l[2:5] = 100 except: print("this will throw an error, we aren't passing in an iterable for the grafting.") finally: # and this is right l[2:5] = [100] print(l)

Using + and * with Sequences

• both + and - create new objects without modding their operands

• Building Lists of Lists

  • Gotcha: Pitfall of references to mutable objects – using a * n where a contains sequence of mutable items can be problematic

    Actually applies to other mutable sequences as well, in this case it’s just a list that we’re using

    Just be careful what the contained element’s properties are like.

    1 2 3 4 5 6 7 8 9 10 11

    my_mutable_elem = ['apple', 'banana'] print(f"my mutable elem ref: {id(my_mutable_elem)}") list_of_lists = [ my_mutable_elem ] * 2 print(f"This creates 2 repeats \n{list_of_lists}") print(f"(first ref, second ref) = {[id(elem) for elem in list_of_lists]}") list_of_lists[0][0] = 'strawberry' print(f"This mods all 2 repeated refs \n{list_of_lists}") print(f"(first ref, second ref) = {[id(elem) for elem in list_of_lists]}")

    Here’s the same gotcha using tic-tac-toe as an example:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

    good_board = [['_'] * 3 for i in range(3)] bad_board = [['_'] * 3] * 3 print(f"BEFORE, the boards look like this:\n\ \tGOOD Board:\n\ \t{ [row for row in good_board] }\n\ \tBAD Board:\n\ \t{ [row for row in bad_board] }\n") # now we make a mark on the boards: good_board[1][2] = 'X' bad_board[1][2] = 'X' print(f"AFTER, the boards look like this:\n\ \tGOOD Board:\n\ \t{ [row for row in good_board] }\n\ \tBAD Board:\n\ \t{ [row for row in bad_board] }\n")

• Augmented Assignment with Sequences

  This refers to the in-place versions of the sequence operators. in ==, there are 2 cases:

  • Case A: Identity of a changes

    • the dunder method __iadd__ was not available for use
    • so a + b had to be evaluated and stored as a new id
    • and that id was then referenced by a as part of the new assignment

  • Case B: Identity of a does not change

    • this would mean that a is actually mutated in-place
    • it would have used the dunder method __iadd__

  In other words, the identity of the object bound to a may or may not change, depending on the availability of __iadd__.

  In general, for mutable sequences, it is a good bet that __iadd__ is implemented and that += happens

  • doing += for repeated concats of immutable sequences is inefficient

    however, str contacts have been optimised in CPython, it’s alright to do that in CPython. Extra space would have been allocated to amortise the new space allocations.

• A += Assignment Puzzler

  • Learnings!

    I take three lessons from this:

    • Avoid putting mutable items in tuples.

    • Augmented assignment is not an atomic operation—we just saw it throwing an exception after doing part of its job.

    • Inspecting Python bytecode is not too difficult, and can be helpful to see what is going on under the hood.

  • Example

    it’s a peculiarity in the += operator!

    Learnings:

    I take three lessons from this:

    • Avoid putting mutable items in tuples.

    • Augmented assignment is not an atomic operation—we just saw it throwing an exception after doing part of its job.

    • Inspecting Python bytecode is not too difficult, and can be helpful to see what is going on under the hood.

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

    t = (1,2, [30, 40]) print(t) try: t[2] += [50, 60] except: print("LMAO complaints") finally: print(t) try: t[2].extend([90, 100]) except: print("this won't error out though") finally: print(t)

list.sort Versus the sorted Built-In

• in-place functions should return None as a convention

  There’s a drawback to this: we can’t cascade calls to this method

• python’s sorting uses timsort!

Managing Ordered Sequences with bisect (extra ref from textbook)

When a List Is Not the Answer

• Arrays: best for containing numbers

  • an array of float values does not hold full-fledged float instances, but only the packed bytes representing their machine values—similar to an array of double in the C language.
  • examples:
    1. typecode b => byte => 8 bits over signed and unsigned regions ==> [-128, 127] range of representation
  • for special cases of numeric arrays for bin data (e.g. raster images), bytes and bytearray types are more appropriate!

• Memory Views

  • Examples

    • id vs context

      The learning from this is that the memoryview objects and the memory that they provide a view of are two different regions of memory. id vs context.

      So here, m2, m3 and all have different id references, but the memory region that they give a view of is all the same.

      That’s why we can mutate using one memory view and every other view also reflects that change.

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

      from array import array # just some bytes, the sequence is buffer-protocol-adherent octets = array("B", range(6)) print(octets) # builds a new memoryview from the array m1 = memoryview(octets) print(m1) # exporting of a memory view to a list, this creates a new list (a copy!) print(m1.tolist()) # builds a new memoryview, with 2 rows and 3 columns m2 = m1.cast('B', [2,3]) print(m2) print(m2.tolist()) m3 = m1.cast('B', [3,2]) print(m3) print(m3.tolist()) # overwrite byte m2[1,1] = 22 # overwrite byte m3[1,1] = 33 print(f"original memory has been changed: \n\t{octets} ") print(f"m1 has been changed:\n\t { m1.tolist() }") print(f"m2 has been changed:\n\t { m2.tolist() }") print(f"m3 has been changed:\n\t {m3.tolist()}")

    • corruption

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

      from array import array from sys import byteorder print(byteorder) numbers = array('h', [-2,-1,0,1,2]) memv = memoryview(numbers) print(len(memv)) print(memv[0]) # cast the half as a byte, so the resultant sequence will have double the elements: memv_oct = memv.cast('B') # the numbers are stored in little endian format print(memv_oct.tolist()) # so -2 as a 2-byte signed short will be (little endian binary) 0xfe 0xff (254, 255) # so we get: # -2: 0xfe 0xff (254, 255) # -1: 0xff 0xff (255, 255) # 0: 0x00 0x00 (0, 0) # 1: 0x01 0x00 (1, 0) # 2: 0x02 0x00 (2, 0) # asisgns the value of 4 to byte-offset 5 memv_oct[5] = 4 print( numbers ) # so this change is to the 2nd byte of the third element of numbers # byte index 5 is the high byte (since it's little endian so bytes are low -> high) # so the 3rd element is now [0, 0x0400] # = a + (b*256) = 0 + (4 * 256) is 1024 in decimal # NOTE: Note the change to numbers: a 4 in the most significant byte of a 2-byte unsigned # integer is 1024.

  • Extra: “Parsing binary records with struct”

    Here’s the reference.

    Some takeaways:

    • Proprietary binary records in the real world are brittle and can be corrupted easily. examples:
      1. string parsing: paddings, null terminated, size limits?
      2. endianness problem: what byteorder was used for representing integers and floats (CPU-architecture-dependent)?
    • always explore pre-built solutions first instead of building yourself:
      • for data exchange, pickle module works great, but have to ensure python versions align since the default binary formats may be different. Reading a pickle also may run arbitrary code.
    • if the binary exchange uses multiple programming languages, standardise the serialisation. Serial forms:
      1. multi-platform binary serialisation formats:
         1. MessagePack
         2. ProtocolBuffers
      2. JSON

  • bot assisted concept mapping

    Here’s a bot-assisted concept map between unix mmap and memoryviews:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

    Memory mapping a file is a powerful technique that allows access to file data as if it were in memory, and the concepts connect naturally between the Unix world (via `mmap` system calls) and Python (via the `mmap` module and `memoryview` objects). **Unix World: mmap** - **Definition:** The Unix `mmap` system call maps files or devices into a process's address space, enabling file I/O by reading and writing memory. This is efficient for large files because data is loaded on demand, and multiple processes can share the same mapped region[1]. - **Usage:** After opening a file, `mmap` associates a region of virtual memory with the file. Reading and writing to this memory behaves as if you were reading and writing to the file itself. The system manages when data is actually read from or written to disk, often using demand paging[1]. - **Types:** Both file-backed (mapping a file) and anonymous (not backed by a file, similar to dynamic allocation) mappings are supported. Shared mappings allow interprocess communication, while private mappings isolate changes to one process[1]. **Python World: mmap Module** - **Definition:** Python’s `mmap` module provides a high-level interface to memory-mapped files, closely mirroring the Unix `mmap` functionality. You must provide a file descriptor (from `fileno()` or `os.open()`) and specify the mapping size and access mode[2][3]. - **Usage:** Memory-mapped file objects behave like both file objects and mutable byte arrays, allowing random access and slicing. You can read and write data by indexing or slicing, and you can seek through the file as if it were a standard file object[2][3]. - **Access Modes:** You can specify read-only, write-through (changes go directly to the file), or copy-on-write (changes are local)[4][5]. **Python World: memoryview** - **Definition:** The `memoryview` object provides a safe way to access the internal data of an object that supports the buffer protocol, such as `bytes`, `bytearray`, or a memory-mapped file object[6]. - **Usage:** With a `memoryview`, you can manipulate large data structures without copying them, which is especially useful for memory-mapped files. It allows you to pass a view of the data to functions or slice it efficiently, avoiding unnecessary memory duplication[6]. - **Connection to mmap:** When you create a memory-mapped file in Python, you can use `memoryview` to work with the data as a buffer, enabling efficient manipulation and sharing of large data sets. **Linking the Concepts** - **Memory Mapping:** Both Unix `mmap` and Python’s `mmap` module allow you to treat file data as if it were in memory, enabling efficient random access and sharing between processes[1][2][3]. - **Efficient Data Access:** Using `memoryview` in Python is analogous to working directly with the mapped memory region in Unix, as both avoid copying large chunks of data and allow efficient manipulation of file contents[6]. - **Interprocess Communication:** In Unix, shared memory mappings (`MAP_SHARED`) allow processes to communicate by reading and writing the same memory region. In Python, you can achieve similar effects by sharing a memory-mapped file object between processes[1][2]. - **Performance:** Both approaches leverage the operating system’s memory management to reduce I/O overhead and enable fast, random access to file data. **Summary Table** | Concept | Unix (`mmap`) | Python (`mmap` module) | Python (`memoryview`) | |------------------------|------------------------------|-----------------------------------|-------------------------------| | Purpose | Map files to memory | Map files to memory | View memory as buffer | | Access Method | System call | Module/object | Object | | Sharing | Shared/private mappings | Shared via file object | View of existing buffer | | Efficiency | Demand paging, no copy | Demand paging, no copy | No copy, efficient slicing | | Use Case | IPC, efficient file I/O | Efficient file I/O, IPC | Efficient data manipulation | By understanding these connections, you can leverage memory mapping for efficient file handling and data sharing across both Unix and Python environments. [1] https://en.wikipedia.org/wiki/Mmap [2] https://docs.python.org/3/library/mmap.html [3] https://github.com/python/cpython/blob/master/Doc/library/mmap.rst [4] https://pymotw.com/3/mmap/ [5] https://realpython.com/python-mmap/ [6] https://smart-spatial.com/data%20science/2017/09/22/MemoryView/ [7] https://stackoverflow.com/questions/63553692/how-to-use-memory-mapped-file-in-python-linux/63554607 [8] https://pymotw.com/3/mmap/index.html [9] https://unix.stackexchange.com/questions/712651/does-mmap-allow-creating-a-mapping-that-is-much-larger-than-the-amount-of-physic [10] https://deepaksood619.github.io/computer-science/operating-system/memory-mapping-mmap/ [11] https://www.cmi.ac.in/~madhavan/courses/prog2-2012/docs/python-3.2.1-docs-html/library/mmap.html [12] https://www.blopig.com/blog/2024/08/memory-mapped-files-for-efficient-data-processing/ [13] https://stackoverflow.com/questions/4991533/sharing-memory-between-processes-through-the-use-of-mmap/4991631 [14] https://documentation.help/Python-2.4/module-mmap.html [15] https://docs.python.org/3.4/library/mmap.html?highlight=mmap [16] https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-mmap-map-pages-memory [17] https://man7.org/linux/man-pages/man2/mmap.2.html [18] https://programmingappliedai.substack.com/p/what-is-mmap-in-linux-and-how-it [19] https://w3.cs.jmu.edu/kirkpams/OpenCSF/Books/csf/html/MMap.html [20] https://www.ibm.com/docs/ssw_ibm_i_74/apis/mmap.htm [21] https://www.unix.com/man-page/linux/3/mmap/ [22] https://stackoverflow.com/questions/47437481/python-can-i-get-a-memoryview-or-a-bytearray-mapping-to-a-mmap [23] https://docs.vultr.com/python/built-in/memoryview [24] https://pymotw.com/2/mmap/ [25] https://www.youtube.com/watch?v=ky1n6luzL3Y

  • “casting” in memoryview vs Elixir ecto “casting”

    The mental models are different.

    “casting” with memoryview is about interpreting memory as a different type, not about automatic type conversion or validation as in Elixir changesets. The term is used more loosely in Python, while in Elixir, it is a formal, declarative operation for data preparation.

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

    The use of the term **"cast"** in the context of Python `memoryview` is not the same as in Elixir changesets, even though both involve types or transformations. **Python `memoryview` and Casting** - **Casting in Python `memoryview`:** When people refer to "casting" with `memoryview`, they usually mean taking a slice of a memoryview or viewing the underlying buffer as a different type (sometimes using modules like `ctypes` or `numpy` to achieve type reinterpretation). However, the built-in `memoryview` object itself does not perform type conversion; it provides a view into the memory buffer of an object that supports the buffer protocol, such as `bytes`, `bytearray`, or a NumPy array[5][6][1]. - **Type Safety:** The `memoryview` object allows you to manipulate the data as if it were a sequence of bytes or another type (if the underlying buffer supports it), but it does not automatically convert the data to a different type. If you want to reinterpret the data as a different type (e.g., casting bytes to an integer array), you typically use additional libraries or manual conversion, not `memoryview` itself[5][1]. - **No Implicit Type Conversion:** The `memoryview` simply provides a window into the existing data, preserving the memory layout and avoiding data copying[6][8]. Any "casting" is more about how you interpret the bytes, not a formal declaration or transformation of types. **Elixir Changesets and Casting** - **Casting in Elixir Changesets:** In Elixir's Ecto changesets, "casting" refers to the explicit declaration of how data (often from a web form or external source) should be converted or validated into a specific type before being inserted into a database. This is a declarative process that ensures data integrity and type safety. - **Type Conversion:** Casting in Elixir changesets involves actual type conversion (e.g., from a string to an integer or date), and is a core part of the data validation and preparation process. **Comparison Table** | Context | Meaning of "Cast" | Type Conversion | Purpose | |------------------------|------------------------------------------|-----------------|--------------------------------| | Python `memoryview` | Viewing/reinterpreting memory as a type | No (manual only)| Efficient data access/manipulation | | Elixir changesets | Declaring/validating/converting types | Yes | Data validation, preparation | **Summary** In Python, "casting" with `memoryview` is about interpreting memory as a different type, not about automatic type conversion or validation as in Elixir changesets. The term is used more loosely in Python, while in Elixir, it is a formal, declarative operation for data preparation[5][6][1]. [1] https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html [2] https://stackoverflow.com/questions/71028367/how-to-make-a-custom-python-class-to-support-memoryview-conversion [3] https://stackoverflow.com/questions/48136122/how-do-i-cast-a-slice-of-memoryview-into-c-string-unsigned-char [4] https://docs.python.org/3/library/stdtypes.html [5] https://blog.finxter.com/converting-python-bytes-to-memoryview/ [6] https://realpython.com/ref/builtin-functions/memoryview/ [7] https://www.stratascratch.com/blog/understanding-data-types-in-python-with-examples/ [8] https://docs.python.org/3/c-api/memoryview.html

• NumPy

  • python’s Global Interpreter Lock (GIL) and how releasing it unlocks better parallelisation

    • it’s a mutex that protects access to python objects and therefore prevents multiple native threads from executing Python bytecode simultaneously within the same process.

      It was intended to be a simplification mechanism to make memory handling simpler but that also means having this mutex limits parallelism.

      Typical workarounds:

      1. multi-processing, separate processes, each with their own GIL

      2. offload CPU-intensive work to C-extensions or libs that release the GIL

    here’s a bot-written outline on it:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

    The **Global Interpreter Lock (GIL)** is a core mechanism in CPython, the reference implementation of Python, that ensures only one thread executes Python bytecode at a time, even on multi-core processors[2][4][5]. Here’s a detailed overview: ## **What Is the GIL?** - **Definition:** The GIL is a mutex (mutual exclusion lock) that protects access to Python objects, preventing multiple native threads from executing Python bytecode simultaneously within the same process[2][4][7]. - **Purpose:** It exists primarily to simplify CPython’s memory management, especially reference counting, which is not thread-safe by default. Without the GIL, concurrent access to Python objects could lead to race conditions and memory corruption[5][7]. ## **How Does the GIL Work?** - **Single Thread Execution:** Only one thread holds the GIL at any moment, meaning only one thread can execute Python code at a time, even if you have multiple threads running[2][4][6]. - **Thread Switching:** The interpreter periodically releases the GIL, allowing other threads to acquire it and execute Python code. This switching happens frequently, but it means that CPU-bound multithreaded Python programs do not benefit from multiple cores for parallel execution of Python code[2][4]. - **Non-Python Code:** Operations that do not require the Python interpreter (such as I/O or some C extensions like NumPy) can release the GIL, allowing other threads to run Python code or the process to use multiple cores for those operations[2][4]. ## **Why Does the GIL Exist?** - **Memory Management:** Simplifies reference counting and garbage collection by ensuring thread safety for Python objects[5][7]. - **C Extensions:** Makes it easier to write and use C extensions by providing a stable, single-threaded environment for their execution[1][3][7]. - **Implementation Simplicity:** Using a single lock is easier to implement and maintain than fine-grained locking for all Python objects[1][7]. ## **Implications of the GIL** - **Limited Parallelism:** The GIL prevents true parallel execution of Python code in multi-threaded programs, making it a bottleneck for CPU-bound tasks[2][4][5]. - **Workarounds:** For parallelism, Python developers often use multiprocessing (which uses separate processes, each with its own GIL) or offload CPU-intensive work to C extensions or libraries that release the GIL[1][4]. - **Performance Impact:** The GIL can degrade performance in multi-threaded, CPU-bound applications. However, for I/O-bound or single-threaded programs, its impact is minimal[2][4][6]. ## **Future of the GIL** - **Potential Removal:** The Python Steering Council has indicated support for PEP 703, which proposes making a version of CPython without the GIL. This could enable true multi-threaded parallelism in Python in the future[3]. - **Challenges:** Removing the GIL is complex due to backward compatibility and the reliance of many extensions on its guarantees[3][2]. ## **Summary Table** | Feature | Description | |------------------------|-----------------------------------------------------------------------------| | Purpose | Protect Python objects, simplify memory management, enable C extensions | | Execution Model | Only one thread executes Python bytecode at a time | | Impact on Parallelism | Limits CPU-bound parallelism in multi-threaded Python code | | Workarounds | Multiprocessing, C extensions, I/O-bound operations | | Future | Potential removal via PEP 703, but challenges remain | The GIL is a key part of Python’s design, balancing simplicity and safety with some limitations for parallel execution[2][4][5]. [1] https://en.wikipedia.org/wiki/Global_interpreter_lock [2] https://wiki.python.org/moin/GlobalInterpreterLock [3] https://developer.vonage.com/en/blog/removing-pythons-gil-its-happening [4] https://realpython.com/python-gil/ [5] https://dev.to/adityabhuyan/understanding-pythons-global-interpreter-lock-gil-and-its-impact-on-concurrency-2da6 [6] https://realpython.com/videos/global-interpreter-lock-overview/ [7] https://dev.to/ohdylan/understanding-pythons-global-interpreter-lock-gil-mechanism-benefits-and-limitations-4aha [8] https://www.pubnub.com/blog/understanding-pythons-global-interpreter-lock/

    NumPy and SciPy are formidable libraries, and are the foundation of other awesome tools such as the Pandas—which implements efficient array types that can hold non‐ numeric data and provides import/export functions for many different formats, like .csv, .xls, SQL dumps, HDF5, etc.—and scikit-learn, currently the most widely used Machine Learning toolset. Most NumPy and SciPy functions are implemented in C or C++, and can leverage all CPU cores because they release Python’s GIL (Global Interpreter Lock). The Dask project supports parallelizing NumPy, Pandas, and scikit-learn processing across clusters of machines. These packages deserve entire books about them.

• Deques and Other Queues

  • issues with list methods

    although we can use list as a stack / queue (by using .append() or .pop()). However, inserting and removing from the head of the list (the 0-idx end) is costly because the entire list must be shifted in memory => this is why just re-purposing lists is not a good idea.

  • Characteristics:

    1. when bounded, every mutation will adhere to the deque capacity for sure.
    2. hidden cost is that removing items from the middle of a deque is not fast
    3. append and popleft are atomic, so can be used for multi-threaded applications without needing locks:w

  • alternative queues in stdlib

    • asyncio provides async-programming focused queues

Chapter Summary

Further Reading

• “numpy is all about vectorisation” oprations on array elements without explicit loops

• More on Flat vs Container Sequences

  ``Flat Versus Container Sequences''

Chapter 3. Dictionaries and Sets

What’s New in This Chapter

Extra: Internals of sets and dicts internalsextra

This info is found in the fluentpython website. It considers the strengths and limitations of container types (dict, set) and how it’s linked to the use of hash tables.

• Running performance tests

  • the trial example of needle in haystack has beautiful ways of writing it

    1 2 3 4

    found = 0 for n in needles: if n in haystack: found += 1

    when using sets, because it’s directly related to set theory, we can use a one-liner to count the needles that occur in the haystack by doing an intersection:

    1	found = len(needles & haystack)

  • This intersection approach is the fastest from the test that the textbook runs.

  • the worst times are if we use the list datastructure for the haystack

  • If your program does any kind of I/O, the lookup time for keys in dicts or sets is negligible, regardless of the dict or set size (as long as it fits in RAM).

• Hashes & Equality

  • the usual uniform random distribution assumption as the goal to reach for hashing functions, just described in a different way: to be effective as hash table indexes, hash codes should scatter around the index space as much as possible. This means that, ideally, objects that are similar but not equal should have hash codes that differ widely.

    • here’s the oficial docs on the hash function

  • A hashcode for an object usually has less info than the object that the hashcode is for.

    • 64-bit CPython hashcodes is a 64-bit number => \(2^{64}\) possible values
    • consider an ascii string of 10 characters (and that there are 100 possible values in ascii) => \(100^{10}\) which is bigger than the possible values for the hashcode.

    By the way it’s actually salted, there’s some nuances on how the salt is derived but it should be such that each shell has a particular salt.

  • The modern hash function is the siphash implementation

• Hash Implementation

  • each row in the table is traditionally a “bucket”. In the case of sets, it’s just a single item that the bucket will hold
  • For 64-bit CPython,
    • It’s a 64-bit hash code that points to a 64 bit pointer to the element value
    • so the table doesn’t need to keep track of indices, offsets work fine since they are fixed-width.
  • Also it keeps 1/3 extra space that gets doubled when encroached so there’s some amortisation happening there also.

• Hash Table Algo for sets

  

  • in the flowchart, notice that the first step includes the modulo operation, this is the reason why the insertion order is not preserved since the output of running the modulo on the hashvalues will not be in order, it will spread about.

  • on hash collisions, the probing can be done in various ways. CPython uses linear probing but also mitigates the harms of using linear probing: Incrementing the index after a collision is called linear probing. This can lead to clusters of occupied buckets, which can degrade the hash table performance, so CPython counts the number of linear probes and after a certain threshold, applies a pseudo random number generator to obtain a different index from other bits of the hash code. This optimization is particularly important in large sets.

  • the last step is to actually do an equality check on the value. this is why for something to be hashable, two dunder functions must be implemented: __hash__ and __eq__

• Hash table usage for dicts

  Dictionary implementation benefits from 2 memory optimisations. Here’s a summary of it:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

  Here’s a summary of the **two major memory optimizations** for modern Python dictionaries, as described in the referenced Fluent Python article: 1. **Key-Sharing Dictionaries (PEP 412)** - Introduced in Python 3.3, this optimization allows multiple dictionaries that share the same set of keys (such as instance `__dict__` for objects of the same class) to share a single "keys table." - Only the values are stored separately for each dictionary; the mapping from keys to indices is shared. - This greatly reduces memory usage for objects of the same type, especially when many objects have the same attributes[1]. 2. **Compact Dictionaries** - Modern Python dictionaries use a split-table design, separating the storage of keys and values from the hash table itself. - The hash table stores indices into a compact array of keys and values, rather than storing the full key-value pairs directly in the hash table. - This reduces memory overhead, improves cache locality, and keeps insertion order predictable and efficient[1]. **In summary:** - **Key-sharing dictionaries** save memory by sharing the key structure among similar dicts. - **Compact dicts** store keys and values in separate, dense arrays, minimizing wasted space and improving performance. [1] https://www.fluentpython.com/extra/internals-of-sets-and-dicts/ [2] https://www.geeksforgeeks.org/python/minimizing-dictionary-memory-usage-in-python/ [3] https://python.plainenglish.io/optimizing-python-dictionaries-a-comprehensive-guide-f9b04063467a [4] https://stackoverflow.com/questions/10264874/python-reducing-memory-usage-of-dictionary [5] https://labex.io/tutorials/python-how-to-understand-python-dict-memory-scaling-450842 [6] https://www.youtube.com/watch?v=aJpk5miPaA8 [7] https://www.reddit.com/r/pythontips/comments/149qlts/some_quick_and_useful_python_memory_optimization/ [8] https://www.tutorialspoint.com/How-to-optimize-Python-dictionary-access-code [9] https://labex.io/tutorials/python-how-to-understand-python-dictionary-sizing-435511 [10] https://www.joeltok.com/posts/2021-06-memory-dataframes-vs-json-like/ [11] https://www.linkedin.com/advice/0/what-strategies-can-you-use-optimize-python-dictionaries-fqcuf

  • Original implementation

    

    • there’s 3 fields to keep, 64 bits each
    • first two fields play the same role as they do in the implementation of sets. To find a key, Python computes the hash code of the key, derives an index from the key, then probes the hash table to find a bucket with a matching hash code and a matching key object. The third field provides the main feature of a dict: mapping a key to an arbitrary value

  • Optimisation 1: Compact implementation

    

    • there’s an indices table extra that has a smaller width (hence compact)
    • Raymond Hettinger observed that significant savings could be made if the hash code and pointers to key and value were held in an entries array with no empty rows, and the actual hash table were a sparse array with much smaller buckets holding indexes into the entries array

  • Optimisation 2: Key-Sharing Dictionary ⭐️

    The combined-table layout is still the default when you create a dict using literal syntax or call dict(). A split-table dictionary is created to fill the __dict__ special attribute of an instance, when it is the first instance of a class. The keys table is then cached in the class object. This leverages the fact that most Object Oriented Python code assigns all instance attributes in the __init__ method. That first instance (and all instances after it) will hold only its own value array. If an instance gets a new attribute not found in the shared keys table, then this instance’s __dict__ is converted to combined-table form. However, if this instance is the only one in its class, the __dict__ is converted back to split-table, since it is assumed that further instances will have the same set of attributes and key sharing will be useful.

• Practical Consequences

  • of how sets work

    1. need to implement the __hash__ and __eq__ functions
    2. efficient membership testing, the possible overheads is the small number of probing the might need to be done to find a matching element or an empty bucket
    3. Memory overhead:
       • an array of pointers is the most compact, sets have significant memory overhead. hash table adds a hash code per entry, and at least ⅓ of empty buckets to minimize collisions
    4. Insertion order is somewhat preserved but it’s not reliable.
    5. Adding elements to a set may change the order of other elements. That’s because, as the hash table is filled, Python may need to recreate it to keep at least ⅓ of the buckets empty. When this happens, elements are reinserted and different collisions may occur.

  • of how dicts work

    1. need to implement both the dunder methods __hash__ and __eq__
    2. key search almost as fast as element searches in sets
    3. Item ordering preserved in the entries table
    4. To save memory, avoid creating instance attributes outside of the init method. If all instance attributes are created in init, the dict of your instances will use the split-table layout, sharing the same indices and key entries array stored with the class.

Modern dict Syntax

• dict Comprehensions

• Unpacking Mappings

  • we can use the unpacking operator ** when keys aer all strings
  • if there’s any duplicates in the keys then the later entries will overwrite the earlier ones

• Merging Mappings with | (the union operator)

  • there’s an inplace merge |= and there’s a normal merge that creates a new mapping |
  • it’s supposed to look like the union operator and you’re doing an union on two mappings

Syntax & Structure: Pattern Matching with Mappings cool

• this will work with anything that is a subclass or virtual subclass of Mapping

• we can use the usual tools for this:

  1. can use partial matching

     1 2 3 4 5 6 7 8 9 10 11

     data = {"a": 1, "b": 2, "c": 3} match data: case {"a": 1}: print("Matched 'a' only") case {"a": 1, "b": 2}: print("Matched 'a' and 'b'") case _: print("No match") # in this case, the order of the cases matter, the first match is evaluated

  2. can capture keys using the **rest syntax

     1 2 3

     match data: case {"a": 1, **rest}: print(f"Matched 'a', rest: {rest}")

  3. can be arbitrarily deeply nested

     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

     data = { "user": { "id": 42, "profile": { "name": "Alice", "address": {"city": "Wonderland"} } } } match data: case { "user": { "profile": { "address": {"city": city_name} } } }: print(f"City is {city_name}") case _: print("No match")

• Keys in the pattern must be literals (not variables), but values can be any valid pattern, including captures, literals, or even further nested patterns

• Pattern matching works with any mapping type (not just dict), as long as it implements the mapping protocol

• Guards (if clauses) can be used to add extra conditions to a match.

• More on virtual sub-classes (and how it’s similar to mixins)

  • should be used when we can’t control the class (e.g. it’s an external module) but we want to adapt it

  • allows the indication that a class conforms to the interface of another – to adapt to multiple interfaces

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

  A **virtual subclass** in Python refers to a class that is recognized as a subclass of an abstract base class (ABC) without actually inheriting from it in the traditional sense. This mechanism is provided by the `abc` module and is achieved by *registering* a class as a virtual subclass of an ABC using the `register()` method[4][5][8]. ### Core Mental Model - **Traditional subclassing**: A class (the subclass) inherits from another (the superclass), forming a direct relationship. Methods and attributes are inherited, and `issubclass()` and `isinstance()` reflect this relationship[3]. - **Virtual subclassing**: A class is *declared* to be a subclass of an ABC at runtime, without modifying its inheritance tree or MRO (Method Resolution Order). This is done by calling `ABC.register(SomeClass)`. After registration, `issubclass(SomeClass, ABC)` and `isinstance(instance, ABC)` will return `True`, but `SomeClass` does not actually inherit from `ABC`[4][5][8]. ### Why Use Virtual Subclasses? - **Third-party integration**: If you want to treat classes from external libraries as conforming to your interface (ABC), but you cannot or do not want to modify their source code to inherit from your ABC, you can register them as virtual subclasses[1][8]. - **Interface compliance**: Virtual subclassing is a way to declare that a class “conforms to” an interface, even if it doesn’t inherit from it, as long as it implements the required methods (i.e., it follows the protocol)[2][5]. - **Decoupling**: It allows you to decouple interface definition (the ABC) from implementation, enabling more flexible and extensible designs. ### Example Suppose you have an ABC and an external class: ```python from abc import ABC class Car(ABC): def drive(self): pass class Tesla: def drive(self): print("Driving in Tesla") ``` You want to use `isinstance(obj, Car)` to check if an object can be driven, but `Tesla` does not inherit from `Car`. You can register it: ```python Car.register(Tesla) print(issubclass(Tesla, Car)) # True print(isinstance(Tesla(), Car)) # True ``` Now, `Tesla` is a *virtual subclass* of `Car`, even though it doesn't inherit from it[4][5][8]. ### Key Properties - **No inheritance**: Virtual subclasses do not inherit methods or properties from the ABC. Registration only affects `issubclass()` and `isinstance()` checks[4][8]. - **No MRO change**: The ABC does not appear in the virtual subclass’s MRO, so `super()` calls and method resolution are unaffected[4]. - **Runtime declaration**: Registration can be done at runtime, providing flexibility for dynamic systems[4][5]. ### Relationship to Other Python Typing Models | Model | Relationship Mechanism | Example Use Case | |-----------------------|-------------------------------|---------------------------------------------------| | Duck typing | Implements required interface | Any object with `drive()` method can be used | | Virtual subclass | Registered with ABC | External class made compatible with ABC interface | | Classical inheritance | Inherits from superclass | Subclass extends or customizes base functionality | Virtual subclassing sits between **duck typing** (no explicit relationship, just interface compliance) and **classical inheritance** (explicit parent-child relationship), allowing you to *declare* interface compliance without direct inheritance[1][6]. ### Reference Implementation The mechanism is built into the `abc.ABCMeta` metaclass, which provides the `register()` method for this purpose[4][5]. > "You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as 'virtual subclasses' – these and their descendants will be considered subclasses of the registering ABC by the built-in function, but the registering ABC won’t show up in their MRO nor will method implementations defined by the registering ABC be callable..."[4] ### Practical Implications - **Type checks**: Enables more expressive and flexible type checks in frameworks and libraries. - **Backward compatibility**: Allows legacy or third-party classes to be integrated into new type hierarchies without code changes. - **Protocol enforcement**: Encourages protocol-oriented design in a language that is otherwise dynamically typed. In summary, a **virtual subclass** in Python is a class that, through explicit registration, is treated as a subclass of an abstract base class for type-checking purposes, without requiring direct inheritance or MRO modification[4][5][8]. This is a powerful tool for interface-oriented design and decoupled architectures. [1] https://stackoverflow.com/questions/51666120/whats-the-usage-of-a-virtual-subclass [2] https://www.reddit.com/r/learnpython/comments/9fgdib/virtual_subclasses/ [3] https://realpython.com/ref/glossary/subclass/ [4] https://docs.python.org/3/library/abc.html [5] https://www.demo2s.com/python/python-virtual-subclasses.html [6] https://stackoverflow.com/questions/51666120/whats-the-usage-of-a-virtual-subclass/51666808 [7] https://en.wikipedia.org/wiki/Virtual_class [8] https://kavianam.ir/Virtual-Subclass-in-Python [9] https://stackoverflow.com/questions/57602862/what-is-indirect-and-virtual-subclass [10] https://hynek.me/articles/python-subclassing-redux/

  • a virtual subclass should be seen more like a mixin rather than multiple inheritance

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

    **Virtual subclassing** in Python is *not* the canonical or idiomatic mechanism for achieving "multiple extends" (i.e., multiple inheritance) or adapting to multiple interfaces in the way you might do in statically-typed languages like Java or C#. Instead, Python supports **multiple inheritance** natively and directly through its class definition syntax, and this is the primary mechanism for a class to inherit from multiple parents and thus "implement multiple interfaces"[1][5][6]. ### Multiple Inheritance: The Pythonic Way Python allows a class to inherit from any number of parent classes simply by listing them in the class definition: ```python class MyClass(InterfaceA, InterfaceB, SomeBase): pass ``` This means `MyClass` will inherit all methods and attributes from `InterfaceA`, `InterfaceB`, and `SomeBase`, and will be considered a subclass of each for purposes of `issubclass()` and `isinstance()` checks[1][5][6]. This is the direct and idiomatic way to "extend" multiple interfaces or base classes in Python, and is how you would typically express that a class conforms to multiple protocols or contracts. ### Virtual Subclassing: Purpose and Limitations **Virtual subclassing** via `ABC.register(SomeClass)` is a mechanism for declaring that a class should be *recognized* as a subclass of an abstract base class (ABC) for the purposes of type checking, without actually inheriting from it. This is especially useful when you want to adapt a class (often from a third-party library) to an interface you control, but cannot or do not want to modify its source code[7]. - **Virtual subclassing does not provide code sharing**: The registered class does not inherit any methods or properties from the ABC; it is only recognized as a subclass for `issubclass()` and `isinstance()` checks. ​ - **No method resolution order (MRO) change**: The ABC does not appear in the registered class's MRO, so method resolution and `super()` calls are unaffected. ​ - **No attribute/method adaptation**: If the registered class does not implement the required methods, you get runtime errors. ### Adapting to Multiple Interfaces If you want a class to *adapt* to multiple interfaces: - **If you control the class**: Use multiple inheritance directly. This is idiomatic and gives you both interface conformance and code sharing[1][5][6]. ​ - **If you do not control the class**: Use virtual subclassing to declare interface conformance, but be aware this is only for type checks and does not provide actual method implementations[7]. ### Mixin Pattern A related idiom is the **mixin**: a class designed to be used as a base class that provides additional methods, typically without defining its own state. Mixins are commonly used in Python to modularly add functionality via multiple inheritance, and are a way to "compose" behaviors without the pitfalls of deep inheritance hierarchies[6]. ### Summary Table | Mechanism | Code Sharing | Interface Conformance | Idiomatic Use | |---------------------|-------------|----------------------|----------------------| | Multiple Inheritance| Yes | Yes | Implementing multiple interfaces, mixins | | Virtual Subclassing | No | Yes (type checks) | Adapting foreign classes to your ABCs | **In summary:** Python's primary and idiomatic way to "extend" multiple interfaces is through direct multiple inheritance. **Virtual subclassing** is a supplementary mechanism for adapting existing classes to an interface for type checks, but it does not provide code sharing or actual interface adaptation[1][5][6][7]. [1] https://www.digitalocean.com/community/tutorials/understanding-class-inheritance-in-python-3 [2] https://stackoverflow.com/questions/66849568/issubclass-and-multiple-inheritance [3] https://stackoverflow.com/questions/27545888/is-multiple-inheritance-from-the-same-base-class-via-different-parent-classes-re [4] https://softwareengineering.stackexchange.com/questions/291061/is-there-a-better-pattern-than-multiple-inheritance-here [5] https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/jython/clementine/python_inheritance.html?audience=wdp&context=dph&context=analytics&context=analytics&context=analytics&context=analytics&context=analytics&context=analytics&context=cpdaas [6] https://openstax.org/books/introduction-python-programming/pages/13-5-multiple-inheritance-and-mixin-classes [7] https://hynek.me/articles/python-subclassing-redux/ [8] https://docs.python.org/3/tutorial/classes.html [9] https://realpython.com/inheritance-composition-python/ [10] https://www.geeksforgeeks.org/python/multiple-inheritance-in-python/

Standard API of Mapping Types

The recommendation is to wrap a dict by composition instead of subclassing the Collection, Mapping, MutableMapping ABCs.

Note that because everything ultimately relies on the hastable, the keys must be hashable (doesn’t matter if the value is hashable)

• What Is Hashable

  • ✅ User Defined Types: for user defined types, the hashcode is the id() of the object and the __eq__ method from the object parent class compares the object ids.

  • gotcha: there’s a salt applied to hashing

    And the salt differs across python processes.

    The hash code of an object may be different depending on the version of Python, the machine architecture, and because of a salt added to the hash computation for secu‐ rity reasons.3 The hash code of a correctly implemented object is guaranteed to be constant only within one Python process.

• Overview of Common Mapping Methods: using dict, defaultdict and OrderedDict

  :NOTER_PAGE: (115 . 0.580146)

• Inserting or Updating Mutable Values: when to use setdefault

  Should use setdefault when you want to mutate the mapping and there’s nothing there

  E.g. you wanna fill in empty default values

  so instead of doing this which has 2 searches through the dict index ⛔️

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

  import re import sys WORD_RE = re.compile(r'\w+') index = {} with open(sys.argv[1], encoding='utf-8') as fp: for line_no, line in enumerate(fp, 1): for match in WORD_RE.finditer(line): word = match.group() column_no = match.start() + 1 location = (line_no, column_no) # this is ugly; coded like this to make a point occurrences = index.get(word, []) occurrences.append(location) index[word] = occurrences # display in alphabetical order for word in sorted(index, key=str.upper): print(word, index[word])

  we could do just do a single search within the dict index and do:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  """Build an index mapping word -> list of occurrences""" import re import sys WORD_RE = re.compile(r'\w+') index = {} with open(sys.argv[1], encoding='utf-8') as fp: for line_no, line in enumerate(fp, 1): for match in WORD_RE.finditer(line): word = match.group() column_no = match.start() + 1 location = (line_no, column_no) index.setdefault(word, []).append(location) # display in alphabetical order for word in sorted(index, key=str.upper): print(word, index[word])

  setdefault returns the value, so it can be updated without requiring a second search.

Automatic Handling of Missing Keys

We have 2 options here.

• defaultdict: Another Take on Missing Keys

  • it’s actually a callable that we are passing as an arg, so when we do things like bool or list we’re actually passing in the constructor to these builtins.
  • callable is stored within the default_factory and we can replace the factory as we wish!
  • interesting: if we do a membership check on a key that doesn’t exist, the default factory won’t be called yet.

• The missing Method

  :PROPERTIES: :NOTER_PAGE: (121 . 0.519175)

  TLDR: subclass UserDict instaed of dict to avoid these issues

  Take note of the nuances in the implementation that is shown because they avoid infinite recursion.

  It’s important to think of how the method delegation may introduce chances of infinite recursion.

  Also, same thing for what the fallback methods are for builtin methods.

• note: k in my_dict faster than k in my_dict.keys()

  Also technically k in my_dict is faster than using the k in my_dict.keys() because it avoids the attribute lookup to find the .keys method.

• Inconsistent Usage of missing in the Standard Library

  TLDR: subclass UserDict instaed of dict to avoid these issues subclassing builtin types is tricky! (will come up later in the book).

  Basically, this dunder method is inconsistently used. Be careful if you wanna subclass this, it may result in infinite recursions.

Variations of dict

• collections.OrderedDict

  Mostly the modern implementation for dict is good enough

  • has some minor differences from the modern implementation of dict:
    • can handle frequent reordering ops better than dict => useful to track recent accesses like in an LRU cache.
  • use it for simple implementations of an LRU cache.

• collections.ChainMap

  • chains together multiple mappings by holding references to the mappings
  • any mods happen to the first-inserted mapping
  • it’s useful to search hierarchically (search from d1, then d2, then d3) and prioritises the results found in the earlier implementation.

• collections.Counter

  • counters are great!!!
  • GOTCHA: when things are tied, only one is returned. example if counter looks like this: Counter({'a': 10, 'z': 3, 'b': 2, 'r': 2, 'c': 1, 'd': 1}) then if we do ct.most_common(3) then even if both b and c are tied, we only get b in the result. Here’s some more context on this:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61

    The behavior you’re observing with `Counter.most_common(n)`—where only one of the tied elements is included when there’s a tie at the cutoff—is by design, and it’s important to understand the underlying mechanics and implications. ### Why does this happen? - **`most_common(n)` returns exactly `n` items**: The method returns the *first* `n` elements from the sorted list of items, sorted by count (descending). If multiple elements have the same count at the cutoff position, only those that appear first (according to the internal order, which is typically insertion order for equal counts) are included[1][2][9]. ​ - **Ties are not expanded**: If two or more elements are tied for the last spot in the top `n`, only as many as needed to reach `n` are returned. The method does not automatically include all items with the same count as the nth item[6]. #### Example: ```python from collections import Counter ct = Counter({'a': 10, 'z': 3, 'b': 2, 'r': 2, 'c': 1, 'd': 1}) print(ct.most_common(3)) # Output might be: [('a', 10), ('z', 3), ('b', 2)] ``` Even though both `'b'` and `'r'` have a count of `2`, only `'b'` is included because you asked for the top 3, and `'b'` appears before `'r'` in the internal ordering[2][6]. ### How is the order determined for ties? - **Insertion order**: For elements with equal counts, their order in the result is based on the order they were first encountered in the input (Python 3.7+ dictionaries preserve insertion order)[2][9]. ​ - **No guarantee for all ties**: There is no built-in mechanism to include all elements with the same count as the nth element unless you implement it yourself[6]. ### What if you want all tied elements included? You need to post-process the result. For example, you can: 1. Get the count of the nth item. 2. Iterate through all items and include those with counts >= nth count. ```python from collections import Counter ct = Counter({'a': 10, 'z': 3, 'b': 2, 'r': 2, 'c': 1, 'd': 1}) n = 3 top_n = ct.most_common(n) cutoff = top_n[-1][1] all_tied = [(k, v) for k, v in ct.items() if v >= cutoff] print(all_tied) ``` This will include all items with counts equal to or greater than the nth most common, including ties[6]. ### Summary - `most_common(n)` returns exactly `n` elements, even if there are ties at the last position. ​ - For ties, only the first encountered elements (in insertion order) are included. ​ - To include all tied elements, you must filter manually after getting the cutoff value[2][6][9]. This behavior is documented and is a common source of confusion—your observation is correct and is a subtle but important aspect of using `Counter.most_common()` in Python. [1] https://www.geeksforgeeks.org/python/python-most_common-function/ [2] https://docs.python.org/3/library/collections.html [3] https://www.digitalocean.com/community/tutorials/python-counter-python-collections-counter [4] https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity [5] https://blog.csdn.net/weixin_43056275/article/details/124384145 [6] https://stackoverflow.com/questions/33791057/counter-most-common-is-a-little-misleading/33791292 [7] https://www.youtube.com/watch?v=fqACZvcsNug [8] https://dev.to/atqed/you-can-be-happy-to-know-python-counter-how-to-get-the-most-common-elements-in-a-list-o1m [9] https://ioflood.com/blog/python-counter-quick-reference-guide/ [10] https://dev.to/kathanvakharia/python-collections-module-counter-2gn

• shelve.Shelf

  • shelves are for storing pickle jars
  • shelves are persistent storage for a mapping of strings to pickle objects
  • A Shelf instance is a context manager, so you can use a with block to make sure it is closed after use.

  • Ref “Pickle’s nine flaws”

    here’s the writeup

    And here’s a bot summary of it:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

    Here is a summary of the nine flaws of Python's `pickle` module as detailed by Ned Batchelder[1][2]: 1. **Insecure** Pickle is fundamentally unsafe for untrusted data. Maliciously crafted pickles can execute arbitrary code during unpickling, as the process can invoke any callable in the Python environment—not just constructors for your objects. 2. **Old pickles look like old code** Pickle stores the exact structure of objects as they existed when pickled. If your code changes (e.g., you add or remove attributes), unpickling old data can result in objects that don't match your current codebase, leading to bugs and inconsistencies. 3. **Implicit** The serialization process is implicit and automatic: you cannot explicitly control how objects are serialized. For example, you cannot choose to serialize a datetime as an ISO8601 string instead of a datetime object. 4. **Over-serializes** Pickle serializes everything in your objects, including data you may not want to persist (like caches or open file handles). There is no straightforward way to exclude unwanted attributes, and if an attribute is not pickle-able, you'll get an exception[2]. 5. **`__init__` isn’t called** When unpickling, the `__init__` method of your class is not invoked. This can leave objects in an inconsistent state, especially if `__init__` does essential setup or side effects. 6. **Python only** Pickle is Python-specific. While there are rare cross-language implementations, they're limited and not practical for general use. This makes pickled data hard to share with non-Python systems. 7. **Unreadable** Pickles are binary blobs, not human-readable. You can't inspect or search them with standard tools, making debugging and data recovery more difficult. 8. **Appears to pickle code** Pickle will serialize references to functions and classes, but not their code—only their names. This can give the false impression that code is being serialized, but on unpickling, the code must already exist in the environment. 9. **Slow** Pickle is slower than many alternative serialization formats, both in terms of speed and efficiency. > Some of these issues can be mitigated with custom methods like `__getstate__` or `__reduce__`, but at that point, alternative serialization formats (e.g., JSON, protocol buffers) may be more robust and maintainable[1]. These flaws highlight why `pickle` is best reserved for trusted, Python-only, and short-lived data interchange scenarios—not for general-purpose or cross-system serialization. [1] https://nedbatchelder.com/blog/202006/pickles_nine_flaws.html [2] https://nedbatchelder.com/blog/tag/python.html [3] https://www.python4data.science/en/latest/data-processing/serialisation-formats/pickle/index.html [4] https://pycoders.com/issues/426 [5] https://stanforddaily.com/2019/10/11/face-it-pickles-are-bad-an-irrefutable-proof/ [6] https://content.ces.ncsu.edu/pickle-and-pickle-product-problems [7] https://diff.blog/post/pickles-nine-flaws-49891/ [8] https://pythonbytes.fm/episodes/show/193/break-out-the-django-testing-toolbox [9] https://www.reddit.com/r/Python/comments/1c5l9px/big_o_cheat_sheet_the_time_complexities_of/ [10] https://podscripts.co/podcasts/python-bytes/189-what-does-strstrip-do-are-you-sure

• Subclassing UserDict Instead of dict

  • key idea here is that it uses composition and keeps an internal dict within the data attribute
  • implementing other functions as we extend it will require us to use the self.data attribute.

Immutable Mappings

• we can use a read-only MappingProxyType from the types module to expose a readonly proxy

  the constructor in a concrete Board subclass would fill a private mapping with the pin objects, and expose it to clients of the API via a public .pins attribute implemented as a mappingproxy. That way the clients would not be able to add, remove, or change pins by accident.

Dictionary Views

• the views are supposed to be proxies as well so they are updated. Any changes to the original mapping will be viewable as well
• because they are not sequences (they are view objects) they are not subscript-able. so doing something like myvals[0] won’t work. If we wish, we could convert it to a list, but then it’s a copy, it’s no longer a live dynamic read-only proxy.

Practical Consequences of How dict Works

• why we should NOT add instance attrs outside of __init__ functions

  That last tip about instance attributes comes from the fact that Python’s default behavior is to store instance attributes in a special dict attribute, which is a dict attached to each instance.9 Since PEP 412—Key-Sharing Dictionary was implemented in Python 3.3, instances of a class can share a common hash table, stored with the class. That common hash table is shared by the dict of each new instance that has the same attributes names as the first instance of that class when init returns. Each instance dict can then hold only its own attribute values as a simple array of pointers. Adding an instance attribute after init forces Python to create a new hash table just for the dict of that one instance

  also KIV the implementation of __slots__ and how that is even better of an optimisation.

Set Theory

As we had found out from the extension writeup, the intersection operator is a great oneliner found = len(needles & haystack) or found = len(set(needles) & set(haystack)) to be more generalisable (though there’s the overhead from building the set)

• Set Literals

  • using the set literal ({1,2,3}) for construction is faster than using the constructor (set([ 1,2,3 ])) because the constructor will have to do a key lookup to fetch the function
  • the literal directly uses a BUILDSET bytecode

• Set Comprehensions

  • looks almost the same as dictcomps

Practical Consequences of How Sets Work

• Set Operations

Set Operations on dict Views

• .keys() and .items() are similar to frozenset

  • .values() may work like this too but only if all the values in the dict are hashable

• Even better: the set operators in dictionary views are compatible with set instances.

Chapter Summary

Further Reading

Chapter 4. Unicode Text Versus Bytes

What’s New in This Chapter

Character Issues

• “string as a sequence of characters” needs the term “character” to be defined well
• in python 3, it’s “unicode
• Unicode char separates:
  • identity of the char => refers to its code point
  • the byte representation for the char => dependent on the encoding used (codec between code points and byte sequences)

Byte Essentials

1. binary sequences, there are 2 builtin types:
   • mutable: bytearray
   • immutable: byte
2. Each item in bytes or bytearray is an integer from 0 to 255
3. literal notation depends (just a visual representation thing):
   • if in ascii range, display in ascii
   • if it’s a special char like tab and such, then escape it
   • if amidst apostrophes, then use escape chars
   • else just use the hex notation for it e.g. \x100
4. most functions work the same, except those that do formatting and those that depend on unicode data so won’t work:
   • case, fold
5. regexes work the same only if regex is compiled from a binary sequence instead of a str
6. how to build bytes or bytearray :
   1. use bytes.fromhex()
   2. use bytes.encode(“mystr”, encoding=“utf-8”)
   3. use soemthing that implements buffer protocol to create from source object to new binary sequence (e.g. memoryview).
      • This needs us to explicitly typecast
      • generally, this will also always copy the bytes from the source – except for memoryview which is a direct proxy to the memory

• Indexing vs slicing & why str is special

  Strings are really the special kind here.

  For bytes, bytearray, list, etc., seq returns a single element (e.g., an int), while seq[:1] returns a new sequence of the same type containing just that element. This is different from str, where both return a string. This distinction is important when working with binary data or non-string sequences in Python.

  More descriptive:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

  Let's break down the meaning of this passage from "Fluent Python" and the behavior it describes. ## The Key Point > For all other sequence types in Python, 1 item is not the same as a slice of length 1. This means: - **Accessing a single element** (e.g., `seq`) returns a single item (e.g., an int, a character, etc.). - **Taking a slice of length 1** (e.g., `seq[:1]` or `seq[0:1]`) returns a new sequence of the same type, containing just that one item. For most sequences in Python, **these two operations do not return the same type or value**. ### The Exception: `str` For strings, `s` and `s[:1]` both return a string (the first character), so they look the same: ```python s = "hello" print(s[0]) # 'h' print(s[:1]) # 'h' print(type(s[0]), type(s[:1])) # ``` ### The Rule: Other Sequences For other sequence types (e.g., `bytes`, `bytearray`, `list`, `tuple`): - **Indexing** returns a single element (not a sequence). - **Slicing** returns a sequence of the same type. #### Example: `bytes` ```python cafe = bytes('café', encoding='utf_8') print(cafe) # b'caf\xc3\xa9' print(cafe[0]) # 99 (int: ASCII code for 'c') print(cafe[:1]) # b'c' (bytes of length 1) print(type(cafe[0]), type(cafe[:1])) # ``` - `cafe` gives you the integer value of the first byte (`99`). - `cafe[:1]` gives you a `bytes` object of length 1: `b'c'`. #### Example: `bytearray` ```python cafe_arr = bytearray(cafe) print(cafe_arr[-1]) # 169 (int: last byte) print(cafe_arr[-1:]) # bytearray(b'\xa9') (bytearray of length 1) ``` - `cafe_arr[-1]` is an integer (the last byte). - `cafe_arr[-1:]` is a `bytearray` object of length 1. #### Example: `list` ```python lst = [10, 20, 30] print(lst[0]) # 10 print(lst[:1]) # [10] ``` - `lst` is an int. - `lst[:1]` is a list of length 1. ## Why Is This Surprising? - In Python's `str` type, `s` and `s[:1]` both return a string (the first character), which is a bit special. - For all other sequences, **indexing returns a single element (not a sequence), slicing returns a sequence**. ## Mental Model - **Indexing** (`seq[i]`): Returns the element at position `i` (type depends on the sequence). - **Slicing** (`seq[i:j]`): Returns a new sequence of the same type, containing elements from `i` to `j-1`. ## References - [Fluent Python, 2nd Edition, Chapter 4: Text versus Bytes](https://www.oreilly.com/library/view/fluent-python-2nd/9781492056348/) - [Python Data Model: Sequence Types](https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range) **In summary:** For `bytes`, `bytearray`, `list`, etc., `seq` returns a single element (e.g., an int), while `seq[:1]` returns a new sequence of the same type containing just that element. This is different from `str`, where both return a string. This distinction is important when working with binary data or non-string sequences in Python.

Basic Encoders/Decoders

• Highlight on page 153

  • Contents

    Each codec has a name, like ‘utf_8’

  • Comment

    so utf-8 is a codec here

• Highlight on page 154

  • Contents

    like ASCII and even the multibyte GB2312, cannot represent every Unicode character. The UTF encod‐ ings, however, are designed to handle every Unicode code point.

Understanding Encode/Decode Problems

Errors that we can expect:

1. (generic) UnicodeError
   1. UnicodeDecodeError
   2. UnicodeEncodeError
2. When loading libraries, might end-up facing SyntaxError also because of encoding issues

• Coping with UnicodeEncodeError

  • the error handlers for encoding error can include 'xmlcharrefreplace'. What this does is XML character reference: &#<unicode code point> and in so doing, there’s no loss of that information Here’s more context on it:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

    The statement from "Fluent Python"— > 'xmlcharrefreplace' replaces unencodable characters with an XML entity. If you can’t use UTF, and you can’t afford to lose data, this is the only option. —means that when you encode a string using a limited encoding (like ASCII) and specify `errors='xmlcharrefreplace'`, **any character that cannot be represented in the target encoding is replaced with an XML numeric character reference** (e.g., `é` for "é"). This ensures that **no information is lost**: all original characters are either encoded directly (if possible) or represented as XML entities, which are reversible. ### How does it work? - When encoding, Python checks each character: ​ - If the character can be encoded in the target encoding (e.g., ASCII), it is kept as-is. ​ - If it cannot, it is replaced with its XML character reference: `&#;` ​ - When decoding, you can later convert these references back to the original characters, so the process is *lossless* in terms of information content. #### Example ```python txt = "Café" encoded = txt.encode("ascii", errors="xmlcharrefreplace") print(encoded) # b'Café' ``` Here, "é" (which is not in ASCII) is replaced with `é`, preserving the character information[2][7]. ### Why is there no data loss? - **All original characters are represented:** Characters that can't be encoded are replaced with their numeric reference, which uniquely identifies the character. ​ - **Reversible:** You can later parse the XML entities back into the original Unicode characters, restoring the original string[1][6]. ### Contrast with other error handlers - `'replace'` swaps unencodable characters for `?` (data loss). ​ - `'ignore'` simply omits them (data loss). ​ - `'backslashreplace'` uses Python escape sequences (reversible, but not standard in XML/HTML). ​ - `'xmlcharrefreplace'` uses XML/HTML-compatible numeric references (reversible, and standard for text interchange). ### Practical implication If you must encode text in a limited character set (like ASCII or Latin-1) but need to ensure that all characters are preserved in some form (for later recovery or interoperability), `'xmlcharrefreplace'` is the safest choice[4][6][7]. **In summary:** Using `'xmlcharrefreplace'` means that **no original character data is lost**—all characters are either encoded directly or replaced with a reversible XML entity. This is why the book says it is the only option if you can't use UTF and can't afford to lose data. [1] https://stackoverflow.com/questions/44293891/python-string-encoding-xmlcharrefreplace-decode [2] https://www.w3schools.com/python/ref_string_encode.asp [3] https://docs.python.org/3/howto/unicode.html [4] https://docs.python.org/3/library/codecs.html [5] https://www.codecademy.com/resources/docs/python/strings/encode [6] https://code.activestate.com/recipes/303668-encoding-unicode-data-for-xml-and-html/ [7] https://www.geeksforgeeks.org/python/python-strings-encode-method/ [8] https://www.digitalocean.com/community/tutorials/python-string-encode-decode [9] https://labex.io/tutorials/python-what-is-the-role-of-the-encoding-and-errors-parameters-in-the-str-function-in-python-395133 [10] https://docs.vultr.com/python/standard-library/str/encode

• Coping with UnicodeDecodeError

  • Highlight on page 156

    • Contents

      On the other hand, many legacy 8-bit encodings like ‘cp1252’, ‘iso8859_1’, and ‘koi8_r’ are able to decode any stream of bytes, including random noise, without reporting errors. Therefore, if your program assumes the wrong 8-bit encoding, it will silently decode garbage.

    • Comment

      utf8/16 will sound off because it’s a strict error check

      the older 8bit codecs will do it silently

  • Highlight on page 157

    • Contents

      “�” (code point U+FFFD), the official Unicode REPLACEMENT CHARACTER intended to represent unknown characters.

    • Comment

      there’s an official REPLACEMENT CHARACTER

• SyntaxError When Loading Modules with Unexpected Encoding

  • utf8 default for python source code
  • fix this by defining explicitly what encoding type to use at the top of the file when writing that file out.

    1	# coding: cp1252

    OR just fix it by converting to UTF-8

• How to Discover the Encoding of a Byte Sequence

  • you can’t but you can make a good guess
  • chardet exists for this reason, it’s an estimated detection of the encoding type.

  • Highlight on page 159

    • Contents

      human languages also have their rules and restrictions, once you assume that a stream of bytes is human plain text, it may be possible to sniff out its encoding using heuristics and statistics. For example, if b’\x00’ bytes are common, it is probably a 16- or 32-bit encoding, and not an 8-bit scheme, because null characters in plain text are bugs. When the byte sequence b’\x20\x00’ appears often, it is more likely to be the space character (U+0020) in a UTF-16LE encoding, rather than the obscure U+2000 EN QUAD character—whatever that is. That is how the package “Chardet—The Universal Character Encoding Detector” works to guess one of more than 30 supported encodings. Chardet is a Python library that you can use in your programs, but also includes a command-line utility, charde tect.

    • Comment

      typically an encoding is declared – so you have to be told what encoding it is

      however, it’s possible to guess probabilistically what the encoding could be.

      there are packages for that (Chardet)

• BOM: A Useful Gremlin

  • Byte-Order Mark: helps us know if the machine that the encoding was performed on is little or big endian.
  • endianness becomes a problem only for any encoding format that takes more than a byte (so for UTF-16 and UTF-32) ==> so BOM only matters for them
  • so BOM not needed for UTF-8
  • but it can still be added in (discouraged though)

  • Highlight on page 160

    • Contents

      UTF-16 encoding prepends the text to be encoded with the special invisible character ZERO WIDTH NO-BREAK SPACE (U+FEFF).

  • Highlight on page 160

    • Contents

      This whole issue of endianness only affects encodings that use words of more than one byte, like UTF-16 and UTF-32

  • Highlight on page 161

    • Contents

      using UTF-8 for general interoperability. For example, Python scripts can be made executable in Unix systems if they start with the comment: #!/usr/bin/env python3. The first two bytes of the file must be b’#!’ for that to work, but the BOM breaks that con‐ vention. If you have a specific requirement to export data to apps that need the BOM, use UTF-8-SIG but be aware that Python’s codecs documentation says: “In UTF-8, the use of the BOM is dis‐ couraged and should generally be avoided.”

    • Comment

      use UTF-8-SIG because will be harmless

      also note that the python codecs documentation says that in utf8, using a BOM (byte order mark) is discouraged.

Handling Text Files & the “Unicode Sandwich”

Here’s the gist of why it’s “unicode sandwich”

1. decode bytes on input
2. process text only (the meat of the sandwich is the business logic that should use strings)
3. encode text on output

The best practice for handling text I/O is the “Unicode sandwich” (Figure 4-2).5 This means that bytes should be decoded to str as early as possible on input (e.g., when opening a file for reading). The “filling” of the sandwich is the business logic of your program, where text handling is done exclusively on str objects. You should never be encoding or decoding in the middle of other processing. On output, the str are encoded to bytes as late as possible.

• Highlight on page 161

  • Contents

    e best practice for handling text I/O is the “Unicode sandwich” (Figure 4-2).5 This means that bytes should be decoded to str as early as possible on input (e.g., when opening a file for reading). The “filling” of the sandwich is the business logic of your program, where text handling is done exclusively on str objects. You should never be encoding or decoding in the middle of other processing. On output, the str are enco‐ ded to bytes as late as possible.

  • Comment

    Unicode sandwhich is the best practices for handling text files and their encoding:

    1. bytes -> str (decode bytes as early as possbile, i.e. on input)

    2. process text only in the business logic

    3. encode text on output only

• Highlight on page 162

  • Contents

    Code that has to run on multiple machines or on multiple occasions should never depend on encoding defaults. Always pass an explicit encoding= argument

  • Comment

    cross-platform code should always explicitly define the encoding value!

    so unix machines will use utf-8 but then when using, say, a windows machine there might be an encoding issue becaue

• Highlight on page 163

  • Contents

    TextIOWrapper with the encoding set to a default from the local

• Beware of Encoding Defaults

  even within say windows itself, not every application would have the same encoding.

  for unix it’s more standardised, so it’s most likely expected to be utf-8

  • Defaults

    Main thing to remember is that the most important encoding setting is the one that is retired by locale.getpreferredencoding()

    The changes can be effected by changing the environment variables.

Normalizing Unicode for Reliable Comparisons

• canonical equivalents exist, but they have different code points under the hood.
• there’s a bunch of different normalisation forms, for extra safety, when saving strings, should normalise that string (using NFC normalistaion for example)
• gotcha: some single characters can be normalised to result in visually similar but they compare different
• string normalisation can be lossy, so there can be actual dataloss from multiple sandwhich creation, destruction, creation
  • NFKC and NFKD are examples of such normalisation forms - these forms should only be used for intermediate representations for search & index
• NFC is not sufficient for search and indexing because it preserves compatibility distinctions that are irrelevant (and even counterproductive) for matching. NFKC/NFKD are used as intermediate representations for search and indexing because they erase these distinctions, enabling robust, user-friendly search behavior—at the cost of losing some original form information, which is why they are not used for storage or display. See more info here:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

  To understand why **NFC normalization is not always suitable for search and indexing**, and why compatibility forms like **NFKC/NFKD** are often used as intermediate representations for these purposes, let's clarify the properties and goals of each normalization form and their implications for search/index use cases. ### **NFC vs. NFKC: What’s the Difference?** - **NFC (Normalization Form C, Canonical Composition):** ​ - Collapses canonically equivalent sequences into a single, composed form. ​ - Preserves distinctions between characters that are *compatible* but not *canonically equivalent* (e.g., ligatures, superscripts, full-width vs. half-width characters). ​ - Designed to be *lossless* for textual content, so that round-tripping (normalize, then denormalize) does not lose data[4][7]. - **NFKC (Normalization Form KC, Compatibility Composition):** ​ - Collapses both canonically equivalent and *compatibility equivalent* sequences. ​ - This means it will, for example, convert ligatures like 'fi' (U+FB01) to 'fi', or full-width Latin letters to their standard forms. ​ - This process is **lossy**: information about the original form (e.g., that a ligature or superscript was used) is lost[4][7]. ### **Why Not Use NFC for Search and Indexing?** **NFC is designed to preserve distinctions that are meaningful in text rendering or data storage, but are often *not* meaningful for search and indexing.** For example: ​ - The string "field" could be encoded as: ​ - `U+0066 U+0069 U+0065 U+006C U+0064` ("field") ​ - `U+FB01 U+0065 U+006C U+0064` ("field" with the ligature 'fi') ​ - **NFC will *not* turn the ligature into "fi"; it will keep the distinction.** This means a search for "field" will not match a document containing "field" (with the ligature) if both are normalized to NFC[4][7]. **For search and indexing, you often want:** ​ - "field" and "field" to be treated as equivalent. ​ - Full-width and half-width forms, superscripts, circled numbers, etc., to be treated as their base forms for matching. **NFKC/NFKD normalization does this:** ​ - It "folds" away these compatibility distinctions, making visually or semantically similar forms compare equal[1][4][7]. ### **Special Role of Compatibility Normalization for Search/Index** - **Intermediate representation**: You normalize both the indexed data and the search queries to NFKC/NFKD, so that *compatibility* distinctions are erased and matches are more robust. ​ - **Not for storage or display**: Because NFKC/NFKD is lossy, you shouldn't use it for storing or displaying user data—only for the internal process of searching and indexing[1][4]. ### **Summary Table** | Use Case | Recommended Normalization | Rationale | |-----------------|--------------------------|--------------------------------------------------------------| | Storage/Display | NFC | Preserves all distinctions, lossless for most text | | Search/Index | NFKC/NFKD | Folds away compatibility differences for robust matching | ### **References from Results** ​ - "NFC removes the distinction between equivalent characters, while preserving the distinction between compatible characters or sequences; NFKC removes the distinction between both equivalent and compatible sequences. NFC conversion is not considered lossy, but NFKC conversion is." [4] ​ - "It's more explicitly lossy; the compatibility forms are recommended for performing searches since they fold additional characters such as plain latin and 'full-width' latin letters." [1] ​ - "The compatibility normalizations are super useful, because ... you often want a search query for `ffi` to match on `ffi` (and vice versa). But they are intended to lose symbolic representation (ffi and ffi are now the same thing with no way to distinguish), where the canonical normalizations are not." [7] ### **In summary** **NFC** is not sufficient for search and indexing because it preserves compatibility distinctions that are irrelevant (and even counterproductive) for matching. **NFKC/NFKD** are used as intermediate representations for search and indexing because they erase these distinctions, enabling robust, user-friendly search behavior—at the cost of losing some original form information, which is why they are not used for storage or display. [1] https://www.mediawiki.org/wiki/Unicode_normalization_considerations [2] https://stackoverflow.com/questions/15985888/when-to-use-unicode-normalization-forms-nfc-and-nfd [3] https://unicode.org/reports/tr15/ [4] https://jazz.net/wiki/bin/view/LinkedData/UseOfUnicodeNormalForms [5] https://www.w3.org/wiki/I18N/CanonicalNormalizationIssues [6] https://blog.reeset.net/archives/2532 [7] https://news.ycombinator.com/item?id=19379965 [8] https://go.dev/blog/normalization [9] https://www.reddit.com/r/programming/comments/b09c0j/when_zo%C3%AB_zo%C3%AB_or_why_you_need_to_normalize_unicode/ [10] https://unicode-org.github.io/icu/design/normalization/custom.html

• Notes for page 140 V: 39% H: 25%

  sequences like ‘é’ and ’e\u0301’ are called “canonical equivalents,” and applications are supposed to treat them as the same. But Python sees two different sequences of code points, and considers them not equal.

• Notes for page 140 V: 82% H: 50%

  it may be good to normalize strings with normalize(‘NFC’, user_text) before saving.

• Case Folding (normalisation tranformation)

  • folding everything into lowercase
  • NOTE: casefold() and str.lower() have ~ 300 code points that return different results

• Utility Functions for Normalized Text Matching

  util functions that might help:

  • nfc_equal
  • fold_equal

• Extreme “Normalization”: Taking Out Diacritics

  • google search uses this aggressive normalisation based on real world attention that people give to diacritics
  • also helps for readable URLs (e.g for latin-based languages)
  • one way to call this transformation is “shaving”. We “shave” the diacritics

Sorting Unicode Text

• python sorts by comparing sequences one by one
• for strings, it compares code points
• so to sort non-ascii text in python, have to use local.strxfrom to have locale-aware comparisons

• Sorting with the Unicode Collation Algorithm

  • stdlib solution: there’s a locale.strxfrm to do locale-specific comparisons

    Python is to use the locale.strxfrm function which, according to the locale module docs, “transforms a string to one that can be used in locale-aware comparisons.”

    1 2 3 4 5 6

    import locale my_locale = locale.setlocale(locale.LC_COLLATE, 'pt_BR.UTF-8') print(my_locale) fruits = ['caju', 'atemoia', 'cajá', 'açaí', 'acerola'] sorted_fruits = sorted(fruits, key=locale.strxfrm) print(sorted_fruits)

  • use the Unicode Collation Algorithm via pyuca lib

The Unicode Database

Db is in the form of multiple text files.

Contains:

• code point to char name mappings
• metadata about the individual characters and how they are related.

That’s how the str methods isalpha, isprintable, isdecimal, and isnumeric work.

• Finding Characters by Name

  use name() function from the unicodedata library

• Numeric Meaning of Characters

  Some useful string functions here:

  1. .isnumeric()
  2. .isdecimal()

  comparisons with the human meaning of these rather than the code point.

• common string functions may lookup this unicode database

  This is responsible for the string functions like isdecimal isnumeric…

  the Unicode database records whether a character is printable, is a letter, is a decimal digit, or is some other numeric symbol. That’s how the str methods isal pha, isprintable, isdecimal, and isnumeric work. str.casefold also uses infor‐ mation from a Unicode table.

Dual-Mode str and bytes APIs

• str Versus bytes in Regular Expressions

  • if given bytes patterns like \d and \w will only match ASCII characters
  • if given str patterns like \d and \w will only match beyond just ASCII characters.

  to make one point: you can use regular expressions on str and bytes, but in the second case, bytes outside the ASCII range are treated as nondigits and nonword characters.

• regex patterns using bytes will treat outside-ASCII range chars as nondigits and nonword chars

  trivial example to make one point: you can use regular expressions on str and bytes, but in the second case, bytes outside the ASCII range are treated as nondigits and nonword characters.

• str Versus bytes in os Functions

  • os functions actually abide by the Unicode Sandwich: they actually call sys.getfilesystemencoding() as soon as they can

Chapter Summary

1. remember that 1 char == 1 byte is only true if it’s utf-8, there’s more than just that.
2. just always be explicit about encodings when reading them Follow the unicode sandwich and ensure the encoding is explicit always.
3. Unicode provides multiple ways of representing some characters, so normalizing is a prerequisite for text matching.

Further Reading

Chapter 5. Data Class Builders

:NOTER_PAGE: (193 . 0.108844)

I think my stance on using data classes is that it should help mock things easily to come up with scaffolds which are easy to replace.

It’s interesting that the type hinting for class vs instance attributes ended up needing to use pseudoclasses specific for this purpose (ClassVar, InitVar)

Link on page 194: typing module documentation

What’s New in This Chapter

Overview of Data Class Builders

• Problem posed:
  • __init__ constructor can become too complex if we’re just going to assign attributes from constructor parameters
• 3 options:
  • collections.namedtuple
  • typing.NamedTuple
    • newer than namedtuple
  • @dataclass decorator from dataclasses module

• How they work:
  • they don’t rely on inheritence
  • typing hints are there if we use NamedTuple or dataclass
  • some of them are subclasses of tuple
  • All of them use metaprogramming techniques to inject methods and data attributes into the class under construction.
  • Some of them are more updated ways of doing things: typed.NamedTuple is newer than namedtuple

• Examples:

  • Named tuple:

    • define inline Coordinate = typing.NamedTuple('Coordinate', lat=float, lon=float)

    • defined with a class statement Although here, NamedTuple is not a superclass, it’s actually a metaclass

      1 2 3 4 5 6 7 8 9 10

      from typing import NamedTuple class Coordinate(NamedTuple): lat: float lon: float def __str__(self): ns = 'N' if self.lat >= 0 else 'S' we = 'E' if self.lon >= 0 else 'W' return f'{abs(self.lat):.1f}°{ns}, {abs(self.lon):.1f}°{we}'

  • Using dataclass

    1 2 3 4 5 6 7 8 9 10 11

    from dataclasses import dataclass @dataclass(frozen=True) class Coordinate: lat: float lon: float def __str__(self): ns = 'N' if self.lat >= 0 else 'S' we = 'E' if self.lon >= 0 else 'W' return f'{abs(self.lat):.1f}°{ns}, {abs(self.lon):.1f}°{we}'

• Main Features

  • Link on page 198: inspect.get_annotations(MyClass)
  • Link on page 198: typing.get_type_hints(MyClass)

  • Mutability

    Out of the 3, only @dataclass allows us to keep the class mutable (if we need we an mark it as frozen btw).

    The rest, since they are subclasses of tuple are immutable.

    For the immutable ones, we can replace the object using replace functions.

  • NamedTuple as a metaclass customization of a class def

    Although NamedTuple appears in the class statement as a super‐ class, it’s actually not. typing.NamedTuple uses the advanced func‐ tionality of a metaclass2 to customize the creation of the user’s class.

  • Correctly reading type hints @ runtime

    It will be discussed in more detail later in the book

    reading from annotations directly is not recom‐ mended. Instead, the recommended best practice to get that information is to call inspect.get_annotations(MyClass) (added in Python 3.10) or typing.get_type_hints(MyClass) (Python 3.5 to 3.9). That’s because those functions provide extra services, like resolving forward references in type hints.

Classic Named Tuples

• collections.namedtuple is a factory function

  So it’s possible to hack things by adding functions to this subclass.

  collections.namedtuple function is a factory that builds subclasses of tuple enhanced with field names, a class name, and an informative repr.

• Memory Use by collections.namedtuple

  There’s no excess mem usage because it’s the class that will store the attribute names

  So it’s same space usage as a tuple.

  Each instance of a class built by namedtuple takes exactly the same amount of memory as a tuple because the field names are stored in the class.

• Injecting methods into the subclass

  this is a hack, shouldn’t be relied upon.

  NOTE: No need to name the first arg as self if you’re hacking things by injecting methods

  the first argument doesn’t need to be named self. Anyway, it will get the receiver when called as a method.

  • normal classes method definition, self attribute is the receiver

    just some extra information about what the receiver is in the context of defining class methods in python

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

    In Python, the **self** parameter in class methods is the conventional name for the **receiver**—the object instance that the method is being called on. While Python documentation and community almost always use the term "self," in some other object-oriented languages (like Ruby or in theoretical OOP discussions), "receiver" is the standard term for the object that receives the method call. ### What is the "receiver"? - The **receiver** is the specific instance of the class on which the method is invoked. - In Python, when you define a method like `def foo(self, ...)`, the `self` parameter is automatically bound to the instance when you call `instance.foo(...)`[6][1][7]. - This allows each method to access and modify the attributes and other methods of the particular object instance, not just the class as a whole[6][1]. ### How does it work? - When you call `obj.method(args)`, Python translates this to `Class.method(obj, args)`. The object `obj` is passed as the first argument to the method, and inside the method, it is referred to as `self`[6][1][7]. - This is how methods can operate on instance-specific data: `self.attribute` refers to the attribute named `attribute` on the particular instance, not on the class or on other instances[1][6][7]. #### Example: ```python class Car: def __init__(self, model): self.model = model # self is the receiver def show(self): print("Model is", self.model) audi = Car("Audi A4") audi.show() # 'self' inside show() refers to the audi instance ``` - Here, `audi.show()` is equivalent to `Car.show(audi)`. The `audi` object is the receiver. ### Why is the receiver important? - It enables **object-specific state and behavior**. Each instance maintains its own data, and methods can access or mutate that data through `self`[1][6][7]. - It allows methods to call other methods on the same object, e.g., `self.other_method()`[3]. - Without the receiver, methods would not know which instance's data to operate on, and all data would have to be global or class-level. ### How does Python implement this? - When you retrieve a method from an instance (e.g., `method = obj.method`), Python returns a **bound method**—a function object with the receiver (`obj`) already attached as its first argument[5][8]. - When you call the bound method, you only need to supply the remaining arguments; `self` is supplied automatically[5]. ### Summary - **self** is the Pythonic name for the **receiver** of a method call in a class. - It refers to the specific instance on which the method was called. - It gives methods access to instance-specific data and behavior, enabling true object-oriented programming in Python[1][6][7]. > “The 'self' parameter in Python class methods represents the class instance (object). Methods can access and manipulate attributes (variables) and call other methods of the same class using this special variable.”[6] This is a foundational mechanism for encapsulation and polymorphism in Python’s object model. [1] https://www.geeksforgeeks.org/python/self-in-python-class/ [2] https://www.reddit.com/r/learnpython/comments/k9f4q7/could_someone_explain_the_use_of_self_when_it/ [3] https://docs.python.org/3/tutorial/classes.html [4] https://stackoverflow.com/questions/14671218/python-class-methods-changing-self [5] https://stackoverflow.com/questions/70076256/how-do-python-handles-self-attribute-of-a-class-internally-when-the-method-is [6] https://www.w3resource.com/python-interview/explain-the-purpose-of-the-self-parameter-in-python-class-methods.php [7] https://blog.finxter.com/understanding-the-self-in-python-classes/ [8] https://docs.python.org/3/reference/datamodel.html [9] https://softwareengineering.stackexchange.com/questions/422364/is-it-better-to-pass-self-or-the-specific-attributes-in-python-methods [10] https://www.pythonmorsels.com/what-is-self/

Typed Named Tuples

• compile-time type annotations: the main feature of named tuples

  Classes built by typing.NamedTuple don’t have any methods beyond those that col lections.namedtuple also generates—and those that are inherited from tuple. The only difference is the presence of the annotations class attribute—which Python completely ignores at runtime.

Typed Named Tuples

• the type annotations are ignored by python at runtime

Type Hints 101

• No Runtime Effect

  • Type hints not enforced by compiler & interpreter

    • main intent is for use by static analysis tools, at rest

    The first thing you need to know about type hints is that they are not enforced at all by the Python bytecode compiler and interpreter.

  • Works at import time!

    that’s why importing libraries may fail.

• Variable Annotation Syntax

  • variable here refers to the fact that variables are being annotated, not that the type hint is variable.
  • the syntax is just var_name: some_type = a_default_value

• The Meaning of Variable Annotations

  • For classic class definitions, survival of annotations & survival of attributes within annotations

    :NOTER_PAGE: (206 . 0.086168)

    This applies to the classic class definitions, without the named tuples and such.

    This makes sense because there’s no reason to keep the annotations.

    surviving of annotation <== if there’s a type hint given

    surviving of the attribute in the class <== if there’s a value assignable

    Note that the annotations special attribute is created by the interpreter to record the type hints that appear in the source code—even in a plain class. The a survives only as an annotation. It doesn’t become a class attribute because no value is bound to it.6 The b and c are stored as class attributes because they are bound to values.

  • Annotations are type annotations for immutable attributes

    This is because NT is extended from Tuple class.

    • Contents

      If you try to assign values to nt.a, nt.b, nt.c, or even nt.z, you’ll get Attribute Error exceptions with subtly different error messages. Try that and reflect on the messages.

    • Comment

      Because it’s read-only instance attribute and it’s expected to be immutable

  • using the @dataclass decorator allows the attrs to persist as instance attributes

    :NOTER_PAGE: (208 . 0.488788)

    • Contents

      However, there is no attribute named a in DemoDataClass—in contrast with DemoNTClass from Example 5-11, which has a descriptor to get a from the instances as read-only attributes (that myste‐ rious <_collections._tuplegetter>). That’s because the a attribute will only exist in instances of DemoDataClass. It will be a public attribute that we can get and set, unless the class is frozen. But b and c exist as class attributes, with b holding the default value for the b instance attribute, while c is just a class attribute that will not be bound to the instances.

    • Comment

      when using a decorator, the descriptor for the class that is ONLY type-hinted will only exist in concrete instances of that class.

  • annotation special attr are for type hints

    annotations special attribute is created by the interpreter to record the type hints that appear in the source code—even in a plain class.

More About @dataclass

• Don’t set a custom attribute outside of its constructor function!

  :NOTER_PAGE: (209 . 0.862182)

  • Contents

    Setting an attribute after init defeats the dict key-sharing memory optimization mentioned in “Practical Consequences of How dict Works” on page 102.

  • Comment

    Reminder: all the attrs for a class should really just be defined within the class itself to benefit from the memory optimisation that it comes with by default

• immutability is emulated by methods

  Which means it can be bypassed by overriding the implementation of these functions! (the settattr and deattr dunder methods)

  emulates immutability by generating setattr and delattr, which raise data class.FrozenInstanceError

• Field Options

  • WARNING: mutable defaults are NOT allowed.

    similar to the assignment gotchas where if we do my arr = [[] * 3], reusing a mutable reference (the inner list) means that the 3 instances all point to the same memory location

    we can how that would be a problematic bug

    therefore, it’s illegal to set default values that are mutable when we use dataclasses.

    we can use default_factory as a solution to this.

  • default_factory helps prevent mutability bugs

    • if a default value is provided that is mutable, then it would mean that many instances can edit the same mutable handle ==> this is a problematic bug. That’s why the default option is only to pass a factory function if you want to assign mutable default values so that each mutable default is a separate reference.

    • but this won’t apply to custom mutable objects, that’s why it’s a common source of mutable data related bugs l

    The default_factory parameter lets you provide a function, class, or any other call‐ able, which will be invoked with zero arguments to build a default value each time an instance of the data class is created. This way, each instance of ClubMember will have its own list—instead of all instances sharing the same list from the class, which is rarely what we want and is often a bug.

  • mental model for sentinel values

    ``sentinel value’’

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

    ### Understanding Sentinel Values **Sentinel values** are special values used in programming to signify that a variable or parameter is missing, not set, or has a specific condition that needs to be distinguished from valid values. They serve as markers that help identify states in data structures or function arguments without conflicting with legitimate data. #### Key Characteristics of Sentinel Values 1. **Uniqueness**: A sentinel value is typically unique and not used as a regular value in the context where it is applied. This uniqueness allows it to serve as a clear indicator of absence or a specific condition. 2. **Common Usage**: - **Function Arguments**: In functions, sentinel values can indicate that an optional parameter was not provided. For example, in Python, you might use `None` or a custom sentinel object to differentiate between "no value" and "a value of None". - **Data Structures**: In data structures like linked lists or trees, sentinel nodes may be used to simplify boundary conditions (e.g., head and tail pointers). 3. **Examples**: ```python # Using None as a sentinel def fetch_data(key, default=None): if key not in my_dict: return default # Return the sentinel if key is missing return my_dict[key] # Using a custom sentinel MISSING = object() def get_value(key, default=MISSING): if key not in my_dict: if default is MISSING: raise KeyError(f"{key} not found") return default return my_dict[key] ``` ### Mental Model for Sentinel Values To form a mental model for sentinel values, consider the **concept of markers or flags** in various contexts: 1. **Etymological Basis**: The term "sentinel" originates from the Latin word "sentinella," meaning "to watch." Just as sentinels guard and signal the presence or absence of something important, sentinel values act as indicators in programming. They help you "watch" for specific conditions in your code. 2. **Mental Model**: Think of sentinel values as **flags on a map**: - Imagine a treasure map where certain locations are marked with flags indicating whether they contain treasure (valid data) or are empty (missing data). - Just like those flags help you navigate the map without confusion, sentinel values guide your logic by clearly indicating when something is absent or when a specific condition applies. 3. **Practical Application**: When designing functions or data structures, consider how you can implement sentinel values to handle edge cases gracefully. This approach can prevent errors and make your code more robust by explicitly managing conditions that would otherwise lead to ambiguity. ### Summary Sentinel values are special markers used to indicate missing or special conditions in programming. They provide clarity and help manage edge cases effectively. By conceptualizing them as flags on a map, you can better understand their role in guiding logic and maintaining code integrity. This mental model emphasizes the importance of clear indicators in navigating complex data scenarios, much like sentinels watch over critical points of interest. Citations: [1] https://www.reddit.com/r/slatestarcodex/comments/83phqz/what_mental_models_do_you_find_to_be_most_helpful/ [2] https://fs.blog/mental-models/ [3] https://pmc.ncbi.nlm.nih.gov/articles/PMC8290163/ [4] https://jamesclear.com/feynman-mental-models [5] https://www.linkedin.com/pulse/10x-your-results-mental-models-mark-green [6] https://forum.valuepickr.com/t/scalability-thinking-mental-models/104746 [7] https://www.safalniveshak.com/latticework-mental-models-active-reading/ [8] https://learn.microsoft.com/ja-jp/globalization/encoding/unicode-standard

• Post-init Processing

  Allows us to insert logic after the contructor, to do things like calculations and such.

  • Highlight on page 213

    • Contents

      method generated by @dataclass only takes the arguments passed and assigns them—or their default values, if missing—to

  • Highlight on page 213

    • Contents

      Common use cases for post_init are validation and computing field values based on other fields.

  • Link on page 214: “Inheritance” section of the dataclasses module documentation

• Typed Class Attributes

  Need to use ClassVar from the typing module.

  This is a pseudotype Read more here:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

  Yes, **this is still the standard and recommended way to declare class variables with type hints in the most recent versions of Python (as of Python 3.13)**. ### Why is `ClassVar` Needed? - By default, when you annotate a variable inside a class, **type checkers (like mypy, pyright, or the built-in `dataclasses` module) assume it is an instance variable**—that is, each instance of the class will have its own copy. - If you want to indicate that a variable is a **class variable** (shared across all instances), you must use `typing.ClassVar` in your annotation[1][2]. - This is especially important for tools like `@dataclass`, which will otherwise treat all annotated variables as fields to be included in the instance's `__init__`. ### The Syntax ```python from typing import ClassVar class Example: all_handles: ClassVar[set[str]] = set() ``` - This tells both the type checker and `@dataclass` (if used) that `all_handles` is a class attribute, not an instance attribute. - The type inside `ClassVar[...]` can be any valid type, including generics like `set[str]`. ### Why is this necessary/ugly? - The reason for this "pseudotype" is that Python's variable annotations (PEP 526) are designed to be explicit and unambiguous for static analysis and code generation tools. - Without `ClassVar`, type checkers and dataclasses would treat the variable as an instance field, which is almost always wrong for shared state. ### Is this still the case in Python 3.13+? **Yes.** - The [official Python docs](https://docs.python.org/3/library/typing.html) (as of Python 3.13.5) still require `ClassVar` for class variables in type annotations[2]. - This is also reflected in community best practices and in the implementation of type checkers and the standard library[1][2][3][4]. ### Example with `@dataclass` ```python from dataclasses import dataclass from typing import ClassVar @dataclass class Example: all_handles: ClassVar[set[str]] = set() # Not included as a dataclass field name: str # This is an instance variable ``` - `all_handles` is not included in the generated `__init__`, nor in the instance's `__dict__`. ### Related: `Final` and `ClassVar` - If you want a class variable to be a constant, you can use `Final` from `typing`, but **you cannot combine `Final` and `ClassVar`** in the annotation (see [1]). You must choose one or the other, as combining them is not supported by type checkers. ### References to Search Results - [PEP 526 and usage of ClassVar](https://stackoverflow.com/questions/66285078/how-to-define-final-classvar-variable-in-python)[1] - [Python 3.13 typing documentation](https://docs.python.org/3/library/typing.html)[2] - [Class variables in Python](https://pynative.com/python-class-variables/)[5] **In summary:** Declaring class variables with type hints using `ClassVar[...]` is still required and is the correct, modern, and supported approach in all current and foreseeable future versions of Python. This ensures type checkers and dataclasses handle your class variables as intended. [1] https://stackoverflow.com/questions/66285078/how-to-define-final-classvar-variable-in-python [2] https://docs.python.org/3/library/typing.html [3] https://github.com/python/typing/discussions/1424 [4] https://programming-25.mooc.fi/part-9/5-class-attributes/ [5] https://pynative.com/python-class-variables/ [6] https://docs.python.org/3/tutorial/classes.html [7] https://www.digitalocean.com/community/tutorials/understanding-class-and-instance-variables-in-python-3 [8] https://realpython.com/python-variables/ [9] https://programming-25.mooc.fi/part-8/3-defining-classes/ [10] https://github.com/python/typing/discussions/1636