↓Skip to main content
  1. Readings/
  2. Books/
  3. Fluent Python: Clear, Concise, and Effective Programming – Luciano Ramalho/

Chapter 2. An Array of Sequences

··4891 words·23 mins
Table of Contents

What’s New in This Chapter #

Overview of Built-in Sequences #

  • two factors to group sequences by:
    1. by container (heterogeneous) / flat (homogeneous) sequences
      1. Container sequences: can be heterogeneous
        • holds references (“pointers”)
      2. Flat sequences: are homogeneous
        • holds values
    2. by mutability / immutability
  • things like generators can be seen in the context of sequences themselves “To fill up sequences of any type”

Mem Representation for Python Objects: have a header (with metadata) and value #

example of meta fields (using float as a reference):

  1. refcount
  2. type
  3. value

Every Python object in memory has a header with metadata. The simplest Python object, a float, has a value field and two metadata fields: • ob_refcnt: the object’s reference count • ob_type: a pointer to the object’s type • ob_fval: a C double holding the value of the float On a 64-bit Python build, each of those fields takes 8 bytes. That’s why an array of floats is much more compact than a tuple of floats: the array is a single object holding the raw values of the floats, while the tuple consists of several objects—the tuple itself and each float object contained in it.

List Comprehensions and Generator Expressions #

List Comprehensions and Readability #

  • a loop has generic purpose, but a listcomp’s purpose is always singular: to build a list
  • we should stick to this purpose and not introduce abuse mechanisms like adding in side-effects from listcomp evaluations
  • List comprehensions build lists from sequences or any other iterable type by filtering and transforming items.

Scope: listcomps have a local scope, use walrus operator to expand the scope to its outer frame #

``Local Scope Within Comprehensions and Generator Expressions''

if that name is modified using global or nonlocal, then the scope is accordingly set

defines the scope of the target of := as the enclos‐ ing function, unless there is a global or nonlocal declaration for that target.

Listcomps Versus map and filter #

Cartesian Products #

This is the part where we have more than one iterable within the listcomp

Generator Expressions #

Tuples Are Not Just Immutable Lists #

The immutable list part is definitely one of the main features.

It should also be seen as a nameless record.

Tuples as Records #

  • some examples of tuple unpacking:
    1. the loop constructs automatically support unpacking, we can assign vars even for each iteration of the loop
    2. the % formatting operator will also unpack values within the tuple when doing string formats

Tuples as Immutable Lists #

2 benefits:

  1. clarity: the length of tuple is fixed thanks to its immutability
  2. performance: memory use is a little better, also allows for some optimisations

Warning: the immutability is w.r.t references contained within the tuple, not values #

So tuples containing mutable items can be a source of bugs Also, unhashable tuple => can’t be inserted as a dict key or set

Tuple’s Performance Efficiency Reasons #

Tuples are more efficient because:

  1. bytecode: tuple has simpler bytecode required: Python compiler generates bytecode for a tuple constant in one operation; but for a list literal, the generated bytecode pushes each element as a separate constant to the data stack, and then builds the list.
  2. constructor: tuple construction from existing doesn’t need any copying, it’s the same reference:
    • the list constructor returns a copy of a given list if its list(l)
    • tuple constructor returns a reference to the same t if we do tuple(t) (because they’re immutable anyway so why not same reference)
  3. amortisation: tuple, since fixed size, doesn’t need to account for future size changes by amortising that operation
  4. no extra layer of indirection The references to the items in a tuple are stored in an array in the tuple struct,while a list holds a pointer to an array of references stored elsewhere. The indirection is necessary because when a list grows beyond the space currently allocated, Python needs to reallocate the array of references to make room. The extra indirection makes CPU caches less effective.

Comparing Tuple and List Methods #

Unpacking Sequences and Iterables #

  • safer extraction of elements from sequences
  • works with any iterable object as the datasource, including iterators.
    • for the iterable case, as long as the iterable yields exactly one item per variable in the receiving end (or * is used to do a glob capture)

Parallel assignment #

This is the multi-name assignments that we do, and how involves sequence unpacking

most visible form of unpacking is parallel assignment; that is, assigning items from an iterable to a tuple of variables, as you can see in this example: >>> lax_coordinates = (33.9425, -118.408056) >>> latitude, longitude = lax_coordinates # unpacking >>> latitude

Using * to Grab Excess Items #

  • classic case is the use of the grabbing part for varargs
  • context of parallel assignment, the * prefix can be applied to exactly one variable, but it can appear in any position

Unpacking with * in Function Calls and Sequence Literals #

  • the use of the unpacking operator is context-dependent, so in the context of function calls and the creation of sequences, they can be used multiple times. In the context of parallel asisgnment, it’s a singular use (else there’s going to be ambiguity on how to partition values in the sequence)

Nested Unpacking #

GOTCHA: single-item tuple syntax may have silent bugs if used improperly #

Both of these could be written with tuples, but don’t forget the syntax quirk that single-item tuples must be written with a trailing comma. So the first target would be (record,) and the second ((field,),). In both cases you get a silent bug if you forget a comma.

Pattern Matching with Sequences #

Pattern matching is an example of declarative programming: the code describes “what” you want to match, instead of “how” to match it. The shape of the code follows the shape of the data, as Table 2-2 illustrates.

  • here’s the OG writeup for structural pattern matching. Some points from it:
    • Therefore, an important exception is that patterns don’t match iterators. Also, to prevent a common mistake, sequence patterns don’t match strings.
    • the matching primitives allow us to use guards on the match conditions (see here)
    • there’s support for defining sub-patterns like so:
          case (Point(x1, y1), Point(x2, y2) as p2): ...
  • here’s a more comprehensive tutorial PEP 636 - Structural Pattern Matching

Pattern-matching is declarative #

Pattern matching is an example of declarative programming: the code describes “what” you want to match, instead of “how” to match it. The shape of the code fol‐ lows the shape of the data, as Table 2-2 illustrates.

python’s match goes beyond just being a switch statement because it supports destructuring similar to elixir #

  • random thought: this features is really useful if we were to write out a toy interpreter for some source code. Here’s lis.py

On the surface, match/case may look like the switch/case statement from the C lan‐ guage—but that’s only half the story.4 One key improvement of match over switch is destructuring—a more advanced form of unpacking. Destructuring is a new word in the Python vocabulary, but it is commonly used in the documentation of languages that support pattern matching—like Scala and Elixir. As a first example of destructuring, Example 2-10 shows part of Example 2-8 rewrit‐ ten with match/case.

class-patterns gift us the ability to do runtime type checks #

case [str(name), _, _, (float(lat), float(lon))]:

  • the constructor-like syntax is not a constructor, it’s a runtime check

  • the names (name, lat, lon) are binded here and are available for referencing thereafter within the codeblock

  • this is really interesting, it’s in the context of patterns that the syntax does runtime type checking and the code does

The expressions str(name) and float(lat) look like constructor calls, which we’d use to convert name and lat to str and float. But in the context of a pattern, that syntax performs a runtime type check: the preceding pattern will match a four-item sequence in which item 0 must be a str, and item 3 must be a pair of floats. Additionally, the str in item 0 will be bound to the name variable, and the floats in item 3 will be bound to lat and lon, respectively. So, although str(name) borrows the syntax of a constructor call, the semantics are completely different in the context of a pattern. Using arbitrary classes in patterns is covered in “Pattern Matching Class Instances” on page 192.

Pattern Matching Sequences in an Interpreter #

  • it’s interesting how the python 2 code was described as “a fan of pattern matching” because it matches on the first element and then the tree of control flow paths does their job, so it’s really like a switch

  • this switch-like pattern-matching style is something abstract even more so than in it’s concrete programming language implementation that we have been discussing so far

  • the catch-all is used for error-handling purposes here. In general there should always be a fallthrough case instead of going for no-ops which will end up being more silent

Slicing #

Why Slices and Ranges Exclude the Last Item #

this refers to the fact that one end of the range is closed (inclusive) and the other is open (exclusive).

  • easy to calculate lengths
  • easy to split / partition without creating overlaps

Slice Objects #

  • useful to know this because it lets you assign names to slices, like spreadsheets allow the naming of cell-ranges

Multidimensional Slicing and Ellipsis #

This is more useful in the context of numpy lib, the book doens’t include examples here for the python stdlib

  • built-ins are single dim, except for memoryview Except for memoryview, the built-in sequence types in Python are one-dimensional, so they support only one index or slice, and not a tuple of them.
  • Multiple indexes or slices get passed in as tuples a[i,j] is evaluated as a.__getitem__((i,j)) e.g. numpy multi-dim array accesses
  • ellipsis class is a singleton, the sole object being Elipsis
    • a similar case is bool class and True, False
  • so in numpy, if x is a four-dimensional array, x[i, ...] is a shortcut for x[i, :, :, :,]

Assigning to Slices #

Applies to mutable sequences.

Gotcha: when LHS of assignment is slice, the RHS must be iterable #

In the example below, we’re trying to graft some sequence to another. With that intent, we can only graft an iterable onto another sequence, not a single element. Hence, the requirement that the RHS must be iterable.

l = list(range(10))
try:
        # so this is wrong:
        l[2:5] = 100
except:
        print("this will throw an error, we aren't passing in an iterable for the grafting.")

finally:
        # and this is right
        l[2:5] = [100]

print(l)

Using + and * with Sequences #

  • both + and - create new objects without modding their operands

Building Lists of Lists #

Gotcha: Pitfall of references to mutable objects – using a * n where a contains sequence of mutable items can be problematic #

Actually applies to other mutable sequences as well, in this case it’s just a list that we’re using

Just be careful what the contained element’s properties are like.


my_mutable_elem = ['apple', 'banana']
print(f"my mutable elem ref: {id(my_mutable_elem)}")

list_of_lists = [ my_mutable_elem ] * 2
print(f"This creates 2 repeats \n{list_of_lists}")
print(f"(first ref, second ref) = {[id(elem) for elem in list_of_lists]}")

list_of_lists[0][0] = 'strawberry'
print(f"This mods all 2 repeated refs \n{list_of_lists}")
print(f"(first ref, second ref) = {[id(elem) for elem in list_of_lists]}")

Here’s the same gotcha using tic-tac-toe as an example:

good_board = [['_'] * 3 for i in range(3)]

bad_board = [['_'] * 3] * 3

print(f"BEFORE, the boards look like this:\n\
\tGOOD Board:\n\
\t{ [row for row in good_board] }\n\
\tBAD Board:\n\
\t{ [row for row in bad_board] }\n")

# now we make a mark on the boards:
good_board[1][2] = 'X'
bad_board[1][2] = 'X'


print(f"AFTER, the boards look like this:\n\
\tGOOD Board:\n\
\t{ [row for row in good_board] }\n\
\tBAD Board:\n\
\t{ [row for row in bad_board] }\n")

Augmented Assignment with Sequences #

This refers to the in-place versions of the sequence operators. in ==, there are 2 cases:

  • Case A: Identity of a changes

    • the dunder method __iadd__ was not available for use
    • so a + b had to be evaluated and stored as a new id
    • and that id was then referenced by a as part of the new assignment

  • Case B: Identity of a does not change

    • this would mean that a is actually mutated in-place
    • it would have used the dunder method __iadd__

In other words, the identity of the object bound to a may or may not change, depending on the availability of __iadd__.

In general, for mutable sequences, it is a good bet that __iadd__ is implemented and that += happens

doing += for repeated concats of immutable sequences is inefficient #

however, str contacts have been optimised in CPython, it’s alright to do that in CPython. Extra space would have been allocated to amortise the new space allocations.

A += Assignment Puzzler #

Learnings! #

I take three lessons from this:

• Avoid putting mutable items in tuples.

• Augmented assignment is not an atomic operation—we just saw it throwing an exception after doing part of its job.

• Inspecting Python bytecode is not too difficult, and can be helpful to see what is going on under the hood.

Example #

it’s a peculiarity in the += operator!

Learnings:

I take three lessons from this:

• Avoid putting mutable items in tuples.

• Augmented assignment is not an atomic operation—we just saw it throwing an exception after doing part of its job.

• Inspecting Python bytecode is not too difficult, and can be helpful to see what is going on under the hood.

t = (1,2, [30, 40])
print(t)
try:
        t[2] += [50, 60]
except:
        print("LMAO complaints")
finally:
        print(t)

try:
        t[2].extend([90, 100])

except:
        print("this won't error out though")

finally:
        print(t)

list.sort Versus the sorted Built-In #

in-place functions should return None as a convention #

There’s a drawback to this: we can’t cascade calls to this method

python’s sorting uses timsort! #

Managing Ordered Sequences with bisect (extra ref from textbook) #

When a List Is Not the Answer #

Arrays: best for containing numbers #

  • an array of float values does not hold full-fledged float instances, but only the packed bytes representing their machine values—similar to an array of double in the C language.
  • examples:
    1. typecode b => byte => 8 bits over signed and unsigned regions ==> [-128, 127] range of representation
  • for special cases of numeric arrays for bin data (e.g. raster images), bytes and bytearray types are more appropriate!

Memory Views #

Examples #

  • id vs context

    The learning from this is that the memoryview objects and the memory that they provide a view of are two different regions of memory. id vs context.

    So here, m2, m3 and all have different id references, but the memory region that they give a view of is all the same.

    That’s why we can mutate using one memory view and every other view also reflects that change.

    from array import array

# just some bytes, the sequence is buffer-protocol-adherent
octets = array("B", range(6))
print(octets)

# builds a new memoryview from the array
m1 = memoryview(octets)
print(m1)
# exporting of a memory view to a list, this creates a new list (a copy!)
print(m1.tolist())

# builds a new memoryview, with 2 rows and 3 columns
m2 = m1.cast('B', [2,3])
print(m2)
print(m2.tolist())


m3 = m1.cast('B', [3,2])
print(m3)
print(m3.tolist())

# overwrite byte
m2[1,1] = 22

# overwrite byte
m3[1,1] = 33

print(f"original memory has been changed: \n\t{octets} ")
print(f"m1 has been changed:\n\t { m1.tolist() }")
print(f"m2 has been changed:\n\t { m2.tolist() }")
print(f"m3 has been changed:\n\t {m3.tolist()}")

  • corruption

    from array import array
from sys import byteorder

print(byteorder)
numbers = array('h', [-2,-1,0,1,2])
memv = memoryview(numbers)

print(len(memv))

print(memv[0])

# cast the half as a byte, so the resultant sequence will have double the elements:
memv_oct = memv.cast('B')
# the numbers are stored in little endian format
print(memv_oct.tolist())
# so -2 as a 2-byte signed short will be (little endian binary) 0xfe 0xff (254, 255)

# so we get:
# -2: 0xfe 0xff (254, 255)
# -1: 0xff 0xff (255, 255)
# 0: 0x00 0x00 (0, 0)
# 1: 0x01 0x00 (1, 0)
# 2: 0x02 0x00 (2, 0)


# asisgns the value of 4 to byte-offset 5
memv_oct[5] = 4
print( numbers )
# so this change is to the 2nd byte of the third element of numbers
# byte index 5 is the high byte (since it's little endian so bytes are low -> high)
# so the 3rd element is now [0, 0x0400]
# = a + (b*256) = 0 + (4 * 256) is 1024 in decimal

# NOTE: Note the change to numbers: a 4 in the most significant byte of a 2-byte unsigned
# integer is 1024.

Extra: “Parsing binary records with struct” #

Here’s the reference.

Some takeaways:

  • Proprietary binary records in the real world are brittle and can be corrupted easily. examples:
    1. string parsing: paddings, null terminated, size limits?
    2. endianness problem: what byteorder was used for representing integers and floats (CPU-architecture-dependent)?
  • always explore pre-built solutions first instead of building yourself:
    • for data exchange, pickle module works great, but have to ensure python versions align since the default binary formats may be different. Reading a pickle also may run arbitrary code.
  • if the binary exchange uses multiple programming languages, standardise the serialisation. Serial forms:
    1. multi-platform binary serialisation formats:
      1. MessagePack
      2. ProtocolBuffers
    2. JSON

bot assisted concept mapping #

Here’s a bot-assisted concept map between unix mmap and memoryviews:

Memory mapping a file is a powerful technique that allows access to file data as if it were in memory, and the concepts connect naturally between the Unix world (via `mmap` system calls) and Python (via the `mmap` module and `memoryview` objects).

**Unix World: mmap**

- **Definition:** The Unix `mmap` system call maps files or devices into a process's address space, enabling file I/O by reading and writing memory. This is efficient for large files because data is loaded on demand, and multiple processes can share the same mapped region[1].
- **Usage:** After opening a file, `mmap` associates a region of virtual memory with the file. Reading and writing to this memory behaves as if you were reading and writing to the file itself. The system manages when data is actually read from or written to disk, often using demand paging[1].
- **Types:** Both file-backed (mapping a file) and anonymous (not backed by a file, similar to dynamic allocation) mappings are supported. Shared mappings allow interprocess communication, while private mappings isolate changes to one process[1].

**Python World: mmap Module**

- **Definition:** Python’s `mmap` module provides a high-level interface to memory-mapped files, closely mirroring the Unix `mmap` functionality. You must provide a file descriptor (from `fileno()` or `os.open()`) and specify the mapping size and access mode[2][3].
- **Usage:** Memory-mapped file objects behave like both file objects and mutable byte arrays, allowing random access and slicing. You can read and write data by indexing or slicing, and you can seek through the file as if it were a standard file object[2][3].
- **Access Modes:** You can specify read-only, write-through (changes go directly to the file), or copy-on-write (changes are local)[4][5].

**Python World: memoryview**

- **Definition:** The `memoryview` object provides a safe way to access the internal data of an object that supports the buffer protocol, such as `bytes`, `bytearray`, or a memory-mapped file object[6].
- **Usage:** With a `memoryview`, you can manipulate large data structures without copying them, which is especially useful for memory-mapped files. It allows you to pass a view of the data to functions or slice it efficiently, avoiding unnecessary memory duplication[6].
- **Connection to mmap:** When you create a memory-mapped file in Python, you can use `memoryview` to work with the data as a buffer, enabling efficient manipulation and sharing of large data sets.

**Linking the Concepts**

- **Memory Mapping:** Both Unix `mmap` and Python’s `mmap` module allow you to treat file data as if it were in memory, enabling efficient random access and sharing between processes[1][2][3].
- **Efficient Data Access:** Using `memoryview` in Python is analogous to working directly with the mapped memory region in Unix, as both avoid copying large chunks of data and allow efficient manipulation of file contents[6].
- **Interprocess Communication:** In Unix, shared memory mappings (`MAP_SHARED`) allow processes to communicate by reading and writing the same memory region. In Python, you can achieve similar effects by sharing a memory-mapped file object between processes[1][2].
- **Performance:** Both approaches leverage the operating system’s memory management to reduce I/O overhead and enable fast, random access to file data.

**Summary Table**

| Concept                | Unix (`mmap`)                | Python (`mmap` module)           | Python (`memoryview`)         |
|------------------------|------------------------------|-----------------------------------|-------------------------------|
| Purpose                | Map files to memory          | Map files to memory               | View memory as buffer         |
| Access Method          | System call                  | Module/object                     | Object                        |
| Sharing                | Shared/private mappings      | Shared via file object            | View of existing buffer       |
| Efficiency             | Demand paging, no copy       | Demand paging, no copy            | No copy, efficient slicing    |
| Use Case               | IPC, efficient file I/O      | Efficient file I/O, IPC           | Efficient data manipulation   |

By understanding these connections, you can leverage memory mapping for efficient file handling and data sharing across both Unix and Python environments.

[1] https://en.wikipedia.org/wiki/Mmap
[2] https://docs.python.org/3/library/mmap.html
[3] https://github.com/python/cpython/blob/master/Doc/library/mmap.rst
[4] https://pymotw.com/3/mmap/
[5] https://realpython.com/python-mmap/
[6] https://smart-spatial.com/data%20science/2017/09/22/MemoryView/
[7] https://stackoverflow.com/questions/63553692/how-to-use-memory-mapped-file-in-python-linux/63554607
[8] https://pymotw.com/3/mmap/index.html
[9] https://unix.stackexchange.com/questions/712651/does-mmap-allow-creating-a-mapping-that-is-much-larger-than-the-amount-of-physic
[10] https://deepaksood619.github.io/computer-science/operating-system/memory-mapping-mmap/
[11] https://www.cmi.ac.in/~madhavan/courses/prog2-2012/docs/python-3.2.1-docs-html/library/mmap.html
[12] https://www.blopig.com/blog/2024/08/memory-mapped-files-for-efficient-data-processing/
[13] https://stackoverflow.com/questions/4991533/sharing-memory-between-processes-through-the-use-of-mmap/4991631
[14] https://documentation.help/Python-2.4/module-mmap.html
[15] https://docs.python.org/3.4/library/mmap.html?highlight=mmap
[16] https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-mmap-map-pages-memory
[17] https://man7.org/linux/man-pages/man2/mmap.2.html
[18] https://programmingappliedai.substack.com/p/what-is-mmap-in-linux-and-how-it
[19] https://w3.cs.jmu.edu/kirkpams/OpenCSF/Books/csf/html/MMap.html
[20] https://www.ibm.com/docs/ssw_ibm_i_74/apis/mmap.htm
[21] https://www.unix.com/man-page/linux/3/mmap/
[22] https://stackoverflow.com/questions/47437481/python-can-i-get-a-memoryview-or-a-bytearray-mapping-to-a-mmap
[23] https://docs.vultr.com/python/built-in/memoryview
[24] https://pymotw.com/2/mmap/
[25] https://www.youtube.com/watch?v=ky1n6luzL3Y

“casting” in memoryview vs Elixir ecto “casting” #

The mental models are different.

“casting” with memoryview is about interpreting memory as a different type, not about automatic type conversion or validation as in Elixir changesets. The term is used more loosely in Python, while in Elixir, it is a formal, declarative operation for data preparation.

The use of the term **"cast"** in the context of Python `memoryview` is not the same as in Elixir changesets, even though both involve types or transformations.

**Python `memoryview` and Casting**

- **Casting in Python `memoryview`:**
  When people refer to "casting" with `memoryview`, they usually mean taking a slice of a memoryview or viewing the underlying buffer as a different type (sometimes using modules like `ctypes` or `numpy` to achieve type reinterpretation). However, the built-in `memoryview` object itself does not perform type conversion; it provides a view into the memory buffer of an object that supports the buffer protocol, such as `bytes`, `bytearray`, or a NumPy array[5][6][1].
- **Type Safety:**
  The `memoryview` object allows you to manipulate the data as if it were a sequence of bytes or another type (if the underlying buffer supports it), but it does not automatically convert the data to a different type. If you want to reinterpret the data as a different type (e.g., casting bytes to an integer array), you typically use additional libraries or manual conversion, not `memoryview` itself[5][1].
- **No Implicit Type Conversion:**
  The `memoryview` simply provides a window into the existing data, preserving the memory layout and avoiding data copying[6][8]. Any "casting" is more about how you interpret the bytes, not a formal declaration or transformation of types.

**Elixir Changesets and Casting**

- **Casting in Elixir Changesets:**
  In Elixir's Ecto changesets, "casting" refers to the explicit declaration of how data (often from a web form or external source) should be converted or validated into a specific type before being inserted into a database. This is a declarative process that ensures data integrity and type safety.
- **Type Conversion:**
  Casting in Elixir changesets involves actual type conversion (e.g., from a string to an integer or date), and is a core part of the data validation and preparation process.

**Comparison Table**

| Context                | Meaning of "Cast"                        | Type Conversion | Purpose                        |
|------------------------|------------------------------------------|-----------------|--------------------------------|
| Python `memoryview`    | Viewing/reinterpreting memory as a type  | No (manual only)| Efficient data access/manipulation |
| Elixir changesets      | Declaring/validating/converting types    | Yes             | Data validation, preparation   |

**Summary**

In Python, "casting" with `memoryview` is about interpreting memory as a different type, not about automatic type conversion or validation as in Elixir changesets. The term is used more loosely in Python, while in Elixir, it is a formal, declarative operation for data preparation[5][6][1].

[1] https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html
[2] https://stackoverflow.com/questions/71028367/how-to-make-a-custom-python-class-to-support-memoryview-conversion
[3] https://stackoverflow.com/questions/48136122/how-do-i-cast-a-slice-of-memoryview-into-c-string-unsigned-char
[4] https://docs.python.org/3/library/stdtypes.html
[5] https://blog.finxter.com/converting-python-bytes-to-memoryview/
[6] https://realpython.com/ref/builtin-functions/memoryview/
[7] https://www.stratascratch.com/blog/understanding-data-types-in-python-with-examples/
[8] https://docs.python.org/3/c-api/memoryview.html

NumPy #

python’s Global Interpreter Lock (GIL) and how releasing it unlocks better parallelisation #

  • it’s a mutex that protects access to python objects and therefore prevents multiple native threads from executing Python bytecode simultaneously within the same process.

    It was intended to be a simplification mechanism to make memory handling simpler but that also means having this mutex limits parallelism.

    Typical workarounds:

    1. multi-processing, separate processes, each with their own GIL

    2. offload CPU-intensive work to C-extensions or libs that release the GIL

here’s a bot-written outline on it:

The **Global Interpreter Lock (GIL)** is a core mechanism in CPython, the reference implementation of Python, that ensures only one thread executes Python bytecode at a time, even on multi-core processors[2][4][5]. Here’s a detailed overview:

## **What Is the GIL?**

- **Definition:** The GIL is a mutex (mutual exclusion lock) that protects access to Python objects, preventing multiple native threads from executing Python bytecode simultaneously within the same process[2][4][7].
- **Purpose:** It exists primarily to simplify CPython’s memory management, especially reference counting, which is not thread-safe by default. Without the GIL, concurrent access to Python objects could lead to race conditions and memory corruption[5][7].

## **How Does the GIL Work?**

- **Single Thread Execution:** Only one thread holds the GIL at any moment, meaning only one thread can execute Python code at a time, even if you have multiple threads running[2][4][6].
- **Thread Switching:** The interpreter periodically releases the GIL, allowing other threads to acquire it and execute Python code. This switching happens frequently, but it means that CPU-bound multithreaded Python programs do not benefit from multiple cores for parallel execution of Python code[2][4].
- **Non-Python Code:** Operations that do not require the Python interpreter (such as I/O or some C extensions like NumPy) can release the GIL, allowing other threads to run Python code or the process to use multiple cores for those operations[2][4].

## **Why Does the GIL Exist?**

- **Memory Management:** Simplifies reference counting and garbage collection by ensuring thread safety for Python objects[5][7].
- **C Extensions:** Makes it easier to write and use C extensions by providing a stable, single-threaded environment for their execution[1][3][7].
- **Implementation Simplicity:** Using a single lock is easier to implement and maintain than fine-grained locking for all Python objects[1][7].

## **Implications of the GIL**

- **Limited Parallelism:** The GIL prevents true parallel execution of Python code in multi-threaded programs, making it a bottleneck for CPU-bound tasks[2][4][5].
- **Workarounds:** For parallelism, Python developers often use multiprocessing (which uses separate processes, each with its own GIL) or offload CPU-intensive work to C extensions or libraries that release the GIL[1][4].
- **Performance Impact:** The GIL can degrade performance in multi-threaded, CPU-bound applications. However, for I/O-bound or single-threaded programs, its impact is minimal[2][4][6].

## **Future of the GIL**

- **Potential Removal:** The Python Steering Council has indicated support for PEP 703, which proposes making a version of CPython without the GIL. This could enable true multi-threaded parallelism in Python in the future[3].
- **Challenges:** Removing the GIL is complex due to backward compatibility and the reliance of many extensions on its guarantees[3][2].

## **Summary Table**

| Feature                | Description                                                                 |
|------------------------|-----------------------------------------------------------------------------|
| Purpose                | Protect Python objects, simplify memory management, enable C extensions     |
| Execution Model        | Only one thread executes Python bytecode at a time                          |
| Impact on Parallelism  | Limits CPU-bound parallelism in multi-threaded Python code                  |
| Workarounds            | Multiprocessing, C extensions, I/O-bound operations                         |
| Future                 | Potential removal via PEP 703, but challenges remain                        |

The GIL is a key part of Python’s design, balancing simplicity and safety with some limitations for parallel execution[2][4][5].

[1] https://en.wikipedia.org/wiki/Global_interpreter_lock
[2] https://wiki.python.org/moin/GlobalInterpreterLock
[3] https://developer.vonage.com/en/blog/removing-pythons-gil-its-happening
[4] https://realpython.com/python-gil/
[5] https://dev.to/adityabhuyan/understanding-pythons-global-interpreter-lock-gil-and-its-impact-on-concurrency-2da6
[6] https://realpython.com/videos/global-interpreter-lock-overview/
[7] https://dev.to/ohdylan/understanding-pythons-global-interpreter-lock-gil-mechanism-benefits-and-limitations-4aha
[8] https://www.pubnub.com/blog/understanding-pythons-global-interpreter-lock/

NumPy and SciPy are formidable libraries, and are the foundation of other awesome tools such as the Pandas—which implements efficient array types that can hold non‐ numeric data and provides import/export functions for many different formats, like .csv, .xls, SQL dumps, HDF5, etc.—and scikit-learn, currently the most widely used Machine Learning toolset. Most NumPy and SciPy functions are implemented in C or C++, and can leverage all CPU cores because they release Python’s GIL (Global Interpreter Lock). The Dask project supports parallelizing NumPy, Pandas, and scikit-learn processing across clusters of machines. These packages deserve entire books about them.

Deques and Other Queues #

issues with list methods #

although we can use list as a stack / queue (by using .append() or .pop()). However, inserting and removing from the head of the list (the 0-idx end) is costly because the entire list must be shifted in memory => this is why just re-purposing lists is not a good idea.

Characteristics: #

  1. when bounded, every mutation will adhere to the deque capacity for sure.
  2. hidden cost is that removing items from the middle of a deque is not fast
  3. append and popleft are atomic, so can be used for multi-threaded applications without needing locks:w

alternative queues in stdlib #

  • asyncio provides async-programming focused queues

Chapter Summary #

Further Reading #

  • “numpy is all about vectorisation” oprations on array elements without explicit loops

More on Flat vs Container Sequences #

``Flat Versus Container Sequences''