Chapter 18. with, match, and else Blocks

Table of Contents

this chapter is about control flow structures that are especially powerful in python
magic of with statements and how the context manager gives safety
the magic of match statements and how that is expressive for languages (including custom DSLs)

What’s New in This Chapter #

Context Managers and with Blocks #

context managers exist to control a with statement
analogous to for statements controlled by iterators
MISCONCEPTIONS:
this is correct: a finally block is always guaranteed to run, even if the try block has a return, sys.exit() or an exception raised.
I just never paid attention to this.
That’s why it’s good for cleanup: resource release / reverting or undoing temporary state changes
with blocks don’t define a new scope like how functions do, that’s why the names are accessible outside of the block
syntax:
- in with open('mirror.py') as fp:,
  - evaluating the expression after the with gives the context manager object, i.e. open('mirror.py')
    the context manager object here is an instance of TextIOWrapper, this is what the open() function returns.
    the __enter__ method of TextIOWrapper returns self
  - the target variable is within the as clause is bound to something
    the as clause is optional
  - that something is the result returned by the __enter__ method of the context manager object (TextIOWrapper), which we determined was self (i.e. the context manager instance)
- for any reason, when the control flow exists the with block, then __exit__ is called on the context manager object.
  This is NOT called on whatever that was returned by __enter__ and stored by the target variable.

example code Custom Context Manager for mirror

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
  """
  A "mirroring" ``stdout`` context.

  While active, the context manager reverses text output to
  ``stdout``::

  # tag::MIRROR_DEMO_1[]

      >>> from mirror import LookingGlass
      >>> with LookingGlass() as what:  # <1>
      ...      print('Alice, Kitty and Snowdrop')  # <2>
      ...      print(what)
      ...
      pordwonS dna yttiK ,ecilA
      YKCOWREBBAJ
      >>> what  # <3>
      'JABBERWOCKY'
      >>> print('Back to normal.')  # <4>
      Back to normal.

  # end::MIRROR_DEMO_1[]


  This exposes the context manager operation::

  # tag::MIRROR_DEMO_2[]

      >>> from mirror import LookingGlass
      >>> manager = LookingGlass()  # <1>
      >>> manager  # doctest: +ELLIPSIS
      <mirror.LookingGlass object at 0x...>
      >>> monster = manager.__enter__()  # <2>
      >>> monster == 'JABBERWOCKY'  # <3>
      eurT
      >>> monster
      'YKCOWREBBAJ'
      >>> manager  # doctest: +ELLIPSIS
      >... ta tcejbo ssalGgnikooL.rorrim<
      >>> manager.__exit__(None, None, None)  # <4>
      >>> monster
      'JABBERWOCKY'

  # end::MIRROR_DEMO_2[]

  The context manager can handle and "swallow" exceptions.

  # tag::MIRROR_DEMO_3[]

      >>> from mirror import LookingGlass
      >>> with LookingGlass():
      ...      print('Humpty Dumpty')
      ...      x = 1/0  # <1>
      ...      print('END')  # <2>
      ...
      ytpmuD ytpmuH
      Please DO NOT divide by zero!
      >>> with LookingGlass():
      ...      print('Humpty Dumpty')
      ...      x = no_such_name  # <1>
      ...      print('END')  # <2>
      ...
      Traceback (most recent call last):
        ...
      NameError: name 'no_such_name' is not defined

  # end::MIRROR_DEMO_3[]

  """


  # tag::MIRROR_EX[]
  import sys

  class LookingGlass:

      def __enter__(self):  # <1>
          self.original_write = sys.stdout.write  # <2>
          sys.stdout.write = self.reverse_write  # <3>
          return 'JABBERWOCKY'  # <4>

      def reverse_write(self, text):  # <5>
          self.original_write(text[::-1])

      def __exit__(self, exc_type, exc_value, traceback):  # <6>
          sys.stdout.write = self.original_write  # <7>
          if exc_type is ZeroDivisionError:  # <8>
              print('Please DO NOT divide by zero!')
              return True  # <9>
          # <10> NOTE: if exit returns None or any falsy value, any exception raised in the =with= block will be propagated.
  # end::MIRROR_EX[]

enter and exit:
1. __enter__ is called without any arguments other than the implicit self (which is the context manager instance)
2. __exit__ is called with 3 arguments:
  - exc_type
  - exc_value the actual exception instance
  - traceback
  these 3 args received by self are the same as what happens if we call sys.exc_info() in the finally block of a try/finally. in the past, calling that was necessary to determine how to do the cleanup.

we can now do parenthesized context managers:

1
2
3
4
5
6
  with (
          CtxManager1() as example1,
          CtxManager2() as example2,
          CtxManager3() as example3,
  ):
          ...

thanks to a new parser from python 3.10 onwards

The contextlib Utilities #

first-reach before writing custom context managers
things that look useful:
- using @contextmanager to build a context manager from a generator function
- ContextDecorator to define class-based context managers
- the async versions of all of them

Using @contextmanager #

just implement a generator with a single yield statement that should produce whatever you want the __enter__ method to return
the yield splits the function body into two parts:
- before yield: gets executed at the beginning of the with block when interpreter calls __enter__
- after yield: gets executed when __exit__ is called at the end of the block

correct example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
  """
  A "mirroring" ``stdout`` context manager.

  While active, the context manager reverses text output to
  ``stdout``::

  # tag::MIRROR_GEN_DEMO_1[]

      >>> from mirror_gen import looking_glass
      >>> with looking_glass() as what:  # <1>
      ...      print('Alice, Kitty and Snowdrop')
      ...      print(what)
      ...
      pordwonS dna yttiK ,ecilA
      YKCOWREBBAJ
      >>> what
      'JABBERWOCKY'

  # end::MIRROR_GEN_DEMO_1[]


  This exposes the context manager operation::

  # tag::MIRROR_GEN_DEMO_2[]

      >>> from mirror_gen import looking_glass
      >>> manager = looking_glass()  # <1>
      >>> manager  # doctest: +ELLIPSIS
      <contextlib._GeneratorContextManager object at 0x...>
      >>> monster = manager.__enter__()  # <2>
      >>> monster == 'JABBERWOCKY'  # <3>
      eurT
      >>> monster
      'YKCOWREBBAJ'
      >>> manager  # doctest: +ELLIPSIS
      >...x0 ta tcejbo reganaMtxetnoCrotareneG_.biltxetnoc<
      >>> manager.__exit__(None, None, None)  # <4>
      False
      >>> monster
      'JABBERWOCKY'

  # end::MIRROR_GEN_DEMO_2[]

  The context manager can handle and "swallow" exceptions.
  The following test does not pass under doctest (a
  ZeroDivisionError is reported by doctest) but passes
  if executed by hand in the Python 3 console (the exception
  is handled by the context manager):

  # tag::MIRROR_GEN_DEMO_3[]

      >>> from mirror_gen_exc import looking_glass
      >>> with looking_glass():
      ...      print('Humpty Dumpty')
      ...      x = 1/0  # <1>
      ...      print('END')  # <2>
      ...
      ytpmuD ytpmuH
      Please DO NOT divide by zero!

  # end::MIRROR_GEN_DEMO_3[]

      >>> with looking_glass():
      ...      print('Humpty Dumpty')
      ...      x = no_such_name  # <1>
      ...      print('END')  # <2>
      ...
      Traceback (most recent call last):
        ...
      NameError: name 'no_such_name' is not defined



  """


  # tag::MIRROR_GEN_EXC[]
  import contextlib
  import sys

  @contextlib.contextmanager
  def looking_glass():
      original_write = sys.stdout.write

      def reverse_write(text):
          original_write(text[::-1])

      sys.stdout.write = reverse_write
      msg = ''  # <1>
      try:
          yield 'JABBERWOCKY'
      except ZeroDivisionError:  # <2>
          msg = 'Please DO NOT divide by zero!'
      finally:
          sys.stdout.write = original_write  # <3>
          if msg:
              print(msg)  # <4>
  # end::MIRROR_GEN_EXC[]

it’s unavoiadable to use the try/except when using the yield when using @contextmanager since we never know what the users of the context managers will do.
GOTCHA:
Generally,
if __exit__ returns truthy even if there’s an exception, then the exception is suppressed. If it’s not truthy, then the exception is propagated out
HOWEVER, with @contextmanager, the default behaviour is inverted. the __exit__ method provided by the decorator assumes any exception sent into the generator is handled and should be suppressed.

(flawed) example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
  """
  A "mirroring" ``stdout`` context manager.

  While active, the context manager reverses text output to
  ``stdout``::

  # tag::MIRROR_GEN_DEMO_1[]

      >>> from mirror_gen import looking_glass
      >>> with looking_glass() as what:  # <1>
      ...      print('Alice, Kitty and Snowdrop')
      ...      print(what)
      ...
      pordwonS dna yttiK ,ecilA
      YKCOWREBBAJ
      >>> what
      'JABBERWOCKY'
      >>> print('back to normal')
      back to normal


  # end::MIRROR_GEN_DEMO_1[]


  This exposes the context manager operation::

  # tag::MIRROR_GEN_DEMO_2[]

      >>> from mirror_gen import looking_glass
      >>> manager = looking_glass()  # <1>
      >>> manager  # doctest: +ELLIPSIS
      <contextlib._GeneratorContextManager object at 0x...>
      >>> monster = manager.__enter__()  # <2>
      >>> monster == 'JABBERWOCKY'  # <3>
      eurT
      >>> monster
      'YKCOWREBBAJ'
      >>> manager  # doctest: +ELLIPSIS
      >...x0 ta tcejbo reganaMtxetnoCrotareneG_.biltxetnoc<
      >>> manager.__exit__(None, None, None)  # <4>
      False
      >>> monster
      'JABBERWOCKY'

  # end::MIRROR_GEN_DEMO_2[]

  The decorated generator also works as a decorator:


  # tag::MIRROR_GEN_DECO[]
      >>> @looking_glass()
      ... def verse():
      ...     print('The time has come')
      ...
      >>> verse()  # <1>
      emoc sah emit ehT
      >>> print('back to normal')  # <2>
      back to normal

  # end::MIRROR_GEN_DECO[]

  """


  # tag::MIRROR_GEN_EX[]
  import contextlib
  import sys

  @contextlib.contextmanager  # <1>
  def looking_glass():
      original_write = sys.stdout.write  # <2>

      def reverse_write(text):  # <3>
          original_write(text[::-1])

      sys.stdout.write = reverse_write  # <4>
      yield 'JABBERWOCKY'  # <5>
      sys.stdout.write = original_write  # <6>
  # end::MIRROR_GEN_EX[]

this is flawed because if an exception is raised in the body of the with block, the Python interpreter will catch it and raise it again in the yield expression inside looking_glass. But there is no error handling there, so the looking_glass generator will terminate without ever restoring the original sys.stdout.write method, leaving the system in an invalid state.

Cleanup not done if there’s an exception raised within the with block.

TRICK: generators decorated with it can also be used as decorators themselves.
happens because @contextmanager is implemented with the contextlib.ContextDecorator class.
1 2 3 4 5
@looking_glass() def verse(): print("the time has come") verse() # returns in reverse
Here, looking_glass does its job before and after the body of verse runs.

Pattern Matching in lis.py: A Case Study #

Scheme Syntax #

Imports and Types #

The Parser #

The Environment #

The REPL #

The Evaluator #

Procedure: A Class Implementing a Closure #

Using OR-patterns #

Do This, Then That: else Blocks Beyond if #

use cases: avoids the need to setup extra control flags or coding extra if statements
- raising pattern:
  1 2 3 4 5
  for item in my_list: if item.flavor == 'banana': break else: raise ValueError('No banana flavor found!')
- keep the try blocks lean in try/except
  The body of the try block should only have the statements that generate the expected exceptions.
  Instead of doing this:
  1 2 3 4 5
  try: dangerous_call() after_call() except OSError: log('OSError...')
  We should do:
  1 2 3 4 5 6
  try: dangerous_call() except OSError: log('OSError...') else: after_call()
This gives clarity.
try block is guarding against possible errors in dangerous_call() and not in after_call(). It’s also explicit that after_call() will only execute if no exceptions are raised in the try block.
IDIOM: try/except is NOT only for error handling, it can be used for control flow as well. E.g. duck typing type checks.
Follows EAFP
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.
this contrasts LBYL:
LBYL
Look before you leap. This coding style explicitly tests for pre-conditions before making calls or lookups. This style contrasts with the EAFP approach and is characterized by the presence of many if statements. In a multi-threaded environment, the LBYL approach can risk introducing a race condition between “the looking” and “the leaping.” For example, the code, if key in mapping: return mapping[key] can fail if another thread removes key from mapping after the test, but before the lookup. This issue can be solved with locks or by using the EAFP approach.

else blocks apply to most control flow constructs, they are closely related to each other but very different from if/else
LANGUAGE_LIMITIATION: GOTCHA: the else keyword is more of more of a “run this loop, then do that” instead of “Run this loop, otherwise do that”
- for: else block will run only if and when the for loop runs to completion (i.e., not if the for is aborted with a break).
- while: else block will run only if and when the while loop exits because the condition became falsy (i.e., not if the while is aborted with a break).
- try: The else block will run only if no exception is raised in the try block.
  NOTE: “Exceptions in the else clause are not handled by the preceding except clauses.”
the else clause is also skipped if an exception or a return, break, or continue statement causes control to jump out of the main block of the compound statement.

Chapter Summary #

an insight:
subroutines are the most important invention in the history of computer languages. If you have sequences of operations like A;B;C and P;B;Q, you can factor out B in a subroutine. It’s like factoring out the filling in a sandwich: using tuna with different breads. But what if you want to factor out the bread, to make sandwiches with wheat bread, using a different filling each time? That’s what the with statement offers. It’s the complement of the subroutine.