KeiruaProd

I help my clients acquire new users and make more money with their web businesses. I have ten years of experience with the technical aspects of growing SaaS projects using data science for decision-making. If that’s something you need help with, we should get in touch!

Python’s context manager

I was surprised with the with open("somefile"): syntax in Python. This is useful for automatically freeing resources, such as opening files:

with open("some_file") as f:
	print(f.read())
# f is automatically closed at the end of the with statement and cannot be used:
print(f.tell())

If we run this code, you’ll have an error because the file is closed and that’s what is expected:

Traceback (most recent call last):
  File "<stdin>", line xy, in <module>
ValueError: I/O operation on closed file.

There are some more examples in the contextlib package, for instance in order to acquire a database connection for a query.

It turns out this pattern is called a context manager and you can write your own. All you have to do is implement __enter__ and __exit__.

Here is an example where we temporarilly increase the allowed memory limit and recursion depth, then we restore it to its initial value when it’s over.

import resource, sys

# Custom context manager for max recursion depth and memory usage
# https://docs.python.org/3/reference/datamodel.html#context-managers
class ResourceLimit:
    def __init__(self, recursion_limit, stack_limit):
        self.recursion_limit = recursion_limit
        self.stack_limit = stack_limit

    def __enter__(self):
        self.old_recursion_limit = sys.getrecursionlimit()
        self.old_memory_limit = resource.getrlimit(resource.RLIMIT_STACK)
        # https://docs.python.org/3/library/sys.html#sys.setrecursionlimit
        # https://docs.python.org/3/library/resource.html#resource.setrlimit
        sys.setrecursionlimit(self.recursion_limit)
        resource.setrlimit(resource.RLIMIT_STACK, (self.stack_limit, -1))

    def __exit__(self, type, value, tb):
        sys.setrecursionlimit(self.old_recursion_limit)
        resource.setrlimit(resource.RLIMIT_STACK, self.old_memory_limit)

print("before")
print(sys.getrecursionlimit())
print(resource.getrlimit(resource.RLIMIT_STACK))
# We may reach 1_000_000 levels of recursion
# and we have up to 512 MB of memory
with ResourceLimit(1_000_000, 512 * 1024 * 1024):
    print("during")
    print(sys.getrecursionlimit())
    print(resource.getrlimit(resource.RLIMIT_STACK))
    complex_function_that_needs_a_lot_of_ram_and_uses_recursion_a_lot()

print("after")
print(sys.getrecursionlimit())
print(resource.getrlimit(resource.RLIMIT_STACK))

The resources are increased then restored, as you can see in the output:

before
1000
(8388608, -1)
during
1000000
(536870912, -1)
after
1000
(8388608, -1)

This is a contrieved example (everything goes back to normal at the end of the program anyway), but this can be useful for chaining some problems that may require extra resources.

See a typo ? You can suggest a modification on Github.