Running Python code in a subprocess with a time limit#

I figured out how to run a subprocess with a time limit for datasette-ripgrep, using the asyncio.create_subprocess_exec() method. The pattern looks like this:

1
import asyncio
2

3
proc = await asyncio.create_subprocess_exec(
4
    "rg",
5
    "-e",
6
    ".*",
7
    stdout=asyncio.subprocess.PIPE,
8
    stdin=asyncio.subprocess.PIPE,
9
)
10

11
try:
12
    stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=0.1)
13
    print(stdout)
14
except asyncio.exceptions.TimeoutError:
15
    print("Command timed out")
16

17
# If it timed out we should terminate the process
18
try:
19
    proc.kill()
20
except OSError:
21
    # Ignore 'no such process' error
22
    pass

For datasette-seaborn I wanted to render a chart using the Python seaborn library with a time limit of five seconds for the render.

I realized I could do this by launching Python itself as the subprocess executable (using sys.executable) and sending Python code to stdin to be executed in a process, using the same time limit mechanism.

It seems to work! Here’s the pattern wrapped up in a function:

1
import asyncio, sys
2

3

4
async def execute_python_with_time_limit(code, time_limit):
5
    proc = await asyncio.create_subprocess_exec(
6
        sys.executable,
7
        "-",
8
        stdout=asyncio.subprocess.PIPE,
9
        stdin=asyncio.subprocess.PIPE,
10
    )
11
    try:
12
        stdout, stderr = await asyncio.wait_for(
13
            proc.communicate(code.encode("utf-8")), timeout=time_limit
14
        )
15
    except asyncio.exceptions.TimeoutError:
16
        try:
17
            proc.kill()
18
        except OSError:
19
            # Ignore 'no such process' error
20
            pass
21
        raise
22
    return stdout, stderr

Example of using it (pasting into the shell you get from python3 -m asyncio in Python 3.8+):

1
>>> await execute_python_with_time_limit('print("hello")', 1)
2
(b'hello\n', None)
3
>>> await execute_python_with_time_limit('import time\ntime.sleep(1)', 0.7)
4
Traceback (most recent call last):
5
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 439, in result
6
    return self.__get_result()
7
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
8
    raise self._exception
9
  File "<console>", line 1, in <module>
10
  File "<console>", line 9, in execute_python_with_time_limit
11
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
12
    raise exceptions.TimeoutError()
13
asyncio.exceptions.TimeoutError

It returns the stdout output of the code, so to use this you’ll need to figure out some kind of serialization format for the data that is returned from the subprocess. JSON or pickle should work fine.

sys.executable is the path to the current Python interpreter. This ensures that any imports will take place in the correct Python virtual environment. Passing - as the first argument causes Python to execute code from standard input, which is then provided using the .communicate() method.

Is this a good idea? I think so, but I’d love to hear from you if there’s a simpler, cleaner way to do this.

Newsletter

Running Python code in a subprocess with a time limit#