CLI tools hidden in the Python standard library#

Seth Michael Larson pointed out that the Python gzip module can be used as a CLI tool like this:

1
python -m gzip --decompress pypi.db.gz

This is a neat Python feature: modules with a if __name__ == "__main__": block that are available on Python’s standard import path can be executed from the terminal using python -m name_of_module.

Seth pointed out this is useful if you are on Windows and don’t have the gzip utility installed.

This made me wonder: what other little tools are lurking in the Python standard library, available on any computer with a working Python installation?

Finding them with ripgrep#

I decided to take a sniff around the standard library and see what I can find.

Jim Crist-Harif pointed me to python -m site, which outputs useful information about your installation:

1
python3.11 -m site

1
sys.path = [
2
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11',
3
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python311.zip',
4
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
5
    '/opt/homebrew/lib/python3.11/site-packages',
6
    '/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
7
]
8
USER_BASE: '/Users/simon/Library/Python/3.11' (doesn't exist)
9
USER_SITE: '/Users/simon/Library/Python/3.11/lib/python/site-packages' (doesn't exist)
10
ENABLE_USER_SITE: True

This showed me that the standard library itself for my Homebrew installation of Python 3.11 is in /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11.

So I switched there and used ripgrep to find likely packages:

1
cd /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11

Then ran rg:

1
rg 'if __name__ =' -l | grep -v 'test/' \
2
  | grep -v 'tests/' | grep -v idlelib | grep -v turtledemo

The -l option causes ripgrep to list matching files without showing the context of the match.

I built up those grep -v exclusions over a few iterations - idlelib/ and turtledemo/ have a bunch of matches that I wasn’t interested in.

Update: Here’s an alternative way of expressing that search:
Terminal window
1
rg 'if __name__ =' -l | grep -v -e 'test/' -e 'tests/' -e idlelib -e turtledemo
grep -v still means “everything that doesn’t match this” - but then the multiple -e '...' patterns are used to construct a “this pattern or this pattern or this pattern” filter. This saves on having to pipe through grep -v multiple times. Thanks for the tip, dfc.

Here’s the result:

1
tabnanny.py
2
pyclbr.py
3
netrc.py
4
heapq.py
5
fileinput.py
6
site.py
7
telnetlib.py
8
smtplib.py
9
timeit.py
10
__hello__.py
11
aifc.py
12
json/tool.py
13
asyncio/__main__.py
14
runpy.py
15
mailcap.py
16
tokenize.py
17
smtpd.py
18
sysconfig.py
19
tarfile.py
20
lib2to3/pgen2/tokenize.py
21
lib2to3/pgen2/driver.py
22
lib2to3/pgen2/literals.py
23
xmlrpc/server.py
24
xmlrpc/client.py
25
getopt.py
26
dbm/__init__.py
27
doctest.py
28
pickle.py
29
imaplib.py
30
compileall.py
31
shlex.py
32
ast.py
33
venv/__init__.py
34
py_compile.py
35
ensurepip/__main__.py
36
ensurepip/_uninstall.py
37
http/server.py
38
pickletools.py
39
poplib.py
40
quopri.py
41
calendar.py
42
pprint.py
43
symtable.py
44
pstats.py
45
inspect.py
46
pdb.py
47
platform.py
48
wsgiref/simple_server.py
49
random.py
50
ftplib.py
51
mimetypes.py
52
turtle.py
53
tkinter/dialog.py
54
xml/sax/expatreader.py
55
tkinter/colorchooser.py
56
tkinter/dnd.py
57
tkinter/filedialog.py
58
tkinter/messagebox.py
59
tkinter/simpledialog.py
60
tkinter/font.py
61
tkinter/scrolledtext.py
62
xml/sax/xmlreader.py
63
tkinter/__init__.py
64
code.py
65
difflib.py
66
pydoc.py
67
uu.py
68
imghdr.py
69
filecmp.py
70
profile.py
71
cgi.py
72
codecs.py
73
modulefinder.py
74
__phello__/__init__.py
75
__phello__/spam.py
76
multiprocessing/spawn.py
77
textwrap.py
78
base64.py
79
curses/textpad.py
80
curses/has_key.py
81
zipapp.py
82
cProfile.py
83
dis.py
84
webbrowser.py
85
nntplib.py
86
sndhdr.py
87
gzip.py
88
ctypes/util.py
89
zipfile.py
90
encodings/rot_13.py
91
distutils/fancy_getopt.py

That’s a lot of neat little tools!

I haven’t explored my way through all of them yet, but running python -m module_name usually outputs something useful, and adding -h frequently provides help.

A few highlights#

Here are a few of the commands I’ve figured out so far.

http.server#

To run a localhost webserver on port 8000, serving the content of the current directory:

1
python -m http.server

This takes an optional port. To change port, do this:

1
python -m http.server 8001

Pass -h for more options.

base64#

1
python3.11 -m base64 -h

1
usage: .../base64.py [-h|-d|-e|-u|-t] [file|-]
2
        -h: print this help message and exit
3
        -d, -u: decode
4
        -e: encode (default)
5
        -t: encode and decode string 'Aladdin:open sesame'

asyncio#

This provides a Python console with top-level await:

1
python -m asyncio

1
asyncio REPL 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
2
Use "await" directly instead of "asyncio.run()".
3
Type "help", "copyright", "credits" or "license" for more information.
4
>>> import asyncio
5
>>> import httpx
6
>>> async with httpx.AsyncClient() as client:
7
...     r = await client.get('https://www.example.com/')
8
...
9
>>> r.text[:50]
10
'<!DOCTYPE html>\n<html lang="en">\n<head>\n  <meta ht'

tokenize#

This is fun - it’s a debug mode for the Python tokenizer. You can run it directly against a file:

1
python -m tokenize cgi.py | head -n 10

1
0,0-0,0:            ENCODING       'utf-8'
2
1,0-1,24:           COMMENT        '#! /usr/local/bin/python'
3
1,24-1,25:          NL             '\n'
4
2,0-2,1:            NL             '\n'
5
3,0-3,66:           COMMENT        '# NOTE: the above "/usr/local/bin/python" is NOT a mistake.  It is'
6
3,66-3,67:          NL             '\n'
7
4,0-4,59:           COMMENT        '# intentionally NOT "/usr/bin/env python".  On many systems'
8
4,59-4,60:          NL             '\n'
9
5,0-5,65:           COMMENT        '# (e.g. Solaris), /usr/local/bin is not in $PATH as passed to CGI'
10
5,65-5,66:          NL             '\n'

ast#

Even more fun than tokenize - a debug mode for the Python AST module!

1
python -m ast cgi.py | head -n 10

1
Module(
2
   body=[
3
      Expr(
4
         value=Constant(value='Support module for CGI (Common Gateway Interface) scripts.\n\nThis module defines a number of utilities for use by CGI scripts\nwritten in Python.\n\nThe global variable maxlen can be set to an integer indicating the maximum size\nof a POST request. POST requests larger than this size will result in a\nValueError being raised during parsing. The default value of this variable is 0,\nmeaning the request size is unlimited.\n')),
5
      Assign(
6
         targets=[
7
            Name(id='__version__', ctx=Store())],
8
         value=Constant(value='2.6')),
9
      ImportFrom(
10
         module='io',

I used this module to build my symbex tool - this debug mode would have helped quite a bit if I’d found out about it earlier.

json.tool#

Pretty-print JSON:

1
echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool

1
{
2
    "foo": "bar",
3
    "baz": [
4
        1,
5
        2,
6
        3
7
    ]
8
}

Tip from Joe Kerhin:

json.tool will also exit with nonzero status if your json document is invalid.

random#

I thought this might provide a utility for generating random numbers, but sadly it’s just a benchmarking suite with no additional command-line options:

1
python -m random

1
0.000 sec, 2000 times random
2
avg 0.49105, stddev 0.290864, min 0.00216092, max 0.999473
3

4
0.001 sec, 2000 times normalvariate
5
avg -0.00286956, stddev 0.996872, min -3.42333, max 4.2012
6

7
0.001 sec, 2000 times lognormvariate
8
avg 1.64228, stddev 2.13138, min 0.0386213, max 34.0379
9

10
0.001 sec, 2000 times vonmisesvariate
11
avg 3.18754, stddev 2.27556, min 0.00336177, max 6.28306
12
...

Update: This is fixed in Python 3.13.

nntplib#

“nntplib built-in demo - display the latest articles in a newsgroup”

It defaults to gmane.comp.python.general:

1
python -m nntplib

1
Group gmane.comp.python.general has 757237 articles, range 23546 to 848591
2
 848582 MRAB via Python-...  Re: Trouble with defaults and timeout ...  (40)
3
 848583 Dave Ohlsson via...  unable to run the basic Embedded Pytho...  (179)
4
 848584 Piergiorgio Sart...  Re: Trouble with defaults and timeout ...  (43)
5
 848585 Dan Kolis via Py...  TKinter in Python - advanced notions - ok  (52)
6
 848586 Fulian Wang via ...  Re: unable to run the basic Embedded P...  (190)
7
 848587 Fulian Wang via ...  Re: unable to run the basic Embedded P...  (220)
8
 848588 Christian Gollwi...  Re: unable to run the basic Embedded P...  (26)
9
 848589 small marcc via ...  my excel file is not updated to add ne...  (42)
10
 848590 Thomas Passin vi...  Re: my excel file is not updated to ad...  (47)
11
 848591 dn via Python-list   Re: my excel file is not updated to ad...  (207)

Those look like real, recent messages - I matched them with this mirror.

When I tried passing other newsgroup names with python -m nntplib -g alt.humor.puns I got an error though:

NNTPTemporaryError: 411 No such group alt.humor.puns

calendar#

Show a calendar for the current year:

1
python -m calendar

1
                                  2023
2

3
      January                   February                   March
4
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
5
                   1             1  2  3  4  5             1  2  3  4  5
6
 2  3  4  5  6  7  8       6  7  8  9 10 11 12       6  7  8  9 10 11 12
7
 9 10 11 12 13 14 15      13 14 15 16 17 18 19      13 14 15 16 17 18 19
8
16 17 18 19 20 21 22      20 21 22 23 24 25 26      20 21 22 23 24 25 26
9
23 24 25 26 27 28 29      27 28                     27 28 29 30 31
10
30 31
11

12
       April                      May                       June
13
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
14
                1  2       1  2  3  4  5  6  7                1  2  3  4
15
 3  4  5  6  7  8  9       8  9 10 11 12 13 14       5  6  7  8  9 10 11
16
10 11 12 13 14 15 16      15 16 17 18 19 20 21      12 13 14 15 16 17 18
17
17 18 19 20 21 22 23      22 23 24 25 26 27 28      19 20 21 22 23 24 25
18
24 25 26 27 28 29 30      29 30 31                  26 27 28 29 30
19

20
        July                     August                  September
21
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
22
                1  2          1  2  3  4  5  6                   1  2  3
23
 3  4  5  6  7  8  9       7  8  9 10 11 12 13       4  5  6  7  8  9 10
24
10 11 12 13 14 15 16      14 15 16 17 18 19 20      11 12 13 14 15 16 17
25
17 18 19 20 21 22 23      21 22 23 24 25 26 27      18 19 20 21 22 23 24
26
24 25 26 27 28 29 30      28 29 30 31               25 26 27 28 29 30
27
31
28

29
      October                   November                  December
30
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
31
                   1             1  2  3  4  5                   1  2  3
32
 2  3  4  5  6  7  8       6  7  8  9 10 11 12       4  5  6  7  8  9 10
33
 9 10 11 12 13 14 15      13 14 15 16 17 18 19      11 12 13 14 15 16 17
34
16 17 18 19 20 21 22      20 21 22 23 24 25 26      18 19 20 21 22 23 24
35
23 24 25 26 27 28 29      27 28 29 30               25 26 27 28 29 30 31
36
30 31

This one has a bunch more options (visible with -h). python -m calendar -t html produces the calendar in HTML, for example.

Loads more#

There are plenty more in there - these are just the ones I’ve explored so far.

Trey Hunner collected a more recent set of these in June 2024 in Python’s many command-line utilities.