Writing a Datasette CLI plugin that mostly duplicates an existing command#

My new datasette-gunicorn plugin adds a new command to Datasette - datasette gunicorn - which mostly replicates the existing datasette serve command but with a few differences.

I learned some useful tricks for modifying and extending existing Click commands building this plugin.

Here’s the relevant section of code, with some extra comments (the full code is here):

1
import click
2
from datasette import hookimpl
3

4
# These options do not work with 'datasette gunicorn':
5
invalid_options = {
6
    "get",
7
    "root",
8
    "open_browser",
9
    "uds",
10
    "reload",
11
    "pdb",
12
    "ssl_keyfile",
13
    "ssl_certfile",
14
}
15

16

17
def serve_with_gunicorn(**kwargs):
18
    # Avoid a circular import error running the tests:
19
    from datasette import cli
20

21
    workers = kwargs.pop("workers")
22
    port = kwargs["port"]
23
    host = kwargs["host"]
24
    # Need to add back default kwargs for everything in invalid_options:
25
    kwargs.update({invalid_option: None for invalid_option in invalid_options})
26
    kwargs["return_instance"] = True
27
    ds = cli.serve.callback(**kwargs)
28
    # ds is now a configured Datasette instance
29
    asgi = StandaloneApplication(
30
        app=ds.app(),
31
        options={
32
            "bind": "{}:{}".format(host, port),
33
            "workers": workers,
34
        },
35
    )
36
    asgi.run()
37

38

39
@hookimpl
40
def register_commands(cli):
41
    # Get a reference to the existing "datasette serve" command
42
    serve_command = cli.commands["serve"]
43
    # Create a new list of params, excluding any in invalid_options
44
    params = [
45
        param for param in serve_command.params if param.name not in invalid_options
46
    ]
47
    # This is the longer way of constructing a new Click option, as an alternative
48
    # to using the @click.option() decorator
49
    params.append(
50
        click.Option(
51
            ["-w", "--workers"],
52
            type=int,
53
            default=1,
54
            help="Number of Gunicorn workers",
55
            # Causes [default: 1] to show in the option help
56
            show_default=True,
57
        )
58
    )
59
    gunicorn_command = click.Command(
60
        name="gunicorn",
61
        params=params,
62
        callback=serve_with_gunicorn,
63
        short_help="Serve Datasette using Gunicorn",
64
        help="Start a Gunicorn server running to serve Datasette",
65
    )
66
    # cli is the Click command group passed to this plugin hook by
67
    # Datasette - this is how we add the "datasette gunicorn" command:
68
    cli.add_command(gunicorn_command, name="gunicorn")

Here’s the documentation for the register_commands() plugin hook. It is passed cli which is a Click command group.

cli.add_command(...) can then be used to register additional commands - in this case the datasette gunicorn one.

I want that command to take almost the same options as the existing datasette serve command - which is defined here in the Datasette codebase.

So… I start by creating a copy of those options. But there are a few options which don’t make sense for my new command (see this issue). So I filtered those out with a list comprehension:

1
params = [
2
    param for param in serve_command.params if param.name not in invalid_options
3
]

I did need one extra option: a -w/--workers integer specifying the number of workers that should be started by Gunicorn.

Here’s the relevant Click documentation. I defined it like this:

1
params.append(
2
    click.Option(
3
        ["-w", "--workers"],
4
        type=int,
5
        default=1,
6
        help="Number of Gunicorn workers",
7
        # Causes [default: 1] to show in the option help
8
        show_default=True,
9
    )
10
)

I defined the new gunicorn command like this:

1
gunicorn_command = click.Command(
2
    name="gunicorn",
3
    params=params,
4
    callback=serve_with_gunicorn,
5
    short_help="Serve Datasette using Gunicorn",
6
    help="Start a Gunicorn server running to serve Datasette",
7
)
8
cli.add_command(gunicorn_command, name="gunicorn")

The short_help is shown in the list of commands displayed by datasette --help.

The help is shown at the top of the list of options when you run datasette gunicorn --help.

The most important argument here is callback= - this is the function which will be executed when the user types datasette gunicorn ....

Here’s a partial implementation of that function:

1
def serve_with_gunicorn(**kwargs):
2
    # Avoid a circular import error running the tests:
3
    from datasette import cli
4

5
    workers = kwargs.pop("workers")
6
    port = kwargs["port"]
7
    host = kwargs["host"]
8
    # Need to add back default kwargs for everything in invalid_options:
9
    kwargs.update({invalid_option: None for invalid_option in invalid_options})
10
    kwargs["return_instance"] = True
11
    ds = cli.serve.callback(**kwargs)
12
    # ds is now a configured Datasette instance
13
    asgi = StandaloneApplication(
14
        app=ds.app(),
15
        options={
16
            "bind": "{}:{}".format(host, port),
17
            "workers": workers,
18
        },
19
    )
20
    asgi.run()

The **kwargs passed to that function are the options and argumenst that have been extracted from the command line by Click.

In this case, I know I’m going to be calling the existing serve function from Datasette. cli.serve is the Click decorated version, but cli.serve.callback() is the original function I defined in my own Datasette source code (linked above).

That function in Datasette takes a list of keyword arguments, which I need to pass through.

The kwargs passed to serve_with_gunicorn() are not quite right - remember, I removed some options earlier, and I also added a workers option that serve() doesn’t know how to handle.

So I pop workers off the dictionary, and I add "name": None keys for the invalid_options that I previously filtered out.

One last trick: my serve() function here in Datasette has an extra return_instance keyword argument, which can be used to shortcut that function and return the configured Datasette instance instead of starting the server.

I originally built this to help with unit tests, but this is also exactly what I need for this particular plugin! I set that to true to get back a configured Datasette object instance, which I can then use to serve the application using Gunicorn.