Using cog to update —help in a Markdown README file#

My csvs-to-sqlite README includes a section that shows the output of the csvs-to-sqlite --help command (relevant issue).

I had been manually copying this in, but I decided to try using cog to automate the process.

Here’s what I came up with:

1
<!-- [[[cog
2
import cog
3
from csvs_to_sqlite import cli
4
from click.testing import CliRunner
5
runner = CliRunner()
6
result = runner.invoke(cli.cli, ["--help"])
7
help = result.output.replace("Usage: cli", "Usage: csvs-to-sqlite")
8
cog.out(
9
    "```\n{}\n```".format(help)
10
)
11
]]] -->
12
```
13
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
14
...
15
```
16
<!-- [[[end]]] -->

Then to update the README file, run this:

1
cog -r README.md

The -r option causes it to modify that file in place.

Cog works by scanning for a [[[cog ... ]]] section, executing the code there, capturing the cog.out() output and using that to replace everything from the end of the code block up to the line containing the [[[end]]] marker.

It’s designed to interact well with comments - in this case HTML comments - such that the cog generation code can be hidden.

Testing with cog —check#

A version of Cog released after I first wrote this TIL added a new --check option, so you can run a test in CI to check if the file needs to be updated using:

1
cog --check README.md

Writing a test (before cog —check)#

Any time I generate content like this in a repo I like to include a test that will fail if I forget to update the content.

cog clearly isn’t designed to be used as an independent library, but I came up with the following pattern pytest test which works well, in my tests/test_csvs_to_sqlite.py module:

1
from cogapp import Cog
2
import sys
3
from io import StringIO
4
import pathlib
5

6

7
def test_if_cog_needs_to_be_run():
8
    _stdout = sys.stdout
9
    sys.stdout = StringIO()
10
    readme = pathlib.Path(__file__).parent.parent / "README.md"
11
    result = Cog().main(["cog", str(readme)])
12
    output = sys.stdout.getvalue()
13
    sys.stdout = _stdout
14
    assert (
15
        output == readme.read_text()
16
    ), "Run 'cog -r README.md' to update help in README"

The key line here is this one:

1
result = Cog().main(["cog", str(readme)])

In cog’s implementation, that code is called like this:

1
Cog().main(sys.argv)

Here I’m faking the command-line arguments to pass in just the path to my README.md file.

Cog then writes the generated output to stdout - which I capture with that sys.stdout trick.

Finally, I compare the generated output to the current file content and fail the test with a reminder to run cog -r if they do not match.

Cog for reStructuredText#

Here’s an example of cog in a .rst file:

1
.. [[[cog
2
    import tabulate
3
    cog.out("\n" + "\n".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + "\n\n")
4
.. ]]]
5

6
- ``fancy_grid``
7
- ``fancy_outline``
8
- ``github``
9

10
.. [[[end]]]

The trailing and leading newlines are important to avoid a warning about “Explicit markup ends without a blank line; unexpected unindent”.

CLI reference pages#

I added these to both Datasette and sqlite-utils: full pages that list the help for every command provided by those tools.

Datasette CLI reference - underlying source code
sqlite-utils CLI reference - underlying source code