Packaging a Python CLI tool for Homebrew#

I finally figured out how to package Datasette for installation with Homebrew. My package was accepted into Homebrew core, which means you can now install it like this:

1
brew install datasette

Prior to being accepted, you needed to install it from my own Homebrew tap like this:

1
brew install simonw/datasette/datasette
2
# wait a bit...
3
datasette --version

Here’s my code that makes this work: https://github.com/simonw/homebrew-datasette

The Python for Formula Authors documentation provides useful background.

Creating a “tap”#

Homebrew taps are just naming conventions. Creating a tap is as simple as creating a GitHub repository with the homebrew- prefix. https://github.com/simonw/homebrew-datasette is the repo that gets tapped when someone runs brew tap simonw/datasette.

The repository needs a Formula/ folder. This contains your formulas, which are Ruby .rb files.

Creating the formula#

The first working version of the datasette.rb formula can be seen here: https://github.com/simonw/homebrew-datasette/blob/e6b71b1aa308d7307f75a6458681fe49f5659098/Formula/datasette.rb

The shape of the formula is this:

1
class Datasette < Formula
2
  include Language::Python::Virtualenv
3
  desc "An open source multi-tool for exploring and publishing data"
4
  homepage "https://datasette.io/"
5
  url "https://files.pythonhosted.org/packages/96/e2/abc76ee41d9895145e43323c591aa77f2b27959deb640278fc1a43f6b222/datasette-0.46.tar.gz"
6
  version "0.46"
7
  sha256 "eb5e5dcb8a0957ed1def841108576afb15a38ce61d222bf54a25d827999ad521"
8

9
  depends_on "python@3.8"
10

11
  resource "aiofiles" do
12
    url "https://files.pythonhosted.org/packages/2b/64/437053d6a4ba3b3eea1044131a25b458489320cb9609e19ac17261e4dc9b/aiofiles-0.5.0.tar.gz"
13
    sha256 "98e6bcfd1b50f97db4980e182ddd509b7cc35909e903a8fe50d8849e02d815af"
14
  end
15

16
  # ... many more resource blocks ...
17

18
  def install
19
    virtualenv_install_with_resources
20
  end
21

22
  test do
23
    system bin/"datasette", "--help"
24
  end
25
end

Every dependency needs to be listed as a resource. They all need to be available as sdist packages - I made sure all of my dependencies had an sdist on PyPI.

Then I used the homebrew-pypi-poet tool to construct the formula.

This must be installed in a fresh virtual environment. If you install it into an environment with other packages those packages will be included in the formula even if they are not used by that tool.

Create a fresh virtual environment like this:

1
cd /tmp
2
mkdir fresh
3
cd fresh
4
python -m venv venv
5
source venv/bin/activate

I’ll demonstrate installing strip-tags here since it is not yet packaged for Homebrew, unlike Datasette.

Install both strip-tags and the homebrew-pypi-poet package:

1
pip install strip-tags homebrew-pypi-poet

Next, run poet -f to create the formula:

1
poet -f strip-tags > strip-tags.rb

You can test installing the formula like this:

1
HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source --verbose --debug strip-tags.rb

If this works, you’ll be able to run strip-tags - use which strip-tags to check where it was installed.

Now add strip-tags.rb to the Formula folder in your repository, then do brew uninstall strip-tags and then brew install yourname/yourtap/strip-tags to test installing from the formula in the GitHub repository.

Implementing the test block#

https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula says:

We want tests that don’t require any user input and test the basic functionality of the application. For example foo build-foo input.foo is a good test and (despite their widespread use) foo --version and foo --help are bad tests. However, a bad test is better than no test at all.

Here’s the test block I ended up using for Datasette:

1
  test do
2
    assert_match "15", shell_output("#{bin}/datasette --get '/:memory:.csv?sql=select+3*5'")
3
    assert_match "<title>Datasette:", shell_output("#{bin}/datasette --get '/'")
4
  end

And here’s my test for sqlite-utils:

1
  test do
2
    assert_match "15", shell_output("#{bin}/sqlite-utils :memory: 'select 3 * 5'")
3
  end

And for llm:

1
  test do
2
    assert_match "llm, version", shell_output("#{bin}/llm --version")
3
  end

Iterating on this#

I found running brew install datasette, seeing if it worked, then running brew uninstall datasette, modifying the .rb file on GitHub and running datasette install datasette again worked fine during development.

If you get any errors, brew install datasette --debug shows more information and drops you into an interactive debugging session when an error occurs.

Submitting to homebrew-core#

If your package gets accepted into homebrew-core users will be able to install it just by running brew install packagename.

More importantly: Homebrew maintain “bottle” versions of all of those core packages. These are pre-compiled bundles of assets (a separate .tar.gz for each recent macOS operating system) which install MUCH faster than regular Homebrew, which has to compile everything.

The Homebrew CONTRIBUTING document tells you how to do this. For Python packages the import things to remember are:

Add a license, e.g. `license “Apache 2.0” - example.
Run brew audit --new-formula datasette and fix any warnings (see here).
Submit a PR with the new formula and a title of e.g. datasette 0.47.1 (new formula) - here’s mine for Datasette and for sqlite-utils.