Running tests against multiple versions of a Python dependency in GitHub Actions#

My datasette-export-notebook plugin worked fine in the stable release of Datasette, currently version 0.64.3, but failed in the Datasette 1.0 alphas. Here’s the issue describing the problem.

Here’s the pattern I figured out for running the tests in GitHub Actions against both Datasette versions. This is my full test.yml from that repository:

1
name: Test
2

3
on: [push]
4

5
jobs:
6
  test:
7
    runs-on: ubuntu-latest
8
    strategy:
9
      matrix:
10
        python-version: ["3.8", "3.9", "3.10", "3.11"]
11
        datasette-version: ["<=1.0a0", ">=1.0a0"]
12
    steps:
13
    - uses: actions/checkout@v3
14
    - name: Set up Python ${{ matrix.python-version }}
15
      uses: actions/setup-python@v4
16
      with:
17
        python-version: ${{ matrix.python-version }}
18
        cache: pip
19
        cache-dependency-path: setup.py
20
    - name: Install dependencies
21
      run: |
22
        pip install -e '.[test]'
23
        pip install 'datasette${{ matrix.datasette-version }}'
24
    - name: Run tests
25
      run: |
26
        pytest

The trick here is to set up a matrix for datasette-version (to accompany my existing python-version one) defining these two installation specifiers:

1
datasette-version: ["<=1.0a0", ">=1.0a0"]

Then later I use those to install the specified version of Datasette like this:

1
pip install 'datasette${{ matrix.datasette-version }}'

The single quotes there are important - without them my shell got confused by the <= and >= symbols.

The end result of this is that tests run against the highest Datasette release in the 0.x series, and also against the highest release in the 1.x series, including alphas if no 1.x stable release is out yet.

Adding extra version information to the pytest report#

When using this pattern, it can be useful to include the Datasette version in the output of the pytest command.

Here’s an easy way to do that: add the following to tests/conftest.py:

1
import datasette
2

3

4
def pytest_report_header():
5
    return "Datasette: {}".format(datasette.__version__)

Running pytest will now output the following:

1
============================ test session starts ============================
2
platform darwin -- Python 3.9.17, pytest-7.4.2, pluggy-1.3.0
3
Datasette: 1.0a6
4
rootdir: /Users/...