Using pytest and Playwright to test a JavaScript web application#

I decided to add automated tests to my Datasette Lite project. Datasette Lite bundles my Datasette Python web application as a client-side application running inside WebAssembly using Pyodide.

I wrote the tests using playwright-pytest, which lets you write tests in Python using Microsoft’s Playwright browser automation library.

Installing playwright-pytest#

Two steps:

1
pip install playwright-pytest

Then a second step to install the browsers using by Playwright itself:

1
playwright install

I had those browsers installed already, but I still needed to run that command since the updated version of playwright-pytest needed more recent versions.

(I had limited internet while doing this, and discovered that you can trick Playwright into using an older browser version by renaming a folder in ~/Library/Caches/ms-playwright to the one that shows up in the error message that says that the browsers cannot be found.)

An initial test#

The first test I wrote looked like this, saved in test_app.py:

1
from playwright.sync_api import Page, expect
2

3
def test_initial_load(page: Page):
4
    page.goto("https://lite.datasette.io/")
5
    loading = page.locator("#loading-indicator")
6
    expect(loading).to_have_css("display", "block")
7
    # Give it up to 60s to finish loading
8
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)
9

10
    # Should load faster the second time thanks to cache
11
    page.goto("https://lite.datasette.io/")
12
    expect(loading).to_have_css("display", "none", timeout=20 * 1000)

Then run the test by running this in the same directory as that file:

1
pytest

playwright-pytest provides the page fixture - annotating it with : Page is optional but if you do that then VS Code knows what it is and can provide autocomplete in the editor.

page.goto() causes the browser to navigate to that URL.

page.locator("#loading-indicator") returns a wrapper “locator” object for the specified CSS selector.

This line is interesting:

1
expect(loading).to_have_css("display", "block")

The expect() helper function encapsulates the concept of polling the page to wait for something to become true within a time limit. This is the key technique for avoiding “flaky” tests when working with Playwright.

The assertions are listed here.

You don’t actually need to use expect() though - that’s useful if you don’t know how long it will take for the page to load, but if you know the page is already loaded you can write assertions like this instead:

1
assert [
2
    el.inner_text()
3
    for el in page.query_selector_all("h2")
4
] == ["fixtures", "content"]

pytest options#

The playwright-pytest package adds a bunch of new options to pytest. The most useful is --headed:

1
pytest --headed

This runs the tests in “headed” mode - which means a visible browser window pops up during the tests so you can see what is happening.

--browser firefox runs them using Firefox instead of Chromium.

--tracing on is really interesting: it generates a trace ZIP file which you can then open using https://trace.playwright.dev/ to explore a detailed trace of the test as it executed.

--video on records a video (as a .webm file) of the test. I’ve not tried it yet, but --video retain-on-failure only keeps that video if the test fails.

Here’s documentation on all of the options.

Running a localhost static server during the tests#

I wanted to run the tests against the most recent version of my code, which consists of an index.html file and a webworker.js file. Because these use web workers they need to be run from an actual localhost web server, so I needed to start one at the beginning of the tests and shut it down at the end.

I wrote about my solution for this in another TIL: Start a server in a subprocess during a pytest session.

My test suite so far#

Here’s where I’ve got to so far:

1
from playwright.sync_api import Browser, Page, expect
2
from subprocess import Popen, PIPE
3
import pathlib
4
import pytest
5
import time
6
from http.client import HTTPConnection
7

8
root = pathlib.Path(__file__).parent.parent.absolute()
9

10

11
@pytest.fixture(scope="module")
12
def static_server():
13
    process = Popen(
14
        ["python", "-m", "http.server", "8123", "--directory", root], stdout=PIPE
15
    )
16
    retries = 5
17
    while retries > 0:
18
        conn = HTTPConnection("localhost:8123")
19
        try:
20
            conn.request("HEAD", "/")
21
            response = conn.getresponse()
22
            if response is not None:
23
                yield process
24
                break
25
        except ConnectionRefusedError:
26
            time.sleep(1)
27
            retries -= 1
28

29
    if not retries:
30
        raise RuntimeError("Failed to start http server")
31
    else:
32
        process.terminate()
33
        process.wait()
34

35

36
@pytest.fixture(scope="module")
37
def dslite(static_server, browser: Browser) -> Page:
38
    page = browser.new_page()
39
    page.goto("http://localhost:8123/")
40
    loading = page.locator("#loading-indicator")
41
    expect(loading).to_have_css("display", "block")
42
    # Give it up to 60s to finish loading
43
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)
44
    return page
45

46

47
def test_initial_load(dslite: Page):
48
    expect(dslite.locator("#loading-indicator")).to_have_css("display", "none")
49

50

51
def test_has_two_databases(dslite: Page):
52
    assert [el.inner_text() for el in dslite.query_selector_all("h2")] == [
53
        "fixtures",
54
        "content",
55
    ]
56

57

58
def test_navigate_to_database(dslite: Page):
59
    h2 = dslite.query_selector("h2")
60
    assert h2.inner_text() == "fixtures"
61
    h2.query_selector("a").click()
62
    expect(dslite).to_have_title("fixtures")
63
    dslite.query_selector("textarea#sql-editor").fill(
64
        "SELECT * FROM no_primary_key limit 1"
65
    )
66
    dslite.query_selector("input[type=submit]").click()
67
    expect(dslite).to_have_title("fixtures: SELECT * FROM no_primary_key limit 1")
68
    table = dslite.query_selector("table.rows-and-columns")
69
    table_html = "".join(table.inner_html().split())
70
    assert table_html == (
71
        '<thead><tr><thclass="col-content"scope="col">content</th>'
72
        '<thclass="col-a"scope="col">a</th><thclass="col-b"scope="col">b</th>'
73
        '<thclass="col-c"scope="col">c</th></tr></thead><tbody><tr>'
74
        '<tdclass="col-content">1</td><tdclass="col-a">a1</td>'
75
        '<tdclass="col-b">b1</td><tdclass="col-c">c1</td></tr></tbody>'
76
    )

Running it in GitHub Actions#

Here’s the GitHub Actions workflow I’m using to run the tests:

1
name: Test
2

3
on:
4
  push:
5
  pull_request:
6

7
permissions:
8
  contents: read
9

10
jobs:
11
  test:
12
    runs-on: ubuntu-latest
13
    steps:
14
    - uses: actions/checkout@v3
15
    - name: Set up Python 3.10
16
      uses: actions/setup-python@v3
17
      with:
18
        python-version: "3.10"
19
        cache: 'pip'
20
        cache-dependency-path: '**/dev-requirements.txt'
21
    - name: Cache Playwright browsers
22
      uses: actions/cache@v3
23
      with:
24
        path: ~/.cache/ms-playwright/
25
        key: ${{ runner.os }}-browsers
26
    - name: Install dependencies
27
      run: |
28
        pip install -r dev-requirements.txt
29
        playwright install
30
    - name: Run test
31
      run: |
32
        pytest

dev-requirements.txt contains this:

1
pytest-playwright==0.3.0
2
playwright==1.24.0