pytest coverage with context#

This tweet from @Mariatta tipped me off to the ability to measure “contexts” when running coverage - as a way to tell which tests exercise which specific lines of code.

My sqlite-utils project uses pytest for the test suite. I decided to figure out how to get this working with pytest-cov.

After some experimentation, this is the recipe that worked for me:

1
# In the virtual environment, make sure pytest-cov is installed:
2
% pip install pytest-cov
3
# First, run pytest to calculate coverage of the `sqlite_utils` package, with context
4
% pytest --cov=sqlite_utils --cov-context=test
5
# The .coverage file is actually a SQLite database:
6
% ls -lah .coverage
7
-rw-r--r--@ 1 simon  staff   716K Mar  4 16:39 .coverage
8
# This command generates the HTML coverage report in `htmlcov/`
9
% coverage html --show-contexts
10
# Open the report in a browser`
11
% open htmlcov/index.html

Here’s what one of the pages looks like, displaying the context for some lines of code:

The code has an expandable section which reveals which tests executed each individual line.

The .coverage schema#

Since .coverage is a SQLite database, here’s the schema - generated using sqlite-utils schema .coverage:

1
CREATE TABLE coverage_schema (
2
    -- One row, to record the version of the schema in this db.
3
    version integer
4
);
5
CREATE TABLE meta (
6
    -- Key-value pairs, to record metadata about the data
7
    key text,
8
    value text,
9
    unique (key)
10
    -- Keys:
11
    --  'has_arcs' boolean      -- Is this data recording branches?
12
    --  'sys_argv' text         -- The coverage command line that recorded the data.
13
    --  'version' text          -- The version of coverage.py that made the file.
14
    --  'when' text             -- Datetime when the file was created.
15
);
16
CREATE TABLE file (
17
    -- A row per file measured.
18
    id integer primary key,
19
    path text,
20
    unique (path)
21
);
22
CREATE TABLE context (
23
    -- A row per context measured.
24
    id integer primary key,
25
    context text,
26
    unique (context)
27
);
28
CREATE TABLE line_bits (
29
    -- If recording lines, a row per context per file executed.
30
    -- All of the line numbers for that file/context are in one numbits.
31
    file_id integer,            -- foreign key to `file`.
32
    context_id integer,         -- foreign key to `context`.
33
    numbits blob,               -- see the numbits functions in coverage.numbits
34
    foreign key (file_id) references file (id),
35
    foreign key (context_id) references context (id),
36
    unique (file_id, context_id)
37
);
38
CREATE TABLE arc (
39
    -- If recording branches, a row per context per from/to line transition executed.
40
    file_id integer,            -- foreign key to `file`.
41
    context_id integer,         -- foreign key to `context`.
42
    fromno integer,             -- line number jumped from.
43
    tono integer,               -- line number jumped to.
44
    foreign key (file_id) references file (id),
45
    foreign key (context_id) references context (id),
46
    unique (file_id, context_id, fromno, tono)
47
);
48
CREATE TABLE tracer (
49
    -- A row per file indicating the tracer used for that file.
50
    file_id integer primary key,
51
    tracer text,
52
    foreign key (file_id) references file (id)
53
);