Writing tests with Copilot#

I needed to write a relatively repetitive collection of tests, for a number of different possible error states.

The code I was testing looks like this:

1
columns = data.get("columns")
2
rows = data.get("rows")
3
row = data.get("row")
4
if not columns and not rows and not row:
5
    return _error(["columns, rows or row is required"])
6

7
if rows and row:
8
    return _error(["Cannot specify both rows and row"])
9

10
if columns:
11
    if rows or row:
12
        return _error(["Cannot specify columns with rows or row"])
13
    if not isinstance(columns, list):
14
        return _error(["columns must be a list"])
15
    for column in columns:
16
        if not isinstance(column, dict):
17
            return _error(["columns must be a list of objects"])
18
        if not column.get("name") or not isinstance(column.get("name"), str):
19
            return _error(["Column name is required"])
20
        if not column.get("type"):
21
            column["type"] = "text"
22
        if column["type"] not in self._supported_column_types:
23
            return _error(
24
                ["Unsupported column type: {}".format(column["type"])]
25
            )
26
    # No duplicate column names
27
    dupes = {c["name"] for c in columns if columns.count(c) > 1}
28
    if dupes:
29
        return _error(["Duplicate column name: {}".format(", ".join(dupes))])

I wanted to write tests for each of the error cases. I’d already constructed the start of a parameterized pytest test for these.

I got Copilot/GPT-3 to write most of the tests for me.

First I used VS Code to select all of the _error(...) lines. I pasted those into a new document and turned them into a sequence of comments, like this:

1
# Error: columns must be a list
2
# Error: columns must be a list of objects
3
# Error: Column name is required
4
# Error: Unsupported column type
5
# Error: Duplicate column name
6
# Error: rows must be a list
7
# Error: rows must be a list of objects
8
# Error: pk must be a string

I pasted those comments into my test file inside the existing list of parameterized tests, then wrote each test by adding a newline beneath a comment and hitting tab until Copilot had written the test for me.

It correctly guessed both the error assertion and the desired invalid input for each one!

Here’s an animated screenshot:

In this animation I start typing below the comment that says rows must be a list of objects - Copilot correctly deduces that I neeed an example with a rows item that is a list, and that the expected status code is 400, and that the returned error should match the text in the comment.

The finished tests are here.