Streaming indented output of a JSON array#

I wanted to produce the following output from a command-line tool:

1
[
2
  {
3
    "id": 1,
4
    "name": "Simon"
5
  },
6
  {
7
    "id": 2,
8
    "name": "Cleo"
9
  },
10
  {
11
    "id": 3,
12
    "name": "Azi"
13
  }
14
]

But I wanted to do it from a streaming iterator - without first buffering the entire output in an in-memory Python list and calling json.dumps() on it.

Here’s the pattern I came up with:

1
import itertools, json, textwrap
2

3
def stream_indented_json(iterator, indent=2):
4
    # We have to iterate two-at-a-time so we can know if we
5
    # should output a trailing comma or if we have reached
6
    # the last item.
7
    current_iter, next_iter = itertools.tee(iterator, 2)
8
    next(next_iter, None)
9
    first = True
10
    for item, next_item in itertools.zip_longest(current_iter, next_iter):
11
        is_last = next_item is None
12
        data = item
13
        line = "{first}{serialized}{separator}{last}".format(
14
            first="[\n" if first else "",
15
            serialized=textwrap.indent(
16
                json.dumps(data, indent=indent, default=repr), " " * indent
17
            ),
18
            separator="," if not is_last else "",
19
            last="\n]" if is_last else "",
20
        )
21
        yield line
22
        first = False
23
    if first:
24
        # We didn't output anything, so yield the empty list
25
        yield "[]"

Example usage:

1
for line in stream_indented_json(item_iterator):
2
    click.echo(line)

I built this for s3-credentials issue #48.