Processing a stream of chunks of JSON with ijson#

A follow-up to Using OpenAI functions and their Python library for data extraction and Using the ChatGPT streaming API from Python. If I have a stream of chunks of a larger JSON document, how can I output full individual JSON objects as soon as they are available?

My completed JSON will look like this:

1
{
2
  "items": [
3
    {
4
      "title": "Classic and Custom Car Show",
5
      "date": "Sat, Sep 2",
6
      "venue_name": "1700 W Hillsdale Blvd • San Mateo, CA",
7
      "start_time": "9:00 AM"
8
    },
9
    {
10
      "title": "【衛蘭、王君馨、衛詩、鍾舒漫、吉中鳴牧師】MMO AINEO LIVE 愛你喲！佈道會（三藩市站）",
11
      "date": "Sat, Aug 26",
12
      "venue_name": "San Mateo Performing Arts Center • San Mateo, CA",
13
      "start_time": "7:30 PM"
14
    },
15
    {
16
      "title": "San Francisco Small Business Expo 2023",
17
      "date": "Fri, Aug 18",
18
      "venue_name": "Hyatt Regency San Francisco Airport • Burlingame, CA",
19
      "start_time": "10:00 AM"
20
    },
21
    {
22
      "title": "Stanford Genetics Conference on Structural Variants and DNA Repeats",
23
      "date": "Thu, Sep 7",
24
      "venue_name": "Stanford Center for Academic Medicine • Palo Alto, CA",
25
      "start_time": "8:00 AM"
26
    },
27
    {
28
      "title": "DJ QUIK / ROJAI / Z-MAN / DJ SAURUS",
29
      "date": "Fri, Aug 18",
30
      "venue_name": "The Longboard Margarita Bar • Pacifica, CALIFORNIA",
31
      "start_time": "9:00 PM"
32
    },
33
    {
34
      "title": "DUE SOUTH W/ CHERRY GLAZERR, MOMMA (DUO), KING ISIS (FREE!)",
35
      "date": "Sat, Aug 26",
36
      "venue_name": "Jerry Garcia Amphitheater • San Francisco, CA",
37
      "start_time": "2:30 PM"
38
    }
39
  ]
40
}

If that’s going to arrive as a sequence of chunks, how can I display those items as soon as they become available?

After much experimentation I figured out this recipe using ijson:

1
import ijson
2
import json
3
import time
4

5
chunks = [
6
    '{\n  "items": [\n    {\n      "ti',
7
    'tle": "Classic and Custom Car ',
8
    'Show",\n      "date": "Sat, Sep',
9
    ' 2",\n      "venue_name": "1700',
10
    " W Hillsdale Blvd • San Mateo,",
11
    ' CA",\n      "start_time": "9:0',
12
    '0 AM"\n    },\n    {\n      "titl',
13
    'e": "【衛蘭、王君馨、衛詩、鍾舒漫、吉中鳴牧師】MMO ',
14
    'AINEO LIVE 愛你喲！佈道會（三藩市站）",\n   ',
15
    '   "date": "Sat, Aug 26",\n    ',
16
    '  "venue_name": "San Mateo Per',
17
    "forming Arts Center • San Mate",
18
    'o, CA",\n      "start_time": "7',
19
    ':30 PM"\n    },\n    {\n      "ti',
20
    'tle": "San Francisco Small Bus',
21
    'iness Expo 2023",\n      "date"',
22
    ': "Fri, Aug 18",\n      "venue_',
23
    'name": "Hyatt Regency San Fran',
24
    "cisco Airport • Burlingame, CA",
25
    '",\n      "start_time": "10:00 ',
26
    'AM"\n    },\n    {\n      "title"',
27
    ': "Stanford Genetics Conferenc',
28
    "e on Structural Variants and D",
29
    'NA Repeats",\n      "date": "Th',
30
    'u, Sep 7",\n      "venue_name":',
31
    ' "Stanford Center for Academic',
32
    ' Medicine • Palo Alto, CA",\n  ',
33
    '    "start_time": "8:00 AM"\n  ',
34
    '  },\n    {\n      "title": "DJ ',
35
    "QUIK / ROJAI / Z-MAN / DJ SAUR",
36
    'US",\n      "date": "Fri, Aug 1',
37
    '8",\n      "venue_name": "The L',
38
    "ongboard Margarita Bar • Pacif",
39
    'ica, CALIFORNIA",\n      "start',
40
    '_time": "9:00 PM"\n    },\n    {',
41
    '\n      "title": "DUE SOUTH W/ ',
42
    "CHERRY GLAZERR, MOMMA (DUO), K",
43
    'ING ISIS (FREE!)",\n      "date',
44
    '": "Sat, Aug 26",\n      "venue',
45
    '_name": "Jerry Garcia Amphithe',
46
    'ater • San Francisco, CA",\n   ',
47
    '   "start_time": "2:30 PM"\n   ',
48
    " }\n  ]\n}",
49
]
50

51

52
events = ijson.sendable_list()
53
coro = ijson.items_coro(events, "items.item")
54

55
seen_events = set()
56

57
for chunk in chunks:
58
    coro.send(chunk.encode("utf-8"))
59
    if events:
60
        # Any we have not seen yet?
61
        unseen_events = [e for e in events if json.dumps(e) not in seen_events]
62
        if unseen_events:
63
            for event in unseen_events:
64
                seen_events.add(json.dumps(event))
65
                print(json.dumps(event))
66
                time.sleep(1)

You create an ijson coroutine, then send it chunks of JSON data. It will write new items to the sendable_list() as soon as they are available.

The hardest part to figure out was this:

1
coro = ijson.items_coro(events, "items.item")

This is the syntax to indicate that I’m interested in the array items in the {"items": [...]} object.

The confusing part is that item here is a reserved word in ijson which means “individual items of an array”. I’m not sure what you are meant to do if your nested object also has a key called "item" - I guess work to avoid that situation from coming up!

This works though. The above code, when executed, prints out each of the nested objects one at a time, with a 1 second sleep between each one.

Newsletter

Processing a stream of chunks of JSON with ijson#