Using the ChatGPT streaming API from Python#

I wanted to stream the results from the ChatGPT API as they were generated, rather than waiting for the entire thing to complete before displaying anything.

Here’s how to do that with the openai Python library:

1
import openai
2
openai.api_key = open("/Users/simon/.openai-api-key.txt").read().strip()
3

4
for chunk in openai.ChatCompletion.create(
5
    model="gpt-3.5-turbo",
6
    messages=[{
7
        "role": "user",
8
        "content": "Generate a list of 20 great names for sentient cheesecakes that teach SQL"
9
    }],
10
    stream=True,
11
):
12
    content = chunk["choices"][0].get("delta", {}).get("content")
13
    if content is not None:
14
        print(content, end='')

Running this code in a Jupyter notebook does the following:

It outputs a list of cheesecakes one token at a time

Using async/await#

The OpenAI Python library can also work with asyncio. Here’s how to do the above using their async/await support - with the .acreate() method:

1
async for chunk in await openai.ChatCompletion.acreate(
2
    model="gpt-3.5-turbo",
3
    messages=[{
4
        "role": "user",
5
        "content": "Generate a list of 20 great names for sentient cheesecakes that teach SQL"
6
    }],
7
    stream=True,
8
):
9
    content = chunk["choices"][0].get("delta", {}).get("content")
10
    if content is not None:
11
        print(content, end='')

Those chunks#

Here’s what those chunks look like - the first two and then the last two:

1
{
2
  "choices": [
3
    {
4
      "delta": {
5
        "role": "assistant"
6
      },
7
      "finish_reason": null,
8
      "index": 0
9
    }
10
  ],
11
  "created": 1680380941,
12
  "id": "chatcmpl-70c8LVUSYoSbdQTyONgJfcVU542wO",
13
  "model": "gpt-3.5-turbo-0301",
14
  "object": "chat.completion.chunk"
15
}
16

17
{
18
  "choices": [
19
    {
20
      "delta": {
21
        "content": "1"
22
      },
23
      "finish_reason": null,
24
      "index": 0
25
    }
26
  ],
27
  "created": 1680380941,
28
  "id": "chatcmpl-70c8LVUSYoSbdQTyONgJfcVU542wO",
29
  "model": "gpt-3.5-turbo-0301",
30
  "object": "chat.completion.chunk"
31
}
32

33
# ... lots more here ...
34

35
{
36
  "choices": [
37
    {
38
      "delta": {
39
        "content": "ina"
40
      },
41
      "finish_reason": null,
42
      "index": 0
43
    }
44
  ],
45
  "created": 1680380941,
46
  "id": "chatcmpl-70c8LVUSYoSbdQTyONgJfcVU542wO",
47
  "model": "gpt-3.5-turbo-0301",
48
  "object": "chat.completion.chunk"
49
}
50

51
{
52
  "choices": [
53
    {
54
      "delta": {},
55
      "finish_reason": "stop",
56
      "index": 0
57
    }
58
  ],
59
  "created": 1680380941,
60
  "id": "chatcmpl-70c8LVUSYoSbdQTyONgJfcVU542wO",
61
  "model": "gpt-3.5-turbo-0301",
62
  "object": "chat.completion.chunk"
63
}