Extracting objects recursively with jq#

The Algolia-powered Hacker News API returns nested comment threads that look like this: https://hn.algolia.com/api/v1/items/27941108

(For this story: https://news.ycombinator.com/item?id=27941108)

1
{
2
    "id": 27941108,
3
    "created_at": "2021-07-24T14:15:05.000Z",
4
    "type": "story",
5
    "author": "edward",
6
    "title": "Fun with Unix domain sockets",
7
    "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/",
8
    "children": [
9
        {
10
            "id": 27942287,
11
            "created_at": "2021-07-24T16:31:18.000Z",
12
            "type": "comment",
13
            "author": "DesiLurker",
14
            "text": "<p>one lesser known...",
15
            "children": []
16
        },
17
        {
18
            "id": 27944615,
19
            "created_at": "2021-07-24T21:26:33.000Z",
20
            "type": "comment",
21
            "author": "galaxyLogic",
22
            "text": "<p>I read this from Wikipedia...",
23
            "children": [
24
                {
25
                    "id": 27944746,
26
                    "created_at": "2021-07-24T21:49:07.000Z",
27
                    "type": "comment",
28
                    "author": "hughrr",
29
                    "text": "<p>Yes although I ...",
30
                    "children": []
31
                }
32
            ]
33
        }
34
    ]
35
}

I wanted to flatten this into an array of items so I could send it to sqlite-utils insert. This recipe worked:

1
curl 'https://hn.algolia.com/api/v1/items/27941108' \
2
  | jq '[recurse(.children[]) | del(.children)]' \
3
  | sqlite-utils insert hn.db items - --pk id

The jq recipe here is:

1
[recurse(.children[]) | del(.children)]

The first recurse(.children[]) recurses through a list of everything in a .children array.

The | del(.children) then deletes that array from the returned objects.

Wrapping it all in [ ] ensures the overall result will be an array.

Applied against the above example, this returns:

1
[
2
    {
3
        "id": 27941108,
4
        "created_at": "2021-07-24T14:15:05.000Z",
5
        "type": "story",
6
        "author": "edward",
7
        "title": "Fun with Unix domain sockets",
8
        "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/"
9
    },
10
    {
11
        "id": 27942287,
12
        "created_at": "2021-07-24T16:31:18.000Z",
13
        "type": "comment",
14
        "author": "DesiLurker",
15
        "text": "<p>one lesser known..."
16
    },
17
    {
18
        "id": 27944615,
19
        "created_at": "2021-07-24T21:26:33.000Z",
20
        "type": "comment",
21
        "author": "galaxyLogic",
22
        "text": "<p>I read this from Wikipedia..."
23
    },
24
    {
25
        "id": 27944746,
26
        "created_at": "2021-07-24T21:49:07.000Z",
27
        "type": "comment",
28
        "author": "hughrr",
29
        "text": "<p>Yes although I ..."
30
    }
31
]

Newsletter

Extracting objects recursively with jq#