Using tree-sitter with Python#

tree-sitter is a “parser generator tool and an incremental parsing library”. It has a very good reputation these days.

Two useful posts by Douglas Creager: Getting started with tree-sitter and A map of the tree-sitter ecosystem.

I want to be able to parse SQLite SQL - in particular I want to be able to parse CREATE TABLE statements, because SQLite stores those directly in its sqlite_master metadata table as the main source of truth about a table, and I want to be able to introspect them beyond what’s possible with pragma table_info() and friends.

It turns out there are tree-sitter grammars for a huge array of languages, including one for SQLite SQL hosted at github.com/dhcmrlchtdj/tree-sitter-sqlite.

Compiling a grammer using Python#

tree-sitter grammars need to be compiled - they generate C code, which should then be compiled into a .so library file.

py-tree-sitter provides Python bindings for tree-sitter that can both work with compiled grammars AND manage the compilation process.

Here’s how I compiled the SQLite grammar:

1
git clone https://github.com/dhcmrlchtdj/tree-sitter-sqlite
2
pipenv shell
3
python -m pip install tree_sitter
4
python

1
>>> from tree_sitter import Language, Parser
2
>>> Language.build_library('/tmp/sqlite.so', ['/tmp/tree-sitter-sqlite'])
3
True

This gave me a /tmp/sqlite.so file - 1.1MB in size.

Parsing text using Python#

Here’s how to use that from Python:

1
from tree_sitter import Language, Parser
2
language = Language('/tmp/sqlite.so', 'sqlite')
3
parser = Parser()
4
parser.set_language(language)
5
sql = b"""CREATE TABLE _datasette_auth_tokens (
6
   id INTEGER PRIMARY KEY,
7
   secret TEXT,
8
   description TEXT,
9
   permissions TEXT,
10
   actor_id TEXT,
11
   created_timestamp INTEGER,
12
   last_used_timestamp INTEGER,
13
   expires_after_seconds INTEGER
14
);"""
15
tree = parser.parse(sql)
16
print(tree.root_node.sexp())

Which outputs:

I explored the tree structure a bit in Python as well:

1
>>> n = tree.root_node
2
>>> dir(n)
3
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'child_by_field_id', 'child_by_field_name', 'child_count', 'children', 'children_by_field_id', 'children_by_field_name', 'end_byte', 'end_point', 'field_name_for_child', 'has_changes', 'has_error', 'id', 'is_missing', 'is_named', 'named_child_count', 'named_children', 'next_named_sibling', 'next_sibling', 'parent', 'prev_named_sibling', 'prev_sibling', 'sexp', 'start_byte', 'start_point', 'text', 'type', 'walk']
4
>>> n.text
5
b'CREATE TABLE _datasette_auth_tokens (\n   id INTEGER PRIMARY KEY,\n   secret TEXT,\n   description TEXT,\n   permissions TEXT,\n   actor_id TEXT,\n   created_timestamp INTEGER,\n   last_used_timestamp INTEGER,\n   expires_after_seconds INTEGER\n);'
6
>>> n.children
7
[<Node type=sql_stmt, start_point=(0, 0), end_point=(9, 1)>, <Node type=";", start_point=(9, 1), end_point=(9, 2)>]
8
>>> n.children[0]
9
<Node type=sql_stmt, start_point=(0, 0), end_point=(9, 1)>
10
>>> n.children[0].children
11
[<Node type=create_table_stmt, start_point=(0, 0), end_point=(9, 1)>]
12
>>> n.children[0].children[0]
13
<Node type=create_table_stmt, start_point=(0, 0), end_point=(9, 1)>
14
>>> n.children[0].children[0].children
15
[<Node type=CREATE, start_point=(0, 0), end_point=(0, 6)>,
16
 <Node type=TABLE, start_point=(0, 7), end_point=(0, 12)>,
17
 <Node type=identifier, start_point=(0, 13), end_point=(0, 35)>,
18
 <Node type="(", start_point=(0, 36), end_point=(0, 37)>,
19
 <Node type=column_def, start_point=(1, 3), end_point=(1, 25)>,
20
 <Node type=",", start_point=(1, 25), end_point=(1, 26)>,
21
 <Node type=column_def, start_point=(2, 3), end_point=(2, 14)>,
22
 <Node type=",", start_point=(2, 14), end_point=(2, 15)>,
23
 <Node type=column_def, start_point=(3, 3), end_point=(3, 19)>,
24
 <Node type=",", start_point=(3, 19), end_point=(3, 20)>,
25
 <Node type=column_def, start_point=(4, 3), end_point=(4, 19)>,
26
 <Node type=",", start_point=(4, 19), end_point=(4, 20)>,
27
 <Node type=column_def, start_point=(5, 3), end_point=(5, 16)>,
28
 <Node type=",", start_point=(5, 16), end_point=(5, 17)>,
29
 <Node type=column_def, start_point=(6, 3), end_point=(6, 28)>,
30
 <Node type=",", start_point=(6, 28), end_point=(6, 29)>,
31
 <Node type=column_def, start_point=(7, 3), end_point=(7, 30)>,
32
 <Node type=",", start_point=(7, 30), end_point=(7, 31)>,
33
 <Node type=column_def, start_point=(8, 3), end_point=(8, 32)>,
34
 <Node type=")", start_point=(9, 0), end_point=(9, 1)>]
35
>>> n.children[0].children[0].text
36
b'CREATE TABLE _datasette_auth_tokens (\n   id INTEGER PRIMARY KEY,\n   secret TEXT,\n   description TEXT,\n   permissions TEXT,\n   actor_id TEXT,\n   created_timestamp INTEGER,\n   last_used_timestamp INTEGER,\n   expires_after_seconds INTEGER\n)'

I’ve only just started exploring tree-sitter - the Python documentation has more details on ways to walk the tree, plus a description of tree-sitter’s pattern matching language which looks like it may be the key to using it effectively.

The tree-sitter-languages package#

grantjenks/py-tree-sitter-languages is a handy Python package that bundles compiled versions of a bunch of grammars for different architectures, as published wheel files:

https://pypi.org/project/tree-sitter-languages/#files

This means you can install that package rather than compiling them yourself:

1
pip install tree-sitter-languages

I had a PR accepted adding the SQLite grammar. Now you can do this:

1
from tree_sitter_languages import get_parser
2

3
parser = get_parser("sqlite")
4
tree = parser.parse(b"""CREATE TABLE [replicate_predictions] (
5
   [id] TEXT PRIMARY KEY,
6
   [_model_guess] TEXT,
7
   [completed_at] TEXT,
8
   [created_at] TEXT,
9
   [error] TEXT,
10
   [input] TEXT,
11
   [logs] TEXT,
12
   [metrics] TEXT,
13
   [output] TEXT,
14
   [started_at] TEXT,
15
   [status] TEXT,
16
   [urls] TEXT,
17
   [version] TEXT,
18
   [webhook_completed] TEXT
19
)""")
20
print(tree.root_node.sexp())

Output:

(sql_stmt_list (sql_stmt (create_table_stmt (CREATE) (TABLE) (identifier) (column_def (identifier) (type_name (identifier)) (column_constraint (PRIMARY) (KEY))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))) (column_def (identifier) (type_name (identifier))))))