Cryptography in Pyodide#

Today I was evaluating if the Python cryptography package was a sensible depedency for one of my projects.

It’s clearly the most responsible package to use for implementing encryption in Python, but I was nervous about adding it as a dependency to a project that could work without using it at all (with some design changes).

My key concern: I want this project to work running in Pyodide in WebAssembly in the future, and was nervous that cryptography, being written partially in Rust, wouldn’t work in that environment.

It turns out it works just fine!

I tested it in the Pyodide REPL at https://pyodide.org/en/stable/console.html

The following import code worked without errors:

1
import micropip
2
await micropip.install("cryptography")
3
from cryptography.fernet import Fernet

Then I tested it like this:

1
key = Fernet.generate_key()
2
cipher_suite = Fernet(key)
3
encrypted_text = cipher_suite.encrypt(b"Secret message")
4
print(encrypted_text)
5
# b'gAAAAABlY7AZ-OuJAEv1J4KufF8vpreyehPeejdPqluXwD0G_mfFg-Zl1AvEza6F2DXbWMEkIcKc4B0Hb0qJ457pje5FhO5Vyw=='
6
decrypted_text = cipher_suite.decrypt(encrypted_text)
7
print(decrypted_text)
8
# b'Secret message'

From sniffing around in the browser DevTools network panel, it turns out Pyodide provides its own packaged version of the Cryptography package in a file called cryptography-39.0.2-cp311-cp311-emscripten_3_1_45_wasm32.whl.

I found the source of this custom package in the Pyodide repository, in the packages/cryptography folder.

More packages#

That packages/ folder has a whole bunch of other useful modules that have been custom packaged to work with Pyodide. A few that caught my eye:

Jinja2
Pillow
Pygments
biopython
fastparquet
ffmpeg
gdal
geopandas
geos
html5lib
jsonschema
libheif
libwebp
lxml
msgpack
shapely
sqlalchemy
xgboost
sqlite3

Each of these comes with a meta.yaml file that defines how it should be compiled, plus a test Python module to verify it and optionally a set of patches to apply before compilation.

Here’s what packages/sqlite3/meta.yaml looks like:

1
package:
2
  name: sqlite3
3
  version: 1.0.0 # Nonsense
4
  tag:
5
    - always
6
  top-level:
7
    - sqlite3
8
    - _sqlite3
9
source:
10
  sha256: $(PYTHON_ARCHIVE_SHA256)
11
  url: $(PYTHON_ARCHIVE_URL)
12
build:
13
  type: cpython_module
14
  script: |
15
    export FILES=(
16
      "Modules/_sqlite/blob.c"
17
      "Modules/_sqlite/connection.c"
18
      "Modules/_sqlite/cursor.c"
19
      "Modules/_sqlite/microprotocols.c"
20
      "Modules/_sqlite/module.c"
21
      "Modules/_sqlite/prepare_protocol.c"
22
      "Modules/_sqlite/row.c"
23
      "Modules/_sqlite/statement.c"
24
      "Modules/_sqlite/util.c"
25
    )
26

27
    embuilder build sqlite3 --pic
28

29
    for file in "${FILES[@]}"; do
30
      emcc $STDLIB_MODULE_CFLAGS -c "${file}" -o "${file/.c/.o}"  \
31
        -sUSE_SQLITE3 -DMODULENAME=sqlite
32
    done
33

34
    OBJECT_FILES=$(find Modules/_sqlite/ -name "*.o")
35
    emcc $OBJECT_FILES -o $DISTDIR/_sqlite3.so $SIDE_MODULE_LDFLAGS \
36
       -sUSE_SQLITE3 -lsqlite3
37

38
    cd Lib && tar --exclude=test -cf - sqlite3 | tar -C $DISTDIR -xf -

One thing I don’t understand is why some pure-Python packages such as Click also get this treatment - since those should work if installed directly from PyPI using micropip.install().

Update: John Ott figured this out. There’s an open issue, Remove pure Python packages from the repository, which says:

Currently, we have dozens of pure Python packages in the repository. Some packages require a patch, and some packages don’t have a wheel, but many packages exist only because other packages with C-extension depend on them in runtime.