Deploying the CLIP embedding model on Fly#

Building the app with Dockerfile and fly.toml#

I ended up needing two files. The first was a Dockerfile, and the second was a fly.toml file to configure the deployment.

Here’s the Dockerfile:

1
FROM python:3.11.0-slim-bullseye
2
COPY . /app
3
WORKDIR /app
4

5
RUN pip install -U torch torchvision --extra-index-url https://download.pytorch.org/whl/cpu
6
RUN pip install -U datasette datasette-scale-to-zero datasette-block-robots datasette-llm-embed llm-clip
7
# Download the CLIP model
8
RUN llm embed -m clip -c 'hello world'
9

10
# Create metadata.json file
11
RUN echo '{"plugins": {"datasette-scale-to-zero": {"duration": "10m"}}}' > /app/metadata.json
12

13
# Expose and start the service
14
ENV PORT 8080
15
EXPOSE 8080
16
CMD datasette serve --host 0.0.0.0 --cors --port $PORT -m /app/metadata.json

The trickiest detail here was installing torch and torchvision. It turns out the Fly builder application I was using had access to CUDA libraries, which caused it to install the ultra-heavy version of PyTorch - multiple GBs of data.

After much experimentation I found that this incantation causes PyTorch to be installed with CPU-only dependencies, which is a fraction of the size:

1
pip install -U torch torchvision \
2
  --extra-index-url https://download.pytorch.org/whl/cpu

The rest of the Dockerfile is pretty standard - it installs some Datasette plugins plus llm-clip.

One trick I’m using is to run llm embed -m clip -c 'hello world' as part of the build process. This causes the CLIP model to be downloaded from Hugging Face and cached as part of the Docker image build process, which means it’s baked into the container and available immediately when the app starts up.

My fly.toml file looks like this:

1
app = "clip-datasette-on-fly"
2

3
[[services]]
4
  internal_port = 8080
5
  protocol = "tcp"
6

7
  [services.concurrency]
8
    hard_limit = 25
9
    soft_limit = 20
10

11
  [[services.ports]]
12
    handlers = ["http"]
13
    port = 80
14

15
  [[services.ports]]
16
    handlers = ["tls", "http"]
17
    port = 443
18

19
  [[services.tcp_checks]]
20
    interval = 10000
21
    timeout = 2000

Deploying the application#

I ran the following commands in the same directory as the fly.toml and Dockerfile files to create the application and then deploy it:

1
flyctl apps create clip-datasette-on-fly
2
flyctl deploy

The app name needs to be unique across all of Fly, so you will need to change this if you want to deploy your own copy.

The default deployment didn’t have enough memory to run CLIP without crashing. I ran these commands to fix that:

1
flyctl scale vm performance-1x
2
flyctl scale memory 8192

If this machine ran all the time it would cost me $61.02/month - but I’ve installed the datasette-scale-to-zero and configured it to stop the server after 10m of no activity, which should save a lot of money.

I also installed datasette-block-robots to prevent search engines from crawling the site and waking it up more than strictly necessary.

Using it from an Observable Notebook#

My Search for Faucets with CLIP, API edition Observable notebook makes calls to my newly deployed API to calculate CLIP embeddings for text, then passes the result to Drew’s Faucet Finder to find matching faucets.

The relevant JavaScript looks like this, where text is the text to be embedded:

1
hex_embedding = {
2
  try {
3
    return (
4
      await (
5
        await fetch(
6
          `https://clip-datasette-on-fly.fly.dev/_memory.json?sql=select+hex(llm_embed(%27clip%27%2C+%3Aq))+as+x&q=${encodeURIComponent(
7
            text
8
          )}&_shape=array`
9
        )
10
      ).json()
11
    )[0].x;
12
  } catch {
13
    return "";
14
  }
15
}

Here’s the HTML version of that SQL query.