Chromadb on fly.io

I've been using the model of having all of these fly.io apps as "services", which scale to zero, and have persistent storage. Everything goes to sleep when you don't use it, and takes only a second or to to come up. Pay for what you need. Pretty nifty.

Lets setup a chromadb instance.

Setup the app

1
fly launch --no-deploy

Lets add a mounts section to have the database stored somewhere, and so we'll end up with something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
  app = 'chromadb-on-fly-io'
  primary_region = 'ewr'

  [http_service]
    internal_port = 8000
    force_https = true
    auto_stop_machines = true
    auto_start_machines = true
    min_machines_running = 0
    processes = ['app']

  [[vm]]
    memory = '2gb'
    cpu_kind = 'shared'

  [mounts]
    source = "chromadb"
    destination = "/chroma/chroma"
    initial_size = "100gb"

Then we can create a simple Dockerfile.

1
FROM chromadb/chroma

And testing

1
curl https://chromadb-on-fly-io.fly.dev/

{"detail":"Not Found"}

Writing test code to connect

1
  npm i chromadb

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
  // no-auth-test.js

  import { ChromaClient } from "chromadb";

  const client = new ChromaClient({
      path: "https://chromadb-on-fly-io.fly.dev"
  });

  const collection = await client.getOrCreateCollection({
      name: "my_collection",
      metadata: {
          description: "My first collection"
      }
  });

  const collections = await client.listCollections();

  console.log( "collections", collections );

1
  node no-auth-test.js

collections [
  {
    name: 'my_collection',
    id: '95d68c89-5ee6-42f2-9421-8cb57b8f9aeb',
    metadata: { description: 'My first collection' },
    tenant: 'default_tenant',
    database: 'default_database'
  }
]

Adding in auth

This is fully open, so lets add some access control. We'll go with access token method, which is super simple. Add this to the fly.toml file.

1
2
3
  [env]
      CHROMA_SERVER_AUTHN_CREDENTIALS="test-token"
      CHROMA_SERVER_AUTHN_PROVIDER="chromadb.auth.token_authn.TokenAuthenticationServerProvider"

And

1
  fly deploy

Try it again:

1
  node no-auth-test.js

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
  // with-auth-test.js

  import { ChromaClient } from "chromadb";

  const client = new ChromaClient({
      path: "https://chromadb-on-fly-io.fly.dev",
      auth: {provider: "token", credentials: "test-token"}
  });

  const collection = await client.getOrCreateCollection({
      name: "my_authed_collection",
      metadata: {
          description: "My second collection"
      }
  });

  const collections = await client.listCollections();

  console.log( "collections", collections );

1
  node with-auth-test.js

collections [
  {
    name: 'my_collection',
    id: '95d68c89-5ee6-42f2-9421-8cb57b8f9aeb',
    metadata: { description: 'My first collection' },
    tenant: 'default_tenant',
    database: 'default_database'
  },
  {
    name: 'my_authed_collection',
    id: '98930e2d-5f72-4a6f-a185-bbe4d606f040',
    metadata: { description: 'My second collection' },
    tenant: 'default_tenant',
    database: 'default_database'
  }
]

Chromadb on fly.io

Setup the app

Writing test code to connect

Adding in auth

References