[READ-ONLY] Mirror of https://github.com/flo-bit/contrail. atproto backend in a bottle flo-bit.dev/contrail/
0

Configure Feed

Select the types of activity you want to include in your feed.

update docs

Florian (Apr 24, 2026, 1:04 PM +0200) e6f545a3 9a6fca7a

+504 -43
+49 -43
README.md
··· 1 1 # Contrail 2 2 3 - > **Work in progress!** Pre-alpha, expect breaking changes. 3 + > **Pre-alpha.** Expect breaking changes. 4 4 5 - Contrail is a library (and a small set of sibling packages) for building AT Protocol appviews. Define collections, get automatic Jetstream ingestion, PDS backfill, typed XRPC endpoints, permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. 5 + A library for building AT Protocol appviews. Declare your collections, get automatic Jetstream ingestion, typed XRPC endpoints, optional permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. Runs on Cloudflare Workers + D1, Node.js + PostgreSQL, or SvelteKit. 6 + 7 + ## Install 8 + 9 + ```bash 10 + pnpm add @atmo-dev/contrail 11 + ``` 12 + 13 + ## Minimal example 14 + 15 + ```ts 16 + import { Contrail } from "@atmo-dev/contrail"; 17 + 18 + const contrail = new Contrail({ 19 + namespace: "com.example", 20 + db, // D1, node:sqlite, or @atmo-dev/contrail/postgres 21 + collections: { 22 + event: { 23 + collection: "community.lexicon.calendar.event", 24 + queryable: { startsAt: { type: "range" } }, 25 + searchable: ["name", "description"], 26 + }, 27 + }, 28 + }); 29 + 30 + await contrail.init(); 31 + await contrail.ingest(); // pulls from Jetstream 32 + ``` 33 + 34 + Mount the XRPC handler in any fetch-style framework: 35 + 36 + ```ts 37 + import { createHandler } from "@atmo-dev/contrail/server"; 38 + export default { fetch: createHandler(contrail) }; 39 + ``` 40 + 41 + ## Docs 42 + 43 + - [Indexing](./docs/indexing.md) — the core: collections, queries, ingestion, adapters 44 + - [Spaces](./docs/spaces.md) — permissioned records stored by the appview 45 + - [Communities](./docs/communities.md) — group-controlled atproto DIDs 46 + - [Sync](./docs/sync.md) — reactive client-side store over `watchRecords` 47 + - [Lexicons](./docs/lexicons.md) — `contrail-lex` CLI and codegen 6 48 7 49 ## Packages 8 50 9 - | Package | Description | 51 + | Package | | 10 52 |---|---| 11 - | [`@atmo-dev/contrail`](./packages/contrail) | Core library — indexing, XRPC server, spaces, communities, realtime publishing. | 12 - | [`@atmo-dev/contrail-sync`](./packages/sync) | Client-side reactive watch-store over `watchRecords`. SSE + WebSocket, IndexedDB cache. | 13 - | [`@atmo-dev/contrail-lexicons`](./packages/lexicons) | Lexicon codegen from a Contrail config + CLI (`contrail-lex`) wrapping `@atcute/lex-cli`. | 53 + | `@atmo-dev/contrail` | Core library — indexing, XRPC server, spaces, communities, realtime | 54 + | `@atmo-dev/contrail-sync` | Client-side reactive watch-store with optional IndexedDB cache | 55 + | `@atmo-dev/contrail-lexicons` | Codegen + `contrail-lex` CLI | 14 56 15 - ## Apps (reference deployments — `workspace:*`-linked) 16 - 17 - | App | Description | 18 - |---|---| 19 - | [`rsvp-atmo`](./apps/rsvp-atmo) | Cloudflare Workers + D1 indexer for `community.lexicon.calendar.*`. | 20 - | [`group-chat`](./apps/group-chat) | Full-featured SvelteKit + Workers group chat using spaces, communities, and realtime. | 21 - | [`postgres`](./apps/postgres) | Node + PostgreSQL minimal indexer. | 22 - | [`cloudflare-workers`](./apps/cloudflare-workers) | Minimal Worker example. | 23 - | [`sveltekit-cloudflare-workers`](./apps/sveltekit-cloudflare-workers) | SvelteKit Statusphere-style example. | 24 - 25 - ## Setup 26 - 27 - ```bash 28 - pnpm install 29 - pnpm build 30 - pnpm test 31 - ``` 32 - 33 - Per-package commands run through Turbo: 34 - 35 - ```bash 36 - pnpm build # turbo run build 37 - pnpm typecheck # turbo run typecheck 38 - pnpm test # turbo run test 39 - pnpm --filter @atmo-dev/contrail build 40 - pnpm --filter rsvp-atmo dev 41 - ``` 42 - 43 - ## Releasing 44 - 45 - Changesets drive versioning. `@atmo-dev/contrail` and `@atmo-dev/contrail-sync` are `linked` so their versions stay aligned (they share the realtime wire protocol). 46 - 47 - ```bash 48 - pnpm changeset # add a changeset 49 - pnpm changeset version # bump versions 50 - pnpm release # build + publish 51 - ``` 57 + Working in this repo? See [development.md](https://github.com/flo-bit/contrail/blob/main/development.md) for the monorepo layout and commands.
+51
development.md
··· 1 + # Contrail 2 + 3 + > **Work in progress!** Pre-alpha, expect breaking changes. 4 + 5 + Contrail is a library (and a small set of sibling packages) for building AT Protocol appviews. Define collections, get automatic Jetstream ingestion, PDS backfill, typed XRPC endpoints, permissioned spaces for private records, group-controlled communities, and a client-side reactive sync layer. 6 + 7 + ## Packages 8 + 9 + | Package | Description | 10 + |---|---| 11 + | [`@atmo-dev/contrail`](./packages/contrail) | Core library — indexing, XRPC server, spaces, communities, realtime publishing. | 12 + | [`@atmo-dev/contrail-sync`](./packages/sync) | Client-side reactive watch-store over `watchRecords`. SSE + WebSocket, IndexedDB cache. | 13 + | [`@atmo-dev/contrail-lexicons`](./packages/lexicons) | Lexicon codegen from a Contrail config + CLI (`contrail-lex`) wrapping `@atcute/lex-cli`. | 14 + 15 + ## Apps (reference deployments — `workspace:*`-linked) 16 + 17 + | App | Description | 18 + |---|---| 19 + | [`rsvp-atmo`](./apps/rsvp-atmo) | Cloudflare Workers + D1 indexer for `community.lexicon.calendar.*`. | 20 + | [`group-chat`](./apps/group-chat) | Full-featured SvelteKit + Workers group chat using spaces, communities, and realtime. | 21 + | [`postgres`](./apps/postgres) | Node + PostgreSQL minimal indexer. | 22 + | [`cloudflare-workers`](./apps/cloudflare-workers) | Minimal Worker example. | 23 + | [`sveltekit-cloudflare-workers`](./apps/sveltekit-cloudflare-workers) | SvelteKit Statusphere-style example. | 24 + 25 + ## Setup 26 + 27 + ```bash 28 + pnpm install 29 + pnpm build 30 + pnpm test 31 + ``` 32 + 33 + Per-package commands run through Turbo: 34 + 35 + ```bash 36 + pnpm build # turbo run build 37 + pnpm typecheck # turbo run typecheck 38 + pnpm test # turbo run test 39 + pnpm --filter @atmo-dev/contrail build 40 + pnpm --filter rsvp-atmo dev 41 + ``` 42 + 43 + ## Releasing 44 + 45 + Changesets drive versioning. `@atmo-dev/contrail` and `@atmo-dev/contrail-sync` are `linked` so their versions stay aligned (they share the realtime wire protocol). 46 + 47 + ```bash 48 + pnpm changeset # add a changeset 49 + pnpm changeset version # bump versions 50 + pnpm release # build + publish 51 + ```
+54
docs/communities.md
··· 1 + # Communities 2 + 3 + Group-controlled atproto DIDs. A community is a DID whose signing/rotation keys are held by the appview on behalf of multiple members, with tiered access levels. Built on top of [spaces](./spaces.md). 4 + 5 + ## When to use this 6 + 7 + When you want atproto records published under a *shared* identity — a team, a project, a channel — not a single user. Think: a group's published calendar events, a community's published posts. 8 + 9 + ## Two modes 10 + 11 + - **Minted** — contrail creates a fresh `did:plc` for the community, holds the signing + rotation keys, publishes from it. 12 + - **Adopted** — contrail takes over an existing DID whose rotation keys were handed over by the owner. 13 + 14 + Either way, the result is the same: a DID that multiple members can act through, gated by access levels. 15 + 16 + ## Access levels 17 + 18 + Each member has a level (ranked). Levels map to write permissions. Owners can grant/revoke levels. Two reserved levels exist: `owner` and `member`. Your deployment defines the rest: 19 + 20 + ```ts 21 + community: { 22 + masterKey: ENV.COMMUNITY_MASTER_KEY, // 32-byte encryption key for stored credentials 23 + serviceDid: "did:web:example.com", 24 + levels: ["admin", "moderator"], // ranked, highest-first 25 + } 26 + ``` 27 + 28 + Stored credentials (app passwords for adopted communities, signing keys for minted) are envelope-encrypted with `masterKey`. Never ship the placeholder. 29 + 30 + ## How it composes with spaces 31 + 32 + A community *owns* spaces. Members of the community get access to community-owned spaces based on their level. Grant access per-space per-level: 33 + 34 + ``` 35 + community.space.grant { spaceUri, level: "admin", perms: "write" } 36 + ``` 37 + 38 + The spaces layer stays ignorant of access levels — it just sees "this DID is a member with these perms." The community layer projects member × level → space perms. 39 + 40 + ## XRPCs 41 + 42 + - `com.example.community.mint | adopt | list | delete` 43 + - `com.example.community.invite.create | redeem | revoke | list` 44 + - `com.example.community.setAccessLevel | revoke | listMembers` 45 + - `com.example.community.space.create | grant | revoke | ...` — community-owned spaces 46 + - `com.example.community.putRecord | deleteRecord` — publish records as the community DID 47 + 48 + ## What's not here 49 + 50 + - No per-record per-level ACLs. Model as spaces. 51 + - No auto-rotation on key compromise yet. 52 + - Adoption is irreversible without manual key surrender back to the owner. 53 + 54 + The design follows zicklag's [Arbiter design sketch](https://zicklag.leaflet.pub/3mjrvb5pul224) for group management on atproto. The post is an early design note; our implementation will track it as the spec firms up.
+96
docs/indexing.md
··· 1 + # Indexing 2 + 3 + Contrail's core job: mirror atproto records into your DB and expose them via XRPC. You describe what to index with a config object; everything else is automatic. 4 + 5 + ## Collection shape 6 + 7 + ```ts 8 + collections: { 9 + event: { 10 + collection: "community.lexicon.calendar.event", // full NSID 11 + queryable: { 12 + mode: {}, // ?mode=online 13 + startsAt: { type: "range" }, // ?startsAtMin=...&startsAtMax=... 14 + }, 15 + searchable: ["name", "description"], // FTS5 / tsvector 16 + relations: { 17 + rsvps: { 18 + collection: "rsvp", 19 + groupBy: "status", 20 + groups: { going: "community.lexicon.calendar.rsvp#going" }, 21 + }, 22 + }, 23 + references: { 24 + event: { collection: "event", field: "subject.uri" }, 25 + }, 26 + }, 27 + } 28 + ``` 29 + 30 + - **queryable** — string equality or range, exposed as query params. 31 + - **searchable** — FTS5 on D1/Postgres. Not available on `node:sqlite`. 32 + - **relations** — materialized many-to-one counts (`rsvpsGoingCount`). 33 + - **references** — forward lookups. Hydrate inline with `?hydrateEvent=true`. 34 + 35 + ## Ingestion 36 + 37 + Three ways records land in the DB: 38 + 39 + ```ts 40 + await contrail.ingest(); // one Jetstream cycle, then stops 41 + await contrail.runPersistent(); // long-lived connection, auto-reconnect 42 + await contrail.notify(uri); // immediate PDS fetch for one record 43 + ``` 44 + 45 + Call `notify()` after your app writes to a PDS and needs the change reflected now. Jetstream catches the same event later; the duplicate is detected by CID. 46 + 47 + ## Discovery + backfill 48 + 49 + ```ts 50 + await contrail.discover(); // find users from relays 51 + await contrail.backfill({ concurrency: 100 }); // pull their history 52 + await contrail.sync(); // both 53 + ``` 54 + 55 + ## Querying 56 + 57 + ```ts 58 + const { records, cursor } = await contrail.query("event", { 59 + filters: { mode: "online" }, 60 + sort: { field: "startsAt", direction: "asc" }, 61 + limit: 20, 62 + }); 63 + ``` 64 + 65 + Or via HTTP once the handler is mounted: 66 + 67 + ``` 68 + /xrpc/com.example.event.listRecords?mode=online&sort=startsAt&limit=20 69 + /xrpc/com.example.event.getRecord?uri=at://did:plc:.../... 70 + ``` 71 + 72 + ## Adapters 73 + 74 + | Adapter | Use when | FTS | 75 + |---|---|---| 76 + | Cloudflare D1 | Workers | ✅ | 77 + | `@atmo-dev/contrail/sqlite` | Node 22+ local dev | ❌ | 78 + | `@atmo-dev/contrail/postgres` | Node server | ✅ | 79 + 80 + ```ts 81 + import { createPostgresDatabase } from "@atmo-dev/contrail/postgres"; 82 + const db = createPostgresDatabase(pool); 83 + ``` 84 + 85 + ## Top-level config 86 + 87 + | Key | Default | | 88 + |---|---|---| 89 + | `namespace` | — | Reverse-domain for XRPC paths | 90 + | `profiles` | `["app.bsky.actor.profile"]` | Profile NSIDs, auto-hydrated via `?profiles=true` | 91 + | `jetstreams` | Bluesky | Jetstream URLs | 92 + | `relays` | Bluesky | Relay URLs for discovery | 93 + | `notify` | off | `true` opens `notifyOfUpdate`; a string requires `Bearer` | 94 + | `spaces` | — | See [Spaces](./spaces.md) | 95 + | `community` | — | See [Communities](./communities.md) | 96 + | `realtime` | — | See [Sync](./sync.md) |
+91
docs/lexicons.md
··· 1 + # Lexicons 2 + 3 + Contrail emits lexicon JSON for every XRPC method it exposes. Your app needs those JSON files for two things: publishing to your PDS (so other apps can discover them) and generating TypeScript types for typed clients. The `contrail-lex` CLI handles both. 4 + 5 + ```bash 6 + pnpm add -D @atmo-dev/contrail-lexicons @atcute/lex-cli 7 + ``` 8 + 9 + `@atcute/lex-cli` is a peer dep — you pin the version. 10 + 11 + ## CLI 12 + 13 + ```bash 14 + contrail-lex generate # emit lexicon JSON from your Contrail config 15 + contrail-lex pull # wraps `lex-cli pull` (fetches external lexicons) 16 + contrail-lex types # wraps `lex-cli generate` (JSON → TS types) 17 + contrail-lex all # generate → pull → generate → pull → types 18 + contrail-lex all --no-types # skip the type step 19 + ``` 20 + 21 + The CLI auto-detects your config at `contrail.config.ts`, `app/config.ts`, or `src/lib/contrail/config.ts`. Override with `--config <path>`. 22 + 23 + ## What lands where 24 + 25 + Running `contrail-lex all` against a config with `namespace: "com.example"` writes: 26 + 27 + ``` 28 + lexicons-generated/ # JSON lexicons emitted from your config 29 + lexicons-pulled/ # external NSIDs fetched by lex-cli pull 30 + lex.config.js # regenerated each run (add to .gitignore) 31 + src/lexicon-types/ # TS types emitted by lex-cli generate 32 + ``` 33 + 34 + `lexicons-generated/` and `lexicons-pulled/` should be **committed** — that way CI doesn't need network access and consumers of your repo can pull the JSON as a lexicon source. `lex.config.js` and `src/lexicon-types/` are generated on demand and safe to gitignore. 35 + 36 + ## Publishing to a PDS 37 + 38 + Once committed, publish the lexicons to an atproto account so other apps can resolve them: 39 + 40 + ```ts 41 + import { publishLexicons } from "@atmo-dev/contrail-lexicons"; 42 + 43 + await publishLexicons({ 44 + generatedDir: "lexicons-generated", 45 + identifier: process.env.LEXICON_ACCOUNT_IDENTIFIER, 46 + password: process.env.LEXICON_ACCOUNT_PASSWORD, 47 + }); 48 + ``` 49 + 50 + Writes each lexicon as a `com.atproto.lexicon.schema` record under `at://<did>/com.atproto.lexicon.schema/<nsid>`. You'll also need DNS TXT records — publishing prints them for you. 51 + 52 + ## Programmatic API 53 + 54 + ```ts 55 + import { generateLexicons, extractXrpcMethods } from "@atmo-dev/contrail-lexicons"; 56 + 57 + const generated = generateLexicons({ 58 + config, 59 + rootDir: process.cwd(), 60 + outputDir: "lexicons-generated", 61 + }); 62 + 63 + const methods = extractXrpcMethods(generated); // NSIDs of every query + procedure 64 + ``` 65 + 66 + Handy for emitting the method list for your OAuth permission set. 67 + 68 + ## Consuming a third-party contrail instance 69 + 70 + Someone else runs a contrail deployment; you want typed client access to their XRPCs. Point `lex-cli pull` at their git repo: 71 + 72 + ```js 73 + // lex.config.js 74 + import { defineLexiconConfig } from "@atcute/lex-cli"; 75 + 76 + export default defineLexiconConfig({ 77 + outdir: "src/lexicon-types/", 78 + imports: ["@atcute/atproto"], 79 + files: ["lexicons/**/*.json"], 80 + pull: { 81 + outdir: "lexicons/", 82 + sources: [{ 83 + type: "git", 84 + remote: "https://github.com/them/their-contrail.git", 85 + pattern: ["lexicons-generated/**/*.json"], 86 + }], 87 + }, 88 + }); 89 + ``` 90 + 91 + Then `npx lex-cli pull && npx lex-cli generate`, and their endpoints are typed in your client.
+70
docs/spaces.md
··· 1 + # Spaces 2 + 3 + Auth-gated store for records that can't live on public PDSes — private events, invite-only groups, members-only chat. Opt-in; zero cost if you don't enable it. 4 + 5 + ## Mental model 6 + 7 + > A **space** is a bag of records with one lock. The **member list** says who has the key. 8 + 9 + - One owner (DID), one type (NSID), one key. Identified by `at://<owner>/<type>/<key>`. 10 + - Members have `read` or `write`. Owner is implicit `write`. 11 + - Optional **app policy** gates which OAuth clients can act in the space. 12 + 13 + Every permission boundary is its own space. No nested ACLs. Richer roles = more spaces or app-layer checks. 14 + 15 + ## Enable 16 + 17 + ```ts 18 + import type { ContrailConfig } from "@atmo-dev/contrail"; 19 + 20 + const config: ContrailConfig = { 21 + namespace: "com.example", 22 + collections: { /* ... */ }, 23 + spaces: { 24 + type: "com.example.event.space", 25 + serviceDid: "did:web:example.com", 26 + }, 27 + }; 28 + ``` 29 + 30 + Each collection gets a parallel `spaces_records_<short>` table. Opt out per-collection: 31 + 32 + ```ts 33 + public_only: { collection: "com.example.public", allowInSpaces: false } 34 + ``` 35 + 36 + ## Auth 37 + 38 + atproto service-auth JWTs via `@atcute/xrpc-server`. Middleware validates signature, `aud`, and `lxm` before the handler runs. Apps acting in a space mint a JWT against the user's PDS with `Atproto-Proxy: did:web:example.com#com_example_space`, forward to your service, it verifies and executes. 39 + 40 + **Note:** Use the plain DID (no `#fragment`) as `serviceDid` — the fragment form only belongs in your DID doc's service entry. 41 + 42 + ## Unified `listRecords` 43 + 44 + | Call | Returns | 45 + |---|---| 46 + | no auth, no `spaceUri` | public only | 47 + | `?spaceUri=…` + JWT | one space (ACL-gated) | 48 + | JWT, no `spaceUri` | public **unioned** with every space the caller is a member of | 49 + 50 + Filters, sorts, hydration, and references work across all three. Records from a space carry a `space: <spaceUri>` field. 51 + 52 + ## Invites 53 + 54 + Stored as hashed tokens. Redemption is a single atomic UPDATE that checks `!revoked && !expired && !exhausted`. Generate, hand out the plaintext once, verify later. 55 + 56 + ## XRPCs 57 + 58 + - `com.example.space.create | get | list | delete` 59 + - `com.example.space.putRecord | deleteRecord | listRecords | getRecord` 60 + - `com.example.space.invite.create | redeem | revoke | list` 61 + - `com.example.space.listMembers | removeMember` 62 + 63 + ## What's not here 64 + 65 + - No E2EE (data is operator-readable). 66 + - No FTS on `?spaceUri=…` yet. 67 + - No per-space sharding — one DB, one operator. 68 + - Not a long-term replacement for real atproto permissioned repos. 69 + 70 + The design follows Daniel Holmgren's [permissioned data rough spec](https://dholms.leaflet.pub/3mhj6bcqats2o). The goal is that when real atproto permissioned repos ship, migration is mostly data movement — the API your app speaks doesn't change.
+93
docs/sync.md
··· 1 + # Sync 2 + 3 + Client-side reactive store over contrail's `watchRecords` endpoints. Subscribes once, reconciles forever, ships with optimistic updates and an optional IndexedDB cache. 4 + 5 + Lives in its own package: 6 + 7 + ```bash 8 + pnpm add @atmo-dev/contrail-sync 9 + ``` 10 + 11 + ## Basic use 12 + 13 + ```ts 14 + import { createWatchStore } from "@atmo-dev/contrail-sync"; 15 + 16 + const store = createWatchStore({ 17 + url: "/xrpc/com.example.message.watchRecords?roomUri=at://...", 18 + transport: "sse", // or "ws" 19 + }); 20 + 21 + store.subscribe(({ records, status }) => { /* re-render */ }); 22 + store.start(); 23 + ``` 24 + 25 + Framework-agnostic — wrap it in Svelte `$state`, React `useSyncExternalStore`, Vue `ref`, whatever. 26 + 27 + ## Transports 28 + 29 + - **SSE** (default) — one HTTP request, simplest. Works everywhere. 30 + - **WS** — a two-step handshake: HTTP GET returns a snapshot + watch-scoped ticket, then you upgrade to WS. On Cloudflare, the WS terminates on a Durable Object that hibernates idle connections. Same event stream either way. 31 + 32 + ## Authenticated watches 33 + 34 + Pass `mintTicket` for any non-public endpoint. One-shot string for SSR-minted tickets, function for fresh tickets per reconnect: 35 + 36 + ```ts 37 + mintTicket: async () => (await fetch("/api/ticket")).then((r) => r.text()), 38 + ``` 39 + 40 + Tickets are minted server-side via `com.example.realtime.ticket` (or any app-specific route). 41 + 42 + ## Optimistic updates 43 + 44 + ```ts 45 + store.addOptimistic({ rkey, did, record: { text: "hi" } }); 46 + // later, on mutation failure: 47 + store.markFailed(rkey, err); 48 + // or explicit rollback: 49 + store.removeOptimistic(rkey); 50 + ``` 51 + 52 + When a real record with the same `rkey` arrives via the stream, the optimistic entry is dropped automatically. 53 + 54 + ## IndexedDB cache 55 + 56 + Instant first paint from last session's records: 57 + 58 + ```ts 59 + import { createIndexedDBCache } from "@atmo-dev/contrail-sync/cache-idb"; 60 + 61 + createWatchStore({ 62 + url, 63 + cache: createIndexedDBCache(), 64 + cacheMaxRecords: 200, 65 + }); 66 + ``` 67 + 68 + Cached records show immediately; the live snapshot reconciles when the connection opens. 69 + 70 + ## Server-side config 71 + 72 + Enable `watchRecords` emission in your Contrail config: 73 + 74 + ```ts 75 + realtime: { 76 + ticketSecret: ENV.REALTIME_TICKET_SECRET, // 32 bytes 77 + pubsub: new DurableObjectPubSub(env.PUBSUB), // or in-memory for dev 78 + } 79 + ``` 80 + 81 + See [indexing.md](./indexing.md) for the full config surface. 82 + 83 + ## Lifecycle 84 + 85 + ``` 86 + idle → connecting → snapshot → live 87 + ↓ (disconnect) 88 + reconnecting → snapshot → live 89 + ↓ (stop) 90 + closed 91 + ``` 92 + 93 + Stale records stay visible across reconnects until the fresh snapshot arrives, at which point anything the server didn't re-send is evicted. Survives offline periods cleanly.