[READ-ONLY] Mirror of https://github.com/flo-bit/contrail. atproto backend in a bottle
flo-bit.dev/contrail/
add search, cleanup, add stuff to readme
+2011
-880
+2
-1
.claude/settings.local.json
+2
-1
.claude/settings.local.json
+3
-1
.gitignore
+3
-1
.gitignore
+74
-9
README.md
+74
-9
README.md
···
74
74
| `relations.*.match` | `"uri"` | Match against parent's `"uri"` or `"did"` |
75
75
| `relations.*.groupBy` | — | Split counts by this field's value |
76
76
| `queries` | `{}` | Custom query handlers |
77
+
| `searchable` | auto-detected | FTS5 search fields. `string[]` = explicit fields, `false` = disabled, omitted = all non-range queryable fields |
77
78
78
79
### Profiles
79
80
···
88
89
| `{collection}.listRecords` | List/filter records |
89
90
| `{collection}.getRecord` | Get single record by URI |
90
91
| `{namespace}.getProfile` | Get a user's profile by DID or handle |
92
+
| `{namespace}.notifyOfUpdate` | Notify of a record change for immediate indexing |
91
93
| `{namespace}.admin.sync` | Discover + backfill (requires `ADMIN_SECRET`) |
92
94
| `{namespace}.admin.getCursor` | Current cursor position |
93
95
| `{namespace}.admin.getOverview` | All collections summary |
···
101
103
|-------|---------|-------------|
102
104
| `actor` | `?actor=did:plc:...` or `?actor=alice.bsky.social` | Filter by DID or handle (triggers on-demand backfill) |
103
105
| `profiles` | `?profiles=true` | Include profile + identity info keyed by DID |
106
+
| `search` | `?search=meetup` | Full-text search across searchable fields (FTS5, ranked) |
104
107
| `{field}` | `?status=going` | Equality filter on queryable string field |
105
108
| `{field}Min` | `?startsAtMin=2026-03-16` | Range minimum (datetime/integer fields) |
106
109
| `{field}Max` | `?endsAtMax=2026-04-01` | Range maximum (datetime/integer fields) |
107
110
| `{rel}CountMin` | `?rsvpsCountMin=10` | Minimum total relation count |
108
111
| `{rel}{Group}CountMin` | `?rsvpsGoingCountMin=10` | Minimum relation count for a specific groupBy value |
109
-
| `hydrate` | `?hydrate=rsvps:10` | Embed latest N related records per record |
112
+
| `hydrate{Rel}` | `?hydrateRsvps=10` | Embed latest N related records (per group if grouped) |
113
+
| `hydrate{Ref}` | `?hydrateEvent=true` | Embed the referenced record |
114
+
| `sort` | `?sort=startsAt` | Sort by a queryable field or count (see below) |
115
+
| `order` | `?order=asc` | Sort direction: `asc` or `desc` (default depends on field type) |
110
116
| `limit` | `?limit=25` | Page size (1-100, default 50) |
111
117
| `cursor` | `?cursor=...` | Pagination cursor |
112
118
113
-
**Hydration** returns related records grouped by `groupBy` value:
119
+
**Sorting** — `sort` accepts any queryable field param name or a count field:
114
120
115
121
```
116
-
?hydrate=rsvps:5 # latest 5 per group (going, interested, etc.)
117
-
?hydrate=rsvps:5&hydrate=followers:10 # multiple hydrations
122
+
?sort=startsAt # by date (default: desc for range fields)
123
+
?sort=name&order=asc # by name ascending
124
+
?sort=rsvpsCount # by total RSVP count (default: desc)
125
+
?sort=rsvpsGoingCount&order=asc # by going count ascending
126
+
```
127
+
128
+
**Search** uses SQLite FTS5 for ranked full-text search. By default, all non-range queryable fields are searchable. Results are ranked by relevance (BM25) with `time_us` as tiebreaker. Supports FTS5 syntax including prefix (`meetup*`), phrases (`"rust meetup"`), and boolean (`rust OR typescript`). Combinable with all other filters.
129
+
130
+
```
131
+
?search=meetup # basic search
132
+
?search=meetup&mode=online # search + filter
133
+
?search=rust*&sort=startsAt&order=asc # search + sort override
134
+
```
135
+
136
+
**Hydration** embeds related or referenced records inline:
137
+
138
+
```
139
+
?hydrateRsvps=5 # latest 5 RSVPs per group (going, interested, etc.)
140
+
?hydrateEvent=true # embed the referenced event record
141
+
?hydrateRsvps=5&hydrateEvent=true # combine both
118
142
```
119
143
120
144
### Examples (events)
121
145
122
146
```
123
147
# Upcoming events with 10+ going RSVPs, with RSVP records and profiles
124
-
/xrpc/community.lexicon.calendar.event.getRecords?startsAtMin=2026-03-16&rsvpsGoingCountMin=10&hydrate=rsvps:5&profiles=true
148
+
/xrpc/community.lexicon.calendar.event.listRecords?startsAtMin=2026-03-16&rsvpsGoingCountMin=10&hydrateRsvps=5&profiles=true
125
149
126
150
# Events for a specific user (by handle)
127
-
/xrpc/community.lexicon.calendar.event.getRecords?actor=alice.bsky.social&profiles=true
151
+
/xrpc/community.lexicon.calendar.event.listRecords?actor=alice.bsky.social&profiles=true
128
152
129
153
# Single event with counts, RSVPs, and profiles
130
-
/xrpc/community.lexicon.calendar.event.getRecord?uri=at://did:plc:.../community.lexicon.calendar.event/...&hydrate=rsvps:10&profiles=true
154
+
/xrpc/community.lexicon.calendar.event.getRecord?uri=at://did:plc:.../community.lexicon.calendar.event/...&hydrateRsvps=10&profiles=true
155
+
156
+
# Search for events by name/description
157
+
/xrpc/community.lexicon.calendar.event.listRecords?search=meetup&profiles=true
158
+
159
+
# RSVPs for a specific event, with the referenced event embedded
160
+
/xrpc/community.lexicon.calendar.rsvp.listRecords?subjectUri=at://did:plc:.../community.lexicon.calendar.event/...&hydrateEvent=true&profiles=true
161
+
```
162
+
163
+
## Notify of Updates
164
+
165
+
By default, Contrail ingests from Jetstream every minute. If your app writes to a user's PDS and needs the change reflected immediately, call `notifyOfUpdate` right after the write:
166
+
167
+
```ts
168
+
// User creates an RSVP via their PDS
169
+
const { uri } = await agent.createRecord({ ... });
170
+
171
+
// Tell Contrail to fetch and index it now
172
+
await fetch("https://your-contrail.workers.dev/xrpc/com.example.notifyOfUpdate", {
173
+
method: "POST",
174
+
headers: { "Content-Type": "application/json" },
175
+
body: JSON.stringify({ uri }),
176
+
});
177
+
```
178
+
179
+
Contrail fetches the record from the user's PDS and figures out what to do:
180
+
181
+
| PDS returns | Already indexed? | Action |
182
+
|---|---|---|
183
+
| Record (new CID) | No | **Create** — indexes it, updates relation counts |
184
+
| Record (new CID) | Yes | **Update** — upserts the record |
185
+
| Record (same CID) | Yes | **Skip** — nothing changed |
186
+
| 404 | Yes | **Delete** — removes it, decrements counts |
187
+
| 404 | No | **No-op** |
131
188
132
-
# RSVPs for a specific event with profiles
133
-
/xrpc/community.lexicon.calendar.rsvp.getRecords?subjectUri=at://did:plc:.../community.lexicon.calendar.event/...&profiles=true
189
+
You can also batch up to 25 URIs in one request:
190
+
191
+
```ts
192
+
await fetch(".../xrpc/com.example.notifyOfUpdate", {
193
+
method: "POST",
194
+
headers: { "Content-Type": "application/json" },
195
+
body: JSON.stringify({ uris: [uri1, uri2, uri3] }),
196
+
});
134
197
```
198
+
199
+
When Jetstream later delivers the same event, the duplicate is detected by CID and skipped.
135
200
136
201
## Typesafe Client Usage
137
202
+4
lexicons-generated/community/lexicon/calendar/event/listRecords.json
+4
lexicons-generated/community/lexicon/calendar/event/listRecords.json
···
26
26
"type": "boolean",
27
27
"description": "Include profile + identity info keyed by DID"
28
28
},
29
+
"search": {
30
+
"type": "string",
31
+
"description": "Full-text search across: mode, name, status, description"
32
+
},
29
33
"mode": {
30
34
"type": "string",
31
35
"description": "Filter by mode"
+4
lexicons-generated/community/lexicon/calendar/rsvp/listRecords.json
+4
lexicons-generated/community/lexicon/calendar/rsvp/listRecords.json
···
26
26
"type": "boolean",
27
27
"description": "Include profile + identity info keyed by DID"
28
28
},
29
+
"search": {
30
+
"type": "string",
31
+
"description": "Full-text search across: status, subject.uri"
32
+
},
29
33
"status": {
30
34
"type": "string",
31
35
"description": "Filter by status"
+59
lexicons-generated/rsvp/atmo/notifyOfUpdate.json
+59
lexicons-generated/rsvp/atmo/notifyOfUpdate.json
···
1
+
{
2
+
"lexicon": 1,
3
+
"id": "rsvp.atmo.notifyOfUpdate",
4
+
"defs": {
5
+
"main": {
6
+
"type": "procedure",
7
+
"description": "Notify of a record change for immediate indexing. Fetches the record from the user's PDS and indexes (or deletes) it.",
8
+
"input": {
9
+
"encoding": "application/json",
10
+
"schema": {
11
+
"type": "object",
12
+
"properties": {
13
+
"uri": {
14
+
"type": "string",
15
+
"format": "at-uri",
16
+
"description": "Single AT URI to fetch and index"
17
+
},
18
+
"uris": {
19
+
"type": "array",
20
+
"items": {
21
+
"type": "string",
22
+
"format": "at-uri"
23
+
},
24
+
"maxLength": 25,
25
+
"description": "Batch of AT URIs to fetch and index (max 25)"
26
+
}
27
+
}
28
+
}
29
+
},
30
+
"output": {
31
+
"encoding": "application/json",
32
+
"schema": {
33
+
"type": "object",
34
+
"required": [
35
+
"indexed",
36
+
"deleted"
37
+
],
38
+
"properties": {
39
+
"indexed": {
40
+
"type": "integer",
41
+
"description": "Number of records created or updated"
42
+
},
43
+
"deleted": {
44
+
"type": "integer",
45
+
"description": "Number of records deleted (not found on PDS)"
46
+
},
47
+
"errors": {
48
+
"type": "array",
49
+
"items": {
50
+
"type": "string"
51
+
},
52
+
"description": "Errors for individual URIs that could not be processed"
53
+
}
54
+
}
55
+
}
56
+
}
57
+
}
58
+
}
59
+
}
+7
-866
scripts/generate-lexicons.ts
+7
-866
scripts/generate-lexicons.ts
···
1
1
/**
2
-
* Generates lexicon TypeScript files from the Contrail config.
3
-
*
4
-
* For each collection, generates:
5
-
* - {nsid}.listRecords — query with queryable field params
6
-
* - {nsid}.getUsers — query with limit/cursor
7
-
* - {nsid}.getStats — query returning collection stats
8
-
*
9
-
* Plus namespaced endpoints:
10
-
* - {namespace}.admin.getCursor
11
-
* - {namespace}.admin.getOverview
12
-
* - {namespace}.admin.sync
13
-
* - {namespace}.admin.reset
14
-
* - {namespace}.getProfile
2
+
* Generates lexicon files, lex.config.js, and queryable.generated.ts from config.
15
3
*
16
4
* Usage: npx tsx scripts/generate-lexicons.ts
17
5
*/
18
6
19
-
import { writeFileSync, mkdirSync, rmSync, existsSync, readFileSync, readdirSync } from "fs";
20
7
import { join } from "path";
21
8
import { config } from "../src/config";
9
+
import { generateLexicons } from "../src/generate";
22
10
23
11
const ROOT_DIR = join(__dirname, "..");
24
-
const USER_LEXICONS_DIR = join(ROOT_DIR, "lexicons");
25
-
const PULLED_LEXICONS_DIR = join(ROOT_DIR, "lexicons-pulled");
26
-
const GENERATED_DIR = join(ROOT_DIR, "lexicons-generated");
27
12
28
-
function fieldToParam(field: string): string {
29
-
return field.replace(/\.(\w)/g, (_, c) => c.toUpperCase());
30
-
}
31
-
32
-
interface QueryableField {
33
-
type?: "range";
34
-
}
35
-
36
-
// Find a collection's lexicon file (user-provided takes priority over pulled)
37
-
function findCollectionLexicon(collection: string): string | null {
38
-
const segments = collection.split(".");
39
-
for (const dir of [USER_LEXICONS_DIR, PULLED_LEXICONS_DIR]) {
40
-
const filePath = join(dir, ...segments) + ".json";
41
-
if (existsSync(filePath)) return filePath;
42
-
}
43
-
return null;
44
-
}
45
-
46
-
// Analyze a collection's lexicon and return auto-detected queryable fields
47
-
function detectQueryableFields(collection: string): Record<string, QueryableField> {
48
-
const filePath = findCollectionLexicon(collection);
49
-
if (!filePath) return {};
50
-
try {
51
-
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
52
-
const mainRecord = doc.defs?.main?.record;
53
-
if (!mainRecord?.properties) return {};
54
-
return analyzeProperties(doc.defs, mainRecord.properties, "");
55
-
} catch {
56
-
return {};
57
-
}
58
-
}
59
-
60
-
function analyzeProperties(
61
-
defs: Record<string, any>,
62
-
properties: Record<string, any>,
63
-
prefix: string
64
-
): Record<string, QueryableField> {
65
-
const result: Record<string, QueryableField> = {};
66
-
67
-
for (const [field, def] of Object.entries(properties)) {
68
-
const path = prefix ? `${prefix}.${field}` : field;
69
-
70
-
if (def.type === "string") {
71
-
if (def.format === "datetime") {
72
-
result[path] = { type: "range" };
73
-
} else if (def.format !== "uri" && def.format !== "at-uri") {
74
-
// Regular strings (enums, free text) → equality
75
-
result[path] = {};
76
-
}
77
-
} else if (def.type === "integer" || def.type === "number") {
78
-
result[path] = { type: "range" };
79
-
} else if (def.type === "ref" && def.ref === "com.atproto.repo.strongRef") {
80
-
result[`${path}.uri`] = {};
81
-
} else if (def.type === "union" && Array.isArray(def.refs) && def.refs.includes("com.atproto.repo.strongRef")) {
82
-
result[`${path}.uri`] = {};
83
-
} else if (def.type === "ref" && def.ref) {
84
-
// Resolve local ref (e.g. #mode → defs.mode)
85
-
const refId = def.ref.includes("#") ? def.ref.split("#")[1] : null;
86
-
if (refId && defs[refId]) {
87
-
const resolved = defs[refId];
88
-
if (resolved.type === "string") {
89
-
// String enum (knownValues) → equality
90
-
result[path] = {};
91
-
}
92
-
}
93
-
}
94
-
}
95
-
96
-
return result;
97
-
}
98
-
99
-
// Extract knownValues for a field from a collection's lexicon
100
-
function getKnownValues(collection: string, fieldName: string): string[] {
101
-
const filePath = findCollectionLexicon(collection);
102
-
if (!filePath) return [];
103
-
try {
104
-
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
105
-
const props = doc.defs?.main?.record?.properties;
106
-
if (!props) return [];
107
-
const field = props[fieldName];
108
-
if (!field) return [];
109
-
if (Array.isArray(field.knownValues)) return field.knownValues;
110
-
return [];
111
-
} catch {
112
-
return [];
113
-
}
114
-
}
115
-
116
-
// Default mapping: "community.lexicon.calendar.rsvp#going" → "going"
117
-
function tokenShortName(token: string): string {
118
-
const hash = token.indexOf("#");
119
-
return hash !== -1 ? token.slice(hash + 1) : token;
120
-
}
121
-
122
-
// Clean generated dir (user-provided lexicons/ is untouched)
123
-
rmSync(GENERATED_DIR, { recursive: true, force: true });
124
-
125
-
function nsidToPath(nsid: string): string {
126
-
return join(GENERATED_DIR, ...nsid.split(".")) + ".json";
127
-
}
128
-
129
-
// Check if a collection lexicon exists (user-provided or pulled)
130
-
function getCollectionLexiconRef(collection: string): string | null {
131
-
const filePath = findCollectionLexicon(collection);
132
-
if (!filePath) return null;
133
-
try {
134
-
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
135
-
if (doc.defs?.main) return `${collection}#main`;
136
-
} catch {}
137
-
return null;
138
-
}
139
-
140
-
function ensureDir(filePath: string) {
141
-
mkdirSync(join(filePath, ".."), { recursive: true });
142
-
}
143
-
144
-
function writeLexicon(nsid: string, doc: object) {
145
-
const filePath = nsidToPath(nsid);
146
-
ensureDir(filePath);
147
-
writeFileSync(filePath, JSON.stringify(doc, null, 2) + "\n");
148
-
console.log(` ${nsid}`);
149
-
}
150
-
151
-
// Build record output shape, optionally typing the record field
152
-
// countFields: e.g. [{ name: "rsvpsTotal", description: "..." }, { name: "rsvpsGoing", ... }]
153
-
interface CountField {
154
-
name: string;
155
-
description: string;
156
-
}
157
-
158
-
interface RelationDef {
159
-
relName: string;
160
-
collection: string;
161
-
groupBy?: string;
162
-
groups: Record<string, string>; // shortName → full token
163
-
}
164
-
165
-
interface ReferenceDef {
166
-
refName: string;
167
-
collection: string;
168
-
}
169
-
170
-
function buildRecordDef(
171
-
collectionRef: string | null,
172
-
countFields?: CountField[],
173
-
relationDefs?: RelationDef[],
174
-
referenceDefs?: ReferenceDef[]
175
-
) {
176
-
const properties: Record<string, any> = {
177
-
uri: { type: "string", format: "at-uri" },
178
-
did: { type: "string", format: "did" },
179
-
collection: { type: "string", format: "nsid" },
180
-
rkey: { type: "string" },
181
-
cid: { type: "string" },
182
-
record: collectionRef
183
-
? { type: "ref", ref: collectionRef }
184
-
: { type: "unknown" },
185
-
time_us: { type: "integer" },
186
-
};
187
-
188
-
if (countFields) {
189
-
for (const cf of countFields) {
190
-
properties[cf.name] = { type: "integer", description: cf.description };
191
-
}
192
-
}
193
-
194
-
if (relationDefs && relationDefs.length > 0) {
195
-
for (const rd of relationDefs) {
196
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
197
-
if (rd.groupBy && Object.keys(rd.groups).length > 0) {
198
-
properties[rd.relName] = {
199
-
type: "ref",
200
-
ref: `#hydrate${capitalize(rd.relName)}`,
201
-
};
202
-
} else {
203
-
properties[rd.relName] = {
204
-
type: "array",
205
-
items: { type: "ref", ref: `#hydrate${capitalize(rd.relName)}Record` },
206
-
};
207
-
}
208
-
}
209
-
}
210
-
211
-
if (referenceDefs && referenceDefs.length > 0) {
212
-
for (const rd of referenceDefs) {
213
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
214
-
properties[rd.refName] = {
215
-
type: "ref",
216
-
ref: `#ref${capitalize(rd.refName)}Record`,
217
-
};
218
-
}
219
-
}
220
-
221
-
return {
222
-
type: "object",
223
-
required: ["uri", "did", "collection", "rkey", "time_us"],
224
-
properties,
225
-
};
226
-
}
227
-
228
-
function buildReferenceDefs(referenceDefs: ReferenceDef[]): Record<string, any> {
229
-
const defs: Record<string, any> = {};
230
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
231
-
232
-
for (const rd of referenceDefs) {
233
-
const refCollectionRef = getCollectionLexiconRef(rd.collection);
234
-
const recordDefName = `ref${capitalize(rd.refName)}Record`;
235
-
defs[recordDefName] = {
236
-
type: "object",
237
-
required: ["uri", "did", "collection", "rkey", "time_us"],
238
-
properties: {
239
-
uri: { type: "string", format: "at-uri" },
240
-
did: { type: "string", format: "did" },
241
-
collection: { type: "string", format: "nsid" },
242
-
rkey: { type: "string" },
243
-
cid: { type: "string" },
244
-
record: refCollectionRef
245
-
? { type: "ref", ref: refCollectionRef }
246
-
: { type: "unknown" },
247
-
time_us: { type: "integer" },
248
-
},
249
-
};
250
-
}
251
-
252
-
return defs;
253
-
}
254
-
255
-
function buildHydrateDefs(relationDefs: RelationDef[]): Record<string, any> {
256
-
const defs: Record<string, any> = {};
257
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
258
-
259
-
for (const rd of relationDefs) {
260
-
const relCollectionRef = getCollectionLexiconRef(rd.collection);
261
-
262
-
// Def for each hydrated record
263
-
const recordDefName = `hydrate${capitalize(rd.relName)}Record`;
264
-
defs[recordDefName] = {
265
-
type: "object",
266
-
required: ["uri", "did", "collection", "rkey", "time_us"],
267
-
properties: {
268
-
uri: { type: "string", format: "at-uri" },
269
-
did: { type: "string", format: "did" },
270
-
collection: { type: "string", format: "nsid" },
271
-
rkey: { type: "string" },
272
-
cid: { type: "string" },
273
-
record: relCollectionRef
274
-
? { type: "ref", ref: relCollectionRef }
275
-
: { type: "unknown" },
276
-
time_us: { type: "integer" },
277
-
},
278
-
};
279
-
280
-
if (rd.groupBy && Object.keys(rd.groups).length > 0) {
281
-
// Grouped relation: object with group keys
282
-
const groupDefName = `hydrate${capitalize(rd.relName)}`;
283
-
const groupProperties: Record<string, any> = {};
284
-
for (const shortName of Object.keys(rd.groups)) {
285
-
groupProperties[shortName] = {
286
-
type: "array",
287
-
items: { type: "ref", ref: `#${recordDefName}` },
288
-
};
289
-
}
290
-
groupProperties["other"] = {
291
-
type: "array",
292
-
items: { type: "ref", ref: `#${recordDefName}` },
293
-
};
294
-
defs[groupDefName] = {
295
-
type: "object",
296
-
properties: groupProperties,
297
-
};
298
-
}
299
-
// Ungrouped relations are typed directly as arrays on the record (no wrapper def needed)
300
-
}
301
-
302
-
return defs;
303
-
}
304
-
305
-
// Read the inner record object schema from a collection's lexicon
306
-
function getRecordObjectSchema(collection: string): any | null {
307
-
const filePath = findCollectionLexicon(collection);
308
-
if (!filePath) return null;
309
-
try {
310
-
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
311
-
const main = doc.defs?.main;
312
-
if (main?.type === "record" && main.record) return main.record;
313
-
return null;
314
-
} catch {
315
-
return null;
316
-
}
317
-
}
318
-
319
-
function profileDefs() {
320
-
const profiles = config.profiles ?? ["app.bsky.actor.profile"];
321
-
const extraDefs: Record<string, any> = {};
322
-
const objectRefs: string[] = [];
323
-
324
-
for (const col of profiles) {
325
-
const schema = getRecordObjectSchema(col);
326
-
if (!schema) continue;
327
-
// Create a local def name from the NSID, e.g. "app.bsky.actor.profile" → "appBskyActorProfile"
328
-
const defName = col.split(".").map((p, i) => i === 0 ? p : p.charAt(0).toUpperCase() + p.slice(1)).join("");
329
-
extraDefs[defName] = schema;
330
-
objectRefs.push(`#${defName}`);
331
-
}
332
-
333
-
let recordField: any;
334
-
if (objectRefs.length === 1) {
335
-
recordField = { type: "ref", ref: objectRefs[0] };
336
-
} else if (objectRefs.length > 1) {
337
-
recordField = { type: "union", refs: objectRefs };
338
-
} else {
339
-
recordField = { type: "unknown" };
340
-
}
341
-
342
-
return {
343
-
profileEntry: {
344
-
type: "object",
345
-
required: ["did"],
346
-
properties: {
347
-
did: { type: "string", format: "did" },
348
-
handle: { type: "string" },
349
-
uri: { type: "string", format: "at-uri" },
350
-
collection: { type: "string", format: "nsid" },
351
-
rkey: { type: "string" },
352
-
cid: { type: "string" },
353
-
record: recordField,
354
-
},
355
-
},
356
-
...extraDefs,
357
-
};
358
-
}
359
-
360
-
// --- Namespace ---
361
-
362
-
const ns = config.namespace!;
363
-
364
-
// --- Admin endpoints ---
365
-
366
-
console.log("Generating admin endpoints...");
367
-
368
-
writeLexicon(`${ns}.admin.getCursor`, {
369
-
lexicon: 1,
370
-
id: `${ns}.admin.getCursor`,
371
-
defs: {
372
-
main: {
373
-
type: "query",
374
-
description: "Get the current cursor position",
375
-
output: {
376
-
encoding: "application/json",
377
-
schema: {
378
-
type: "object",
379
-
properties: {
380
-
time_us: { type: "integer" },
381
-
date: { type: "string" },
382
-
seconds_ago: { type: "integer" },
383
-
},
384
-
},
385
-
},
386
-
},
387
-
},
13
+
generateLexicons({
14
+
config,
15
+
rootDir: ROOT_DIR,
16
+
outputDir: join(ROOT_DIR, "lexicons-generated"),
17
+
writeRuntimeFiles: true,
388
18
});
389
-
390
-
writeLexicon(`${ns}.admin.getOverview`, {
391
-
lexicon: 1,
392
-
id: `${ns}.admin.getOverview`,
393
-
defs: {
394
-
main: {
395
-
type: "query",
396
-
description: "Get an overview of all indexed collections",
397
-
output: {
398
-
encoding: "application/json",
399
-
schema: {
400
-
type: "object",
401
-
required: ["total_records", "collections"],
402
-
properties: {
403
-
total_records: { type: "integer" },
404
-
collections: {
405
-
type: "array",
406
-
items: { type: "ref", ref: "#collectionStats" },
407
-
},
408
-
},
409
-
},
410
-
},
411
-
},
412
-
collectionStats: {
413
-
type: "object",
414
-
required: ["collection", "records", "unique_users"],
415
-
properties: {
416
-
collection: { type: "string" },
417
-
records: { type: "integer" },
418
-
unique_users: { type: "integer" },
419
-
},
420
-
},
421
-
},
422
-
});
423
-
424
-
writeLexicon(`${ns}.admin.sync`, {
425
-
lexicon: 1,
426
-
id: `${ns}.admin.sync`,
427
-
defs: {
428
-
main: {
429
-
type: "query",
430
-
description: "Discover users from relays and backfill their records from PDS",
431
-
parameters: {
432
-
type: "params",
433
-
properties: {
434
-
concurrency: {
435
-
type: "integer",
436
-
minimum: 1,
437
-
maximum: 50,
438
-
default: 10,
439
-
},
440
-
},
441
-
},
442
-
output: {
443
-
encoding: "application/json",
444
-
schema: {
445
-
type: "object",
446
-
required: ["discovered", "backfilled", "remaining", "done"],
447
-
properties: {
448
-
discovered: { type: "integer" },
449
-
backfilled: { type: "integer" },
450
-
remaining: { type: "integer" },
451
-
done: { type: "boolean" },
452
-
},
453
-
},
454
-
},
455
-
},
456
-
},
457
-
});
458
-
459
-
writeLexicon(`${ns}.admin.reset`, {
460
-
lexicon: 1,
461
-
id: `${ns}.admin.reset`,
462
-
defs: {
463
-
main: {
464
-
type: "query",
465
-
description: "Delete all data from all tables",
466
-
output: {
467
-
encoding: "application/json",
468
-
schema: {
469
-
type: "object",
470
-
required: ["ok"],
471
-
properties: {
472
-
ok: { type: "boolean" },
473
-
},
474
-
},
475
-
},
476
-
},
477
-
},
478
-
});
479
-
480
-
// --- getProfile endpoint ---
481
-
482
-
writeLexicon(`${ns}.getProfile`, {
483
-
lexicon: 1,
484
-
id: `${ns}.getProfile`,
485
-
defs: {
486
-
main: {
487
-
type: "query",
488
-
description: "Get a user's profile by DID or handle",
489
-
parameters: {
490
-
type: "params",
491
-
required: ["actor"],
492
-
properties: {
493
-
actor: {
494
-
type: "string",
495
-
format: "at-identifier",
496
-
description: "DID or handle of the user",
497
-
},
498
-
},
499
-
},
500
-
output: {
501
-
encoding: "application/json",
502
-
schema: {
503
-
type: "ref",
504
-
ref: "#profileEntry",
505
-
},
506
-
},
507
-
},
508
-
...profileDefs(),
509
-
},
510
-
});
511
-
512
-
// --- Per-collection endpoints ---
513
-
514
-
console.log("Generating collection endpoints...");
515
-
516
-
// Collect resolved queryable fields for all collections
517
-
const resolvedQueryable: Record<string, Record<string, { type?: "range" }>> = {};
518
-
// Collect resolved relation mappings (short name → full token) for runtime
519
-
const resolvedRelations: Record<string, Record<string, { collection: string; groupBy: string; groups: Record<string, string> }>> = {};
520
-
521
-
for (const [collection, colConfig] of Object.entries(config.collections)) {
522
-
const collectionRef = getCollectionLexiconRef(collection);
523
-
if (collectionRef) {
524
-
console.log(` → ${collection} record typed via lexicon`);
525
-
}
526
-
527
-
// Auto-detect queryable fields from lexicon, then merge manual overrides
528
-
const autoDetected = detectQueryableFields(collection);
529
-
const manual = colConfig.queryable ?? {};
530
-
const merged = { ...autoDetected, ...manual };
531
-
resolvedQueryable[collection] = merged;
532
-
533
-
if (Object.keys(autoDetected).length > 0) {
534
-
const autoOnly = Object.keys(autoDetected).filter((k) => !manual[k]);
535
-
if (autoOnly.length > 0) {
536
-
console.log(` → auto-detected queryable: ${autoOnly.join(", ")}`);
537
-
}
538
-
}
539
-
540
-
// --- listRecords ---
541
-
const listRecordsParamProps: Record<string, any> = {
542
-
limit: { type: "integer", minimum: 1, maximum: 100, default: 50 },
543
-
cursor: { type: "string" },
544
-
actor: { type: "string", format: "at-identifier", description: "Filter by DID or handle (triggers on-demand backfill)" },
545
-
profiles: { type: "boolean", description: "Include profile + identity info keyed by DID" },
546
-
};
547
-
548
-
for (const [field, fieldConfig] of Object.entries(merged)) {
549
-
const param = fieldToParam(field);
550
-
if (fieldConfig.type === "range") {
551
-
listRecordsParamProps[`${param}Min`] = {
552
-
type: "string",
553
-
description: `Minimum value for ${field}`,
554
-
};
555
-
listRecordsParamProps[`${param}Max`] = {
556
-
type: "string",
557
-
description: `Maximum value for ${field}`,
558
-
};
559
-
} else {
560
-
listRecordsParamProps[param] = {
561
-
type: "string",
562
-
description: `Filter by ${field}`,
563
-
};
564
-
}
565
-
}
566
-
567
-
// Build count fields, hydrate params, and relation defs from relations + knownValues
568
-
const countFields: CountField[] = [];
569
-
const relationDefs: RelationDef[] = [];
570
-
for (const [relName, rel] of Object.entries(colConfig.relations ?? {})) {
571
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
572
-
573
-
// Total count
574
-
countFields.push({ name: `${relName}Count`, description: `Total ${relName} count` });
575
-
listRecordsParamProps[`${relName}CountMin`] = {
576
-
type: "integer",
577
-
description: `Minimum total ${relName} count`,
578
-
};
579
-
580
-
// Per-relation hydrate param (e.g. hydrateRsvps=5)
581
-
listRecordsParamProps[`hydrate${capitalize(relName)}`] = {
582
-
type: "integer",
583
-
minimum: 1,
584
-
maximum: 50,
585
-
description: `Number of ${relName} records to embed per record`,
586
-
};
587
-
588
-
// Per-group counts from knownValues
589
-
const groupMapping: Record<string, string> = {};
590
-
if (rel.groupBy) {
591
-
const knownValues = getKnownValues(rel.collection, rel.groupBy);
592
-
for (const token of knownValues) {
593
-
const shortName = tokenShortName(token);
594
-
groupMapping[shortName] = token;
595
-
countFields.push({
596
-
name: `${relName}${capitalize(shortName)}Count`,
597
-
description: `${relName} count where ${rel.groupBy} = ${shortName}`,
598
-
});
599
-
listRecordsParamProps[`${relName}${capitalize(shortName)}CountMin`] = {
600
-
type: "integer",
601
-
description: `Minimum ${relName} count where ${rel.groupBy} = ${shortName}`,
602
-
};
603
-
}
604
-
// Store mapping for runtime use
605
-
if (!resolvedRelations[collection]) resolvedRelations[collection] = {};
606
-
resolvedRelations[collection][relName] = {
607
-
collection: rel.collection,
608
-
groupBy: rel.groupBy,
609
-
groups: groupMapping,
610
-
};
611
-
}
612
-
613
-
relationDefs.push({
614
-
relName,
615
-
collection: rel.collection,
616
-
groupBy: rel.groupBy,
617
-
groups: groupMapping,
618
-
});
619
-
}
620
-
621
-
// Build reference defs
622
-
const referenceDefs: ReferenceDef[] = [];
623
-
for (const [refName, ref] of Object.entries(colConfig.references ?? {})) {
624
-
referenceDefs.push({ refName, collection: ref.collection });
625
-
}
626
-
627
-
// Add per-reference hydrate params (e.g. hydrateEvent=true)
628
-
for (const refName of Object.keys(colConfig.references ?? {})) {
629
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
630
-
listRecordsParamProps[`hydrate${capitalize(refName)}`] = {
631
-
type: "boolean",
632
-
description: `Embed the referenced ${refName} record`,
633
-
};
634
-
}
635
-
636
-
// Build sortable field values: queryable fields + count fields
637
-
const sortableValues: string[] = [];
638
-
for (const field of Object.keys(merged)) {
639
-
sortableValues.push(fieldToParam(field));
640
-
}
641
-
for (const cf of countFields) {
642
-
sortableValues.push(cf.name);
643
-
}
644
-
645
-
if (sortableValues.length > 0) {
646
-
listRecordsParamProps["sort"] = {
647
-
type: "string",
648
-
knownValues: sortableValues,
649
-
description: "Field to sort by (default: time_us)",
650
-
};
651
-
listRecordsParamProps["order"] = {
652
-
type: "string",
653
-
knownValues: ["asc", "desc"],
654
-
description: "Sort direction (default: desc for dates/numbers/counts, asc for strings)",
655
-
};
656
-
}
657
-
658
-
const hydrateDefs = buildHydrateDefs(relationDefs);
659
-
const refDefs = buildReferenceDefs(referenceDefs);
660
-
661
-
writeLexicon(`${collection}.listRecords`, {
662
-
lexicon: 1,
663
-
id: `${collection}.listRecords`,
664
-
defs: {
665
-
main: {
666
-
type: "query",
667
-
description: `Query ${collection} records with filters`,
668
-
parameters: {
669
-
type: "params",
670
-
properties: listRecordsParamProps,
671
-
},
672
-
output: {
673
-
encoding: "application/json",
674
-
schema: {
675
-
type: "object",
676
-
required: ["records"],
677
-
properties: {
678
-
records: { type: "array", items: { type: "ref", ref: "#record" } },
679
-
cursor: { type: "string" },
680
-
profiles: { type: "array", items: { type: "ref", ref: "#profileEntry" } },
681
-
},
682
-
},
683
-
},
684
-
},
685
-
record: buildRecordDef(collectionRef, countFields, relationDefs, referenceDefs),
686
-
...hydrateDefs,
687
-
...refDefs,
688
-
...profileDefs(),
689
-
},
690
-
});
691
-
692
-
// --- getRecord ---
693
-
const getRecordParamProps: Record<string, any> = {
694
-
uri: { type: "string", format: "at-uri", description: "AT URI of the record" },
695
-
profiles: { type: "boolean", description: "Include profile + identity info keyed by DID" },
696
-
};
697
-
698
-
// Add per-relation hydrate params to getRecord too
699
-
for (const rd of relationDefs) {
700
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
701
-
getRecordParamProps[`hydrate${capitalize(rd.relName)}`] = {
702
-
type: "integer",
703
-
minimum: 1,
704
-
maximum: 50,
705
-
description: `Number of ${rd.relName} records to embed`,
706
-
};
707
-
}
708
-
709
-
// Add per-reference hydrate params to getRecord too
710
-
for (const refName of Object.keys(colConfig.references ?? {})) {
711
-
const capitalize = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
712
-
getRecordParamProps[`hydrate${capitalize(refName)}`] = {
713
-
type: "boolean",
714
-
description: `Embed the referenced ${refName} record`,
715
-
};
716
-
}
717
-
718
-
writeLexicon(`${collection}.getRecord`, {
719
-
lexicon: 1,
720
-
id: `${collection}.getRecord`,
721
-
defs: {
722
-
main: {
723
-
type: "query",
724
-
description: `Get a single ${collection} record by AT URI`,
725
-
parameters: {
726
-
type: "params",
727
-
required: ["uri"],
728
-
properties: getRecordParamProps,
729
-
},
730
-
output: {
731
-
encoding: "application/json",
732
-
schema: {
733
-
type: "object",
734
-
required: ["uri", "did", "collection", "rkey", "time_us"],
735
-
properties: {
736
-
...buildRecordDef(collectionRef, countFields, relationDefs, referenceDefs).properties,
737
-
profiles: { type: "array", items: { type: "ref", ref: "#profileEntry" } },
738
-
},
739
-
},
740
-
},
741
-
},
742
-
...hydrateDefs,
743
-
...refDefs,
744
-
...profileDefs(),
745
-
},
746
-
});
747
-
748
-
749
-
// --- Custom queries ---
750
-
for (const queryName of Object.keys(colConfig.queries ?? {})) {
751
-
writeLexicon(`${collection}.${queryName}`, {
752
-
lexicon: 1,
753
-
id: `${collection}.${queryName}`,
754
-
defs: {
755
-
main: {
756
-
type: "query",
757
-
description: `Custom query: ${queryName}`,
758
-
output: {
759
-
encoding: "application/json",
760
-
schema: { type: "object", properties: {} },
761
-
},
762
-
},
763
-
},
764
-
});
765
-
}
766
-
}
767
-
768
-
// --- Auto-generate lex.config.js ---
769
-
770
-
// Collect all collection NSIDs from config
771
-
const collectionNsids = Object.keys(config.collections);
772
-
773
-
// Scan pulled lexicons for external refs to find transitive deps
774
-
function findRefsInLexicon(filePath: string): string[] {
775
-
try {
776
-
const content = readFileSync(filePath, "utf-8");
777
-
const refs: string[] = [];
778
-
// Match all "ref": "some.nsid.here" or "refs": ["some.nsid.here"]
779
-
const refPattern = /"ref":\s*"([a-z][a-zA-Z0-9]*(?:\.[a-zA-Z0-9]+)+)(?:#\w+)?"/g;
780
-
let match;
781
-
while ((match = refPattern.exec(content)) !== null) {
782
-
refs.push(match[1]);
783
-
}
784
-
// Also match refs arrays
785
-
const refsArrayPattern = /"refs":\s*\[([^\]]+)\]/g;
786
-
while ((match = refsArrayPattern.exec(content)) !== null) {
787
-
const inner = match[1];
788
-
const nsidPattern = /"([a-z][a-zA-Z0-9]*(?:\.[a-zA-Z0-9]+)+)(?:#\w+)?"/g;
789
-
let innerMatch;
790
-
while ((innerMatch = nsidPattern.exec(inner)) !== null) {
791
-
refs.push(innerMatch[1]);
792
-
}
793
-
}
794
-
return refs;
795
-
} catch {
796
-
return [];
797
-
}
798
-
}
799
-
800
-
function scanLexiconsDir(dir: string): string[] {
801
-
const files: string[] = [];
802
-
if (!existsSync(dir)) return files;
803
-
for (const entry of readdirSync(dir, { withFileTypes: true })) {
804
-
const fullPath = join(dir, entry.name);
805
-
if (entry.isDirectory()) {
806
-
files.push(...scanLexiconsDir(fullPath));
807
-
} else if (entry.name.endsWith(".json")) {
808
-
files.push(fullPath);
809
-
}
810
-
}
811
-
return files;
812
-
}
813
-
814
-
// Find all NSIDs referenced by pulled lexicons
815
-
const pulledFiles = [
816
-
...scanLexiconsDir(USER_LEXICONS_DIR),
817
-
...scanLexiconsDir(PULLED_LEXICONS_DIR),
818
-
];
819
-
const allRefs = new Set<string>();
820
-
for (const file of pulledFiles) {
821
-
for (const ref of findRefsInLexicon(file)) {
822
-
allRefs.add(ref);
823
-
}
824
-
}
825
-
826
-
// Merge: collection NSIDs + profile NSIDs + transitive deps (excluding com.atproto.* which comes from imports)
827
-
const profileNsids = config.profiles ?? ["app.bsky.actor.profile"];
828
-
const pullNsids = new Set([...collectionNsids, ...profileNsids]);
829
-
for (const ref of allRefs) {
830
-
if (!ref.startsWith("com.atproto.")) {
831
-
pullNsids.add(ref);
832
-
}
833
-
}
834
-
835
-
const sortedNsids = [...pullNsids].sort();
836
-
837
-
const lexConfigContent = `import { defineLexiconConfig } from "@atcute/lex-cli";
838
-
839
-
export default defineLexiconConfig({
840
-
files: ["lexicons/**/*.json", "lexicons-pulled/**/*.json", "lexicons-generated/**/*.json"],
841
-
outdir: "src/lexicon-types/",
842
-
imports: ["@atcute/atproto"],
843
-
pull: {
844
-
outdir: "lexicons-pulled/",
845
-
sources: [
846
-
{
847
-
type: "atproto",
848
-
mode: "nsids",
849
-
nsids: ${JSON.stringify(sortedNsids, null, 10).replace(/^/gm, " ").trim()},
850
-
},
851
-
],
852
-
},
853
-
});
854
-
`;
855
-
856
-
writeFileSync(join(ROOT_DIR, "lex.config.js"), lexConfigContent);
857
-
console.log(`\nGenerated lex.config.js with ${sortedNsids.length} pull NSIDs`);
858
-
859
-
// Generate resolved queryable config for runtime use
860
-
const queryableContent = `// Auto-generated — do not edit. Run \`pnpm generate\` to regenerate.
861
-
import type { QueryableField } from "./types";
862
-
863
-
export const resolvedQueryable: Record<string, Record<string, QueryableField>> = ${JSON.stringify(resolvedQueryable, null, 2)};
864
-
865
-
export interface ResolvedRelation {
866
-
collection: string;
867
-
groupBy: string;
868
-
groups: Record<string, string>; // shortName → full token value
869
-
}
870
-
871
-
export const resolvedRelationsMap: Record<string, Record<string, ResolvedRelation>> = ${JSON.stringify(resolvedRelations, null, 2)};
872
-
`;
873
-
874
-
writeFileSync(join(ROOT_DIR, "src", "core", "queryable.generated.ts"), queryableContent);
875
-
console.log("Generated src/core/queryable.generated.ts");
876
-
877
-
console.log("\nDone!");
+58
-1
src/core/db/records.ts
+58
-1
src/core/db/records.ts
···
7
7
RecordRow,
8
8
} from "../types";
9
9
import { getNestedValue, getRelationField } from "../types";
10
+
import { getSearchableFields, ftsTableName, buildFtsContent } from "../search";
10
11
11
12
// --- Counts ---
12
13
···
71
72
return statements;
72
73
}
73
74
75
+
// --- FTS ---
76
+
77
+
function buildFtsStatements(
78
+
db: Database,
79
+
event: IngestEvent,
80
+
config: ContrailConfig
81
+
): Statement[] {
82
+
const colConfig = config.collections[event.collection];
83
+
if (!colConfig) return [];
84
+
85
+
const fields = getSearchableFields(event.collection, colConfig);
86
+
if (!fields || fields.length === 0) return [];
87
+
88
+
const table = ftsTableName(event.collection);
89
+
const stmts: Statement[] = [];
90
+
91
+
if (event.operation === "delete") {
92
+
stmts.push(db.prepare(`DELETE FROM ${table} WHERE uri = ?`).bind(event.uri));
93
+
} else {
94
+
const record = event.record ? JSON.parse(event.record) : null;
95
+
if (!record) return [];
96
+
97
+
const content = buildFtsContent(record, fields);
98
+
if (!content) return [];
99
+
100
+
// Delete-then-insert handles both create and update
101
+
stmts.push(db.prepare(`DELETE FROM ${table} WHERE uri = ?`).bind(event.uri));
102
+
stmts.push(
103
+
db.prepare(`INSERT INTO ${table} (uri, content) VALUES (?, ?)`).bind(event.uri, content)
104
+
);
105
+
}
106
+
107
+
return stmts;
108
+
}
109
+
74
110
// --- Cursor ---
75
111
76
112
export async function getLastCursor(db: Database): Promise<number | null> {
···
128
164
129
165
if (config) {
130
166
batch.push(...buildCountStatements(db, e, config));
167
+
batch.push(...buildFtsStatements(db, e, config));
131
168
}
132
169
}
133
170
···
151
188
rangeFilters?: Record<string, { min?: string; max?: string }>;
152
189
countFilters?: Record<string, number>;
153
190
sort?: SortOption;
191
+
search?: string;
154
192
}
155
193
156
194
export async function queryRecords(
···
167
205
rangeFilters = {},
168
206
countFilters = {},
169
207
sort,
208
+
search,
170
209
} = options;
171
210
172
211
const limit = Math.min(Math.max(1, rawLimit ?? 50), 100);
···
218
257
}
219
258
}
220
259
260
+
// FTS search
261
+
let ftsJoin = "";
262
+
if (search) {
263
+
const colConfig2 = config.collections[collection];
264
+
const fields = colConfig2 ? getSearchableFields(collection, colConfig2) : null;
265
+
if (fields && fields.length > 0) {
266
+
const table = ftsTableName(collection);
267
+
ftsJoin = `JOIN ${table} fts ON fts.uri = r.uri`;
268
+
conditions.push("fts.content MATCH ?");
269
+
bindings.push(search);
270
+
}
271
+
}
272
+
221
273
const colConfig = config.collections[collection];
222
274
const relations = colConfig?.relations ?? {};
223
275
const sortByCount = sort?.countType != null;
···
254
306
const select = needsCounts
255
307
? "r.uri, r.did, r.collection, r.rkey, r.cid, r.record, r.time_us, r.indexed_at, GROUP_CONCAT(c.type || ':' || c.count) as _counts"
256
308
: "r.uri, r.did, r.collection, r.rkey, r.cid, r.record, r.time_us, r.indexed_at";
257
-
const join = needsCounts ? "LEFT JOIN counts c ON c.uri = r.uri" : "";
309
+
const joinParts: string[] = [];
310
+
if (ftsJoin) joinParts.push(ftsJoin);
311
+
if (needsCounts) joinParts.push("LEFT JOIN counts c ON c.uri = r.uri");
312
+
const join = joinParts.join(" ");
258
313
const group = needsCounts ? "GROUP BY r.uri" : "";
259
314
const having = countHaving.length > 0 ? `HAVING ${countHaving.join(" AND ")}` : "";
260
315
···
267
322
const dir = sort.direction === "desc" ? "DESC" : "ASC";
268
323
orderBy = `COALESCE(SUM(CASE WHEN c.type = ? THEN c.count END), 0) ${dir}, r.time_us DESC`;
269
324
orderBindings.push(sort.countType);
325
+
} else if (ftsJoin) {
326
+
orderBy = "fts.rank, r.time_us DESC";
270
327
} else {
271
328
orderBy = "r.time_us DESC";
272
329
}
+16
-1
src/core/db/schema.ts
+16
-1
src/core/db/schema.ts
···
1
1
import type { ContrailConfig, Database } from "../types";
2
2
import { getRelationField } from "../types";
3
3
import { resolvedQueryable } from "../queryable.generated";
4
+
import { getSearchableFields, ftsTableName } from "../search";
4
5
5
6
const BASE_SCHEMA = `
6
7
CREATE TABLE IF NOT EXISTS records (
···
76
77
return indexes;
77
78
}
78
79
80
+
function buildFtsTables(config: ContrailConfig): string[] {
81
+
const stmts: string[] = [];
82
+
for (const [collection, colConfig] of Object.entries(config.collections)) {
83
+
const fields = getSearchableFields(collection, colConfig);
84
+
if (!fields || fields.length === 0) continue;
85
+
const table = ftsTableName(collection);
86
+
stmts.push(
87
+
`CREATE VIRTUAL TABLE IF NOT EXISTS ${table} USING fts5(uri UNINDEXED, content)`
88
+
);
89
+
}
90
+
return stmts;
91
+
}
92
+
79
93
const MIGRATIONS = [
80
94
"ALTER TABLE backfills ADD COLUMN retries INTEGER NOT NULL DEFAULT 0",
81
95
"ALTER TABLE backfills ADD COLUMN last_error TEXT",
···
100
114
.filter((s) => s.length > 0);
101
115
102
116
const indexStatements = buildDynamicIndexes(config);
103
-
const all = [...baseStatements, ...indexStatements];
117
+
const ftsStatements = buildFtsTables(config);
118
+
const all = [...baseStatements, ...indexStatements, ...ftsStatements];
104
119
105
120
await db.batch(all.map((s) => db.prepare(s)));
106
121
await runMigrations(db);
+3
src/core/router/collection.ts
+3
src/core/router/collection.ts
···
110
110
}
111
111
}
112
112
113
+
const search = params.get("search") || undefined;
114
+
113
115
const result = await queryRecords(db, config, {
114
116
collection,
115
117
did,
···
119
121
rangeFilters,
120
122
countFilters,
121
123
sort,
124
+
search,
122
125
});
123
126
124
127
const rows = result.records;
+2
src/core/router/index.ts
+2
src/core/router/index.ts
···
3
3
import type { Database, ContrailConfig } from "../types";
4
4
import { registerAdminRoutes } from "./admin";
5
5
import { registerCollectionRoutes } from "./collection";
6
+
import { registerNotifyRoute } from "./notify";
6
7
import { resolveActor } from "../identity";
7
8
import { resolveProfiles } from "./profiles";
8
9
import { backfillUser } from "../backfill";
···
43
44
44
45
registerAdminRoutes(app, db, config, adminSecret);
45
46
registerCollectionRoutes(app, db, config);
47
+
registerNotifyRoute(app, db, config);
46
48
47
49
return app;
48
50
}
+149
src/core/router/notify.ts
+149
src/core/router/notify.ts
···
1
+
import type { Hono } from "hono";
2
+
import type { Database, ContrailConfig, IngestEvent } from "../types";
3
+
import { applyEvents } from "../db/records";
4
+
import { getPDS } from "../client";
5
+
import type { Did } from "@atcute/lexicons";
6
+
7
+
/** Parse an AT URI into its components. */
8
+
export function parseAtUri(uri: string): { did: string; collection: string; rkey: string } | null {
9
+
const match = uri.match(/^at:\/\/(did:[^/]+)\/([^/]+)\/([^/]+)$/);
10
+
if (!match) return null;
11
+
return { did: match[1], collection: match[2], rkey: match[3] };
12
+
}
13
+
14
+
/**
15
+
* Fetch a single record from the user's PDS.
16
+
* Returns the record + cid on success, null if not found.
17
+
*/
18
+
async function fetchRecordFromPDS(
19
+
pds: string,
20
+
did: string,
21
+
collection: string,
22
+
rkey: string
23
+
): Promise<{ value: unknown; cid: string } | null> {
24
+
const url = new URL(`/xrpc/com.atproto.repo.getRecord`, pds);
25
+
url.searchParams.set("repo", did);
26
+
url.searchParams.set("collection", collection);
27
+
url.searchParams.set("rkey", rkey);
28
+
29
+
const res = await fetch(url.toString());
30
+
if (!res.ok) return null;
31
+
32
+
const data = (await res.json()) as { value?: unknown; cid?: string };
33
+
if (!data.value || !data.cid) return null;
34
+
return { value: data.value, cid: data.cid };
35
+
}
36
+
37
+
export function registerNotifyRoute(
38
+
app: Hono,
39
+
db: Database,
40
+
config: ContrailConfig
41
+
) {
42
+
const ns = config.namespace;
43
+
44
+
app.post(`/xrpc/${ns}.notifyOfUpdate`, async (c) => {
45
+
const body = await c.req.json<{ uri?: string; uris?: string[] }>().catch(() => null);
46
+
const uris: string[] = [];
47
+
48
+
if (body?.uris && Array.isArray(body.uris)) {
49
+
uris.push(...body.uris);
50
+
} else if (body?.uri) {
51
+
uris.push(body.uri);
52
+
} else {
53
+
return c.json({ error: "uri or uris required" }, 400);
54
+
}
55
+
56
+
if (uris.length > 25) {
57
+
return c.json({ error: "max 25 URIs per request" }, 400);
58
+
}
59
+
60
+
const events: IngestEvent[] = [];
61
+
const errors: string[] = [];
62
+
63
+
for (const uri of uris) {
64
+
const parsed = parseAtUri(uri);
65
+
if (!parsed) {
66
+
errors.push(`invalid AT URI: ${uri}`);
67
+
continue;
68
+
}
69
+
70
+
// Only accept collections we're tracking
71
+
if (!config.collections[parsed.collection]) {
72
+
errors.push(`collection not tracked: ${parsed.collection}`);
73
+
continue;
74
+
}
75
+
76
+
const pds = await getPDS(parsed.did as Did, db);
77
+
if (!pds) {
78
+
errors.push(`could not resolve PDS for ${parsed.did}`);
79
+
continue;
80
+
}
81
+
82
+
const result = await fetchRecordFromPDS(
83
+
pds,
84
+
parsed.did,
85
+
parsed.collection,
86
+
parsed.rkey
87
+
);
88
+
89
+
const now = Date.now() * 1000; // microseconds
90
+
91
+
// Check if this record already exists locally
92
+
const existing = await db
93
+
.prepare("SELECT cid FROM records WHERE uri = ?")
94
+
.bind(uri)
95
+
.first<{ cid: string | null }>();
96
+
97
+
if (result) {
98
+
if (existing?.cid === result.cid) {
99
+
// Same CID — nothing changed, skip to avoid double-counting
100
+
continue;
101
+
}
102
+
103
+
events.push({
104
+
uri,
105
+
did: parsed.did,
106
+
collection: parsed.collection,
107
+
rkey: parsed.rkey,
108
+
// "update" skips count statements, "create" increments them.
109
+
// Only use "create" if the record is genuinely new.
110
+
operation: existing ? "update" : "create",
111
+
cid: result.cid,
112
+
record: JSON.stringify(result.value),
113
+
time_us: now,
114
+
indexed_at: now,
115
+
});
116
+
} else if (existing) {
117
+
// Record gone from PDS but exists locally — delete it.
118
+
// We need the old record data so buildCountStatements can decrement counts.
119
+
const existingRecord = await db
120
+
.prepare("SELECT record FROM records WHERE uri = ?")
121
+
.bind(uri)
122
+
.first<{ record: string | null }>();
123
+
124
+
events.push({
125
+
uri,
126
+
did: parsed.did,
127
+
collection: parsed.collection,
128
+
rkey: parsed.rkey,
129
+
operation: "delete",
130
+
cid: null,
131
+
record: existingRecord?.record ?? null,
132
+
time_us: now,
133
+
indexed_at: now,
134
+
});
135
+
}
136
+
// If not on PDS and not local, nothing to do
137
+
}
138
+
139
+
if (events.length > 0) {
140
+
await applyEvents(db, events, config);
141
+
}
142
+
143
+
return c.json({
144
+
indexed: events.filter((e) => e.operation === "create" || e.operation === "update").length,
145
+
deleted: events.filter((e) => e.operation === "delete").length,
146
+
errors: errors.length > 0 ? errors : undefined,
147
+
});
148
+
});
149
+
}
+40
src/core/search.ts
+40
src/core/search.ts
···
1
+
import type { CollectionConfig } from "./types";
2
+
import { getNestedValue } from "./types";
3
+
import { resolvedQueryable } from "./queryable.generated";
4
+
5
+
/**
6
+
* Resolve which fields are searchable for a collection.
7
+
* Returns null if search is disabled or no fields found.
8
+
*/
9
+
export function getSearchableFields(
10
+
collection: string,
11
+
colConfig: CollectionConfig
12
+
): string[] | null {
13
+
if (colConfig.searchable === false) return null;
14
+
if (Array.isArray(colConfig.searchable)) {
15
+
return colConfig.searchable.length > 0 ? colConfig.searchable : null;
16
+
}
17
+
// Auto-detect: all non-range queryable fields
18
+
const queryable = resolvedQueryable[collection] ?? colConfig.queryable ?? {};
19
+
const fields = Object.entries(queryable)
20
+
.filter(([, f]) => f.type !== "range")
21
+
.map(([name]) => name);
22
+
return fields.length > 0 ? fields : null;
23
+
}
24
+
25
+
/** Sanitized FTS table name for a collection. */
26
+
export function ftsTableName(collection: string): string {
27
+
return `fts_${collection.replace(/[^a-zA-Z0-9]/g, "_")}`;
28
+
}
29
+
30
+
/** Extract searchable field values from a record and join them into a single string. */
31
+
export function buildFtsContent(record: unknown, fields: string[]): string | null {
32
+
const parts: string[] = [];
33
+
for (const field of fields) {
34
+
const value = getNestedValue(record, field);
35
+
if (typeof value === "string" && value.length > 0) {
36
+
parts.push(value);
37
+
}
38
+
}
39
+
return parts.length > 0 ? parts.join(" ") : null;
40
+
}
+7
src/core/types.ts
+7
src/core/types.ts
···
44
44
/** Forward references: fields on this collection's records that point at another collection. */
45
45
references?: Record<string, ReferenceConfig>;
46
46
queries?: Record<string, CustomQueryHandler>;
47
+
/** FTS5 search fields. string[] = explicit fields, false = disabled, omitted = auto-detect non-range queryable fields */
48
+
searchable?: string[] | false;
47
49
}
48
50
49
51
export const DEFAULT_PROFILES = ["app.bsky.actor.profile"];
···
132
134
for (const [, rel] of Object.entries(colConfig.relations ?? {})) {
133
135
if (rel.field) validateFieldName(rel.field);
134
136
if (rel.groupBy) validateFieldName(rel.groupBy);
137
+
}
138
+
if (Array.isArray(colConfig.searchable)) {
139
+
for (const field of colConfig.searchable) {
140
+
validateFieldName(field);
141
+
}
135
142
}
136
143
}
137
144
}
+525
src/generate.ts
+525
src/generate.ts
···
1
+
/**
2
+
* Core lexicon generation logic, importable for testing.
3
+
*
4
+
* Builds lexicon JSON objects from a ContrailConfig. Separated from the
5
+
* script entry point so tests can call it with custom configs and output dirs.
6
+
*/
7
+
8
+
import { writeFileSync, mkdirSync, rmSync, existsSync, readFileSync, readdirSync } from "fs";
9
+
import { join } from "path";
10
+
import type { ContrailConfig } from "./core/types";
11
+
12
+
export interface GenerateOptions {
13
+
config: ContrailConfig;
14
+
/** Root project directory (for finding lexicon source files). */
15
+
rootDir: string;
16
+
/** Directory to write generated lexicons into (will be cleaned first). Omit for in-memory only. */
17
+
outputDir?: string;
18
+
/** Additional lexicon source directories to search for collection schemas. */
19
+
lexiconDirs?: string[];
20
+
/** If true, also writes lex.config.js and queryable.generated.ts. */
21
+
writeRuntimeFiles?: boolean;
22
+
/** Suppress console output. */
23
+
quiet?: boolean;
24
+
}
25
+
26
+
function fieldToParam(field: string): string {
27
+
return field.replace(/\.(\w)/g, (_, c) => c.toUpperCase());
28
+
}
29
+
30
+
interface QueryableField {
31
+
type?: "range";
32
+
}
33
+
34
+
interface CountField {
35
+
name: string;
36
+
description: string;
37
+
}
38
+
39
+
interface RelationDef {
40
+
relName: string;
41
+
collection: string;
42
+
groupBy?: string;
43
+
groups: Record<string, string>;
44
+
}
45
+
46
+
interface ReferenceDef {
47
+
refName: string;
48
+
collection: string;
49
+
}
50
+
51
+
export function generateLexicons(options: GenerateOptions): Record<string, object> {
52
+
const { config, rootDir, outputDir, quiet } = options;
53
+
const lexiconDirs = options.lexiconDirs ?? [
54
+
join(rootDir, "lexicons"),
55
+
join(rootDir, "lexicons-pulled"),
56
+
];
57
+
58
+
const log = quiet ? () => {} : console.log;
59
+
const generated: Record<string, object> = {};
60
+
61
+
// --- Helpers that depend on lexiconDirs ---
62
+
63
+
function findCollectionLexicon(collection: string): string | null {
64
+
const segments = collection.split(".");
65
+
for (const dir of lexiconDirs) {
66
+
const filePath = join(dir, ...segments) + ".json";
67
+
if (existsSync(filePath)) return filePath;
68
+
}
69
+
return null;
70
+
}
71
+
72
+
function detectQueryableFields(collection: string): Record<string, QueryableField> {
73
+
const filePath = findCollectionLexicon(collection);
74
+
if (!filePath) return {};
75
+
try {
76
+
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
77
+
const mainRecord = doc.defs?.main?.record;
78
+
if (!mainRecord?.properties) return {};
79
+
return analyzeProperties(doc.defs, mainRecord.properties, "");
80
+
} catch {
81
+
return {};
82
+
}
83
+
}
84
+
85
+
function getKnownValues(collection: string, fieldName: string): string[] {
86
+
const filePath = findCollectionLexicon(collection);
87
+
if (!filePath) return [];
88
+
try {
89
+
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
90
+
const props = doc.defs?.main?.record?.properties;
91
+
if (!props) return [];
92
+
const field = props[fieldName];
93
+
if (!field) return [];
94
+
if (Array.isArray(field.knownValues)) return field.knownValues;
95
+
return [];
96
+
} catch {
97
+
return [];
98
+
}
99
+
}
100
+
101
+
function getCollectionLexiconRef(collection: string): string | null {
102
+
const filePath = findCollectionLexicon(collection);
103
+
if (!filePath) return null;
104
+
try {
105
+
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
106
+
if (doc.defs?.main) return `${collection}#main`;
107
+
} catch {}
108
+
return null;
109
+
}
110
+
111
+
function getRecordObjectSchema(collection: string): any | null {
112
+
const filePath = findCollectionLexicon(collection);
113
+
if (!filePath) return null;
114
+
try {
115
+
const doc = JSON.parse(readFileSync(filePath, "utf-8"));
116
+
const main = doc.defs?.main;
117
+
if (main?.type === "record" && main.record) return main.record;
118
+
return null;
119
+
} catch {
120
+
return null;
121
+
}
122
+
}
123
+
124
+
// --- Writing ---
125
+
126
+
if (outputDir) {
127
+
rmSync(outputDir, { recursive: true, force: true });
128
+
}
129
+
130
+
function writeLexicon(nsid: string, doc: object) {
131
+
if (outputDir) {
132
+
const filePath = join(outputDir, ...nsid.split(".")) + ".json";
133
+
mkdirSync(join(filePath, ".."), { recursive: true });
134
+
writeFileSync(filePath, JSON.stringify(doc, null, 2) + "\n");
135
+
}
136
+
generated[nsid] = doc;
137
+
log(` ${nsid}`);
138
+
}
139
+
140
+
// --- Building ---
141
+
142
+
function buildRecordDef(
143
+
collectionRef: string | null,
144
+
countFields?: CountField[],
145
+
relationDefs?: RelationDef[],
146
+
referenceDefs?: ReferenceDef[]
147
+
) {
148
+
const properties: Record<string, any> = {
149
+
uri: { type: "string", format: "at-uri" },
150
+
did: { type: "string", format: "did" },
151
+
collection: { type: "string", format: "nsid" },
152
+
rkey: { type: "string" },
153
+
cid: { type: "string" },
154
+
record: collectionRef ? { type: "ref", ref: collectionRef } : { type: "unknown" },
155
+
time_us: { type: "integer" },
156
+
};
157
+
if (countFields) {
158
+
for (const cf of countFields) {
159
+
properties[cf.name] = { type: "integer", description: cf.description };
160
+
}
161
+
}
162
+
if (relationDefs && relationDefs.length > 0) {
163
+
for (const rd of relationDefs) {
164
+
const cap = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
165
+
if (rd.groupBy && Object.keys(rd.groups).length > 0) {
166
+
properties[rd.relName] = { type: "ref", ref: `#hydrate${cap(rd.relName)}` };
167
+
} else {
168
+
properties[rd.relName] = { type: "array", items: { type: "ref", ref: `#hydrate${cap(rd.relName)}Record` } };
169
+
}
170
+
}
171
+
}
172
+
if (referenceDefs && referenceDefs.length > 0) {
173
+
for (const rd of referenceDefs) {
174
+
const cap = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
175
+
properties[rd.refName] = { type: "ref", ref: `#ref${cap(rd.refName)}Record` };
176
+
}
177
+
}
178
+
return { type: "object", required: ["uri", "did", "collection", "rkey", "time_us"], properties };
179
+
}
180
+
181
+
function buildHydrateDefs(relationDefs: RelationDef[]): Record<string, any> {
182
+
const defs: Record<string, any> = {};
183
+
const cap = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
184
+
for (const rd of relationDefs) {
185
+
const relCollectionRef = getCollectionLexiconRef(rd.collection);
186
+
const recordDefName = `hydrate${cap(rd.relName)}Record`;
187
+
defs[recordDefName] = {
188
+
type: "object",
189
+
required: ["uri", "did", "collection", "rkey", "time_us"],
190
+
properties: {
191
+
uri: { type: "string", format: "at-uri" },
192
+
did: { type: "string", format: "did" },
193
+
collection: { type: "string", format: "nsid" },
194
+
rkey: { type: "string" },
195
+
cid: { type: "string" },
196
+
record: relCollectionRef ? { type: "ref", ref: relCollectionRef } : { type: "unknown" },
197
+
time_us: { type: "integer" },
198
+
},
199
+
};
200
+
if (rd.groupBy && Object.keys(rd.groups).length > 0) {
201
+
const groupDefName = `hydrate${cap(rd.relName)}`;
202
+
const groupProperties: Record<string, any> = {};
203
+
for (const shortName of Object.keys(rd.groups)) {
204
+
groupProperties[shortName] = { type: "array", items: { type: "ref", ref: `#${recordDefName}` } };
205
+
}
206
+
groupProperties["other"] = { type: "array", items: { type: "ref", ref: `#${recordDefName}` } };
207
+
defs[groupDefName] = { type: "object", properties: groupProperties };
208
+
}
209
+
}
210
+
return defs;
211
+
}
212
+
213
+
function buildReferenceDefs(referenceDefs: ReferenceDef[]): Record<string, any> {
214
+
const defs: Record<string, any> = {};
215
+
const cap = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
216
+
for (const rd of referenceDefs) {
217
+
const refCollectionRef = getCollectionLexiconRef(rd.collection);
218
+
const recordDefName = `ref${cap(rd.refName)}Record`;
219
+
defs[recordDefName] = {
220
+
type: "object",
221
+
required: ["uri", "did", "collection", "rkey", "time_us"],
222
+
properties: {
223
+
uri: { type: "string", format: "at-uri" },
224
+
did: { type: "string", format: "did" },
225
+
collection: { type: "string", format: "nsid" },
226
+
rkey: { type: "string" },
227
+
cid: { type: "string" },
228
+
record: refCollectionRef ? { type: "ref", ref: refCollectionRef } : { type: "unknown" },
229
+
time_us: { type: "integer" },
230
+
},
231
+
};
232
+
}
233
+
return defs;
234
+
}
235
+
236
+
function profileDefs() {
237
+
const profiles = config.profiles ?? ["app.bsky.actor.profile"];
238
+
const extraDefs: Record<string, any> = {};
239
+
const objectRefs: string[] = [];
240
+
for (const col of profiles) {
241
+
const schema = getRecordObjectSchema(col);
242
+
if (!schema) continue;
243
+
const defName = col.split(".").map((p, i) => i === 0 ? p : p.charAt(0).toUpperCase() + p.slice(1)).join("");
244
+
extraDefs[defName] = schema;
245
+
objectRefs.push(`#${defName}`);
246
+
}
247
+
let recordField: any;
248
+
if (objectRefs.length === 1) recordField = { type: "ref", ref: objectRefs[0] };
249
+
else if (objectRefs.length > 1) recordField = { type: "union", refs: objectRefs };
250
+
else recordField = { type: "unknown" };
251
+
return {
252
+
profileEntry: {
253
+
type: "object",
254
+
required: ["did"],
255
+
properties: {
256
+
did: { type: "string", format: "did" },
257
+
handle: { type: "string" },
258
+
uri: { type: "string", format: "at-uri" },
259
+
collection: { type: "string", format: "nsid" },
260
+
rkey: { type: "string" },
261
+
cid: { type: "string" },
262
+
record: recordField,
263
+
},
264
+
},
265
+
...extraDefs,
266
+
};
267
+
}
268
+
269
+
function tokenShortName(token: string): string {
270
+
const hash = token.indexOf("#");
271
+
return hash !== -1 ? token.slice(hash + 1) : token;
272
+
}
273
+
274
+
// --- Generate ---
275
+
276
+
const ns = config.namespace;
277
+
278
+
log("Generating admin endpoints...");
279
+
280
+
writeLexicon(`${ns}.admin.getCursor`, {
281
+
lexicon: 1, id: `${ns}.admin.getCursor`,
282
+
defs: { main: { type: "query", description: "Get the current cursor position", output: { encoding: "application/json", schema: { type: "object", properties: { time_us: { type: "integer" }, date: { type: "string" }, seconds_ago: { type: "integer" } } } } } },
283
+
});
284
+
285
+
writeLexicon(`${ns}.admin.getOverview`, {
286
+
lexicon: 1, id: `${ns}.admin.getOverview`,
287
+
defs: { main: { type: "query", description: "Get an overview of all indexed collections", output: { encoding: "application/json", schema: { type: "object", required: ["total_records", "collections"], properties: { total_records: { type: "integer" }, collections: { type: "array", items: { type: "ref", ref: "#collectionStats" } } } } } }, collectionStats: { type: "object", required: ["collection", "records", "unique_users"], properties: { collection: { type: "string" }, records: { type: "integer" }, unique_users: { type: "integer" } } } },
288
+
});
289
+
290
+
writeLexicon(`${ns}.admin.sync`, {
291
+
lexicon: 1, id: `${ns}.admin.sync`,
292
+
defs: { main: { type: "query", description: "Discover users from relays and backfill their records from PDS", parameters: { type: "params", properties: { concurrency: { type: "integer", minimum: 1, maximum: 50, default: 10 } } }, output: { encoding: "application/json", schema: { type: "object", required: ["discovered", "backfilled", "remaining", "done"], properties: { discovered: { type: "integer" }, backfilled: { type: "integer" }, remaining: { type: "integer" }, done: { type: "boolean" } } } } } },
293
+
});
294
+
295
+
writeLexicon(`${ns}.admin.reset`, {
296
+
lexicon: 1, id: `${ns}.admin.reset`,
297
+
defs: { main: { type: "query", description: "Delete all data from all tables", output: { encoding: "application/json", schema: { type: "object", required: ["ok"], properties: { ok: { type: "boolean" } } } } } },
298
+
});
299
+
300
+
writeLexicon(`${ns}.getProfile`, {
301
+
lexicon: 1, id: `${ns}.getProfile`,
302
+
defs: { main: { type: "query", description: "Get a user's profile by DID or handle", parameters: { type: "params", required: ["actor"], properties: { actor: { type: "string", format: "at-identifier", description: "DID or handle of the user" } } }, output: { encoding: "application/json", schema: { type: "ref", ref: "#profileEntry" } } }, ...profileDefs() },
303
+
});
304
+
305
+
writeLexicon(`${ns}.notifyOfUpdate`, {
306
+
lexicon: 1, id: `${ns}.notifyOfUpdate`,
307
+
defs: { main: { type: "procedure", description: "Notify of a record change for immediate indexing. Fetches the record from the user's PDS and indexes (or deletes) it.", input: { encoding: "application/json", schema: { type: "object", properties: { uri: { type: "string", format: "at-uri", description: "Single AT URI to fetch and index" }, uris: { type: "array", items: { type: "string", format: "at-uri" }, maxLength: 25, description: "Batch of AT URIs to fetch and index (max 25)" } } } }, output: { encoding: "application/json", schema: { type: "object", required: ["indexed", "deleted"], properties: { indexed: { type: "integer", description: "Number of records created or updated" }, deleted: { type: "integer", description: "Number of records deleted (not found on PDS)" }, errors: { type: "array", items: { type: "string" }, description: "Errors for individual URIs that could not be processed" } } } } } },
308
+
});
309
+
310
+
// --- Per-collection ---
311
+
312
+
log("Generating collection endpoints...");
313
+
314
+
const resolvedQueryableMap: Record<string, Record<string, { type?: "range" }>> = {};
315
+
const resolvedRelationsMap: Record<string, Record<string, { collection: string; groupBy: string; groups: Record<string, string> }>> = {};
316
+
317
+
for (const [collection, colConfig] of Object.entries(config.collections)) {
318
+
const collectionRef = getCollectionLexiconRef(collection);
319
+
320
+
const autoDetected = detectQueryableFields(collection);
321
+
const manual = colConfig.queryable ?? {};
322
+
const merged = { ...autoDetected, ...manual };
323
+
resolvedQueryableMap[collection] = merged;
324
+
325
+
// --- listRecords ---
326
+
const listParams: Record<string, any> = {
327
+
limit: { type: "integer", minimum: 1, maximum: 100, default: 50 },
328
+
cursor: { type: "string" },
329
+
actor: { type: "string", format: "at-identifier", description: "Filter by DID or handle (triggers on-demand backfill)" },
330
+
profiles: { type: "boolean", description: "Include profile + identity info keyed by DID" },
331
+
};
332
+
333
+
// Search param
334
+
if (colConfig.searchable !== false) {
335
+
const allQueryable = { ...autoDetected, ...manual };
336
+
const searchableFields = Array.isArray(colConfig.searchable)
337
+
? colConfig.searchable
338
+
: Object.entries(allQueryable).filter(([, f]) => f.type !== "range").map(([name]) => name);
339
+
if (searchableFields.length > 0) {
340
+
listParams["search"] = {
341
+
type: "string",
342
+
description: `Full-text search across: ${searchableFields.join(", ")}`,
343
+
};
344
+
}
345
+
}
346
+
347
+
for (const [field, fieldConfig] of Object.entries(merged)) {
348
+
const param = fieldToParam(field);
349
+
if (fieldConfig.type === "range") {
350
+
listParams[`${param}Min`] = { type: "string", description: `Minimum value for ${field}` };
351
+
listParams[`${param}Max`] = { type: "string", description: `Maximum value for ${field}` };
352
+
} else {
353
+
listParams[param] = { type: "string", description: `Filter by ${field}` };
354
+
}
355
+
}
356
+
357
+
const countFields: CountField[] = [];
358
+
const relationDefs: RelationDef[] = [];
359
+
const cap = (s: string) => s.charAt(0).toUpperCase() + s.slice(1);
360
+
361
+
for (const [relName, rel] of Object.entries(colConfig.relations ?? {})) {
362
+
countFields.push({ name: `${relName}Count`, description: `Total ${relName} count` });
363
+
listParams[`${relName}CountMin`] = { type: "integer", description: `Minimum total ${relName} count` };
364
+
listParams[`hydrate${cap(relName)}`] = { type: "integer", minimum: 1, maximum: 50, description: `Number of ${relName} records to embed per record` };
365
+
366
+
const groupMapping: Record<string, string> = {};
367
+
if (rel.groupBy) {
368
+
const knownValues = getKnownValues(rel.collection, rel.groupBy);
369
+
for (const token of knownValues) {
370
+
const shortName = tokenShortName(token);
371
+
groupMapping[shortName] = token;
372
+
countFields.push({ name: `${relName}${cap(shortName)}Count`, description: `${relName} count where ${rel.groupBy} = ${shortName}` });
373
+
listParams[`${relName}${cap(shortName)}CountMin`] = { type: "integer", description: `Minimum ${relName} count where ${rel.groupBy} = ${shortName}` };
374
+
}
375
+
if (!resolvedRelationsMap[collection]) resolvedRelationsMap[collection] = {};
376
+
resolvedRelationsMap[collection][relName] = { collection: rel.collection, groupBy: rel.groupBy, groups: groupMapping };
377
+
}
378
+
379
+
relationDefs.push({ relName, collection: rel.collection, groupBy: rel.groupBy, groups: groupMapping });
380
+
}
381
+
382
+
const referenceDefs: ReferenceDef[] = [];
383
+
for (const [refName, ref] of Object.entries(colConfig.references ?? {})) {
384
+
referenceDefs.push({ refName, collection: ref.collection });
385
+
}
386
+
for (const refName of Object.keys(colConfig.references ?? {})) {
387
+
listParams[`hydrate${cap(refName)}`] = { type: "boolean", description: `Embed the referenced ${refName} record` };
388
+
}
389
+
390
+
const sortableValues: string[] = [];
391
+
for (const field of Object.keys(merged)) sortableValues.push(fieldToParam(field));
392
+
for (const cf of countFields) sortableValues.push(cf.name);
393
+
if (sortableValues.length > 0) {
394
+
listParams["sort"] = { type: "string", knownValues: sortableValues, description: "Field to sort by (default: time_us)" };
395
+
listParams["order"] = { type: "string", knownValues: ["asc", "desc"], description: "Sort direction (default: desc for dates/numbers/counts, asc for strings)" };
396
+
}
397
+
398
+
const hydrateDefs = buildHydrateDefs(relationDefs);
399
+
const refDefs = buildReferenceDefs(referenceDefs);
400
+
401
+
writeLexicon(`${collection}.listRecords`, {
402
+
lexicon: 1, id: `${collection}.listRecords`,
403
+
defs: {
404
+
main: { type: "query", description: `Query ${collection} records with filters`, parameters: { type: "params", properties: listParams }, output: { encoding: "application/json", schema: { type: "object", required: ["records"], properties: { records: { type: "array", items: { type: "ref", ref: "#record" } }, cursor: { type: "string" }, profiles: { type: "array", items: { type: "ref", ref: "#profileEntry" } } } } } },
405
+
record: buildRecordDef(collectionRef, countFields, relationDefs, referenceDefs),
406
+
...hydrateDefs, ...refDefs, ...profileDefs(),
407
+
},
408
+
});
409
+
410
+
// --- getRecord ---
411
+
const getParams: Record<string, any> = {
412
+
uri: { type: "string", format: "at-uri", description: "AT URI of the record" },
413
+
profiles: { type: "boolean", description: "Include profile + identity info keyed by DID" },
414
+
};
415
+
for (const rd of relationDefs) {
416
+
getParams[`hydrate${cap(rd.relName)}`] = { type: "integer", minimum: 1, maximum: 50, description: `Number of ${rd.relName} records to embed` };
417
+
}
418
+
for (const refName of Object.keys(colConfig.references ?? {})) {
419
+
getParams[`hydrate${cap(refName)}`] = { type: "boolean", description: `Embed the referenced ${refName} record` };
420
+
}
421
+
422
+
writeLexicon(`${collection}.getRecord`, {
423
+
lexicon: 1, id: `${collection}.getRecord`,
424
+
defs: {
425
+
main: { type: "query", description: `Get a single ${collection} record by AT URI`, parameters: { type: "params", required: ["uri"], properties: getParams }, output: { encoding: "application/json", schema: { type: "object", required: ["uri", "did", "collection", "rkey", "time_us"], properties: { ...buildRecordDef(collectionRef, countFields, relationDefs, referenceDefs).properties, profiles: { type: "array", items: { type: "ref", ref: "#profileEntry" } } } } } },
426
+
...hydrateDefs, ...refDefs, ...profileDefs(),
427
+
},
428
+
});
429
+
430
+
for (const queryName of Object.keys(colConfig.queries ?? {})) {
431
+
writeLexicon(`${collection}.${queryName}`, {
432
+
lexicon: 1, id: `${collection}.${queryName}`,
433
+
defs: { main: { type: "query", description: `Custom query: ${queryName}`, output: { encoding: "application/json", schema: { type: "object", properties: {} } } } },
434
+
});
435
+
}
436
+
}
437
+
438
+
// --- Runtime files (only when called from script) ---
439
+
if (options.writeRuntimeFiles) {
440
+
// lex.config.js
441
+
const collectionNsids = Object.keys(config.collections);
442
+
const pulledFiles = [...scanLexiconsDir(lexiconDirs), ...scanLexiconsDir([])].flat();
443
+
const allRefs = new Set<string>();
444
+
for (const file of pulledFiles) {
445
+
for (const ref of findRefsInLexicon(file)) allRefs.add(ref);
446
+
}
447
+
const profileNsids = config.profiles ?? ["app.bsky.actor.profile"];
448
+
const pullNsids = new Set([...collectionNsids, ...profileNsids]);
449
+
for (const ref of allRefs) {
450
+
if (!ref.startsWith("com.atproto.")) pullNsids.add(ref);
451
+
}
452
+
const sortedNsids = [...pullNsids].sort();
453
+
const lexConfigContent = `import { defineLexiconConfig } from "@atcute/lex-cli";\n\nexport default defineLexiconConfig({\n files: ["lexicons/**/*.json", "lexicons-pulled/**/*.json", "lexicons-generated/**/*.json"],\n outdir: "src/lexicon-types/",\n imports: ["@atcute/atproto"],\n pull: {\n outdir: "lexicons-pulled/",\n sources: [\n {\n type: "atproto",\n mode: "nsids",\n nsids: ${JSON.stringify(sortedNsids, null, 10).replace(/^/gm, " ").trim()},\n },\n ],\n },\n});\n`;
454
+
writeFileSync(join(rootDir, "lex.config.js"), lexConfigContent);
455
+
log(`\nGenerated lex.config.js with ${sortedNsids.length} pull NSIDs`);
456
+
457
+
// queryable.generated.ts
458
+
const queryableContent = `// Auto-generated — do not edit. Run \`pnpm generate\` to regenerate.\nimport type { QueryableField } from "./types";\n\nexport const resolvedQueryable: Record<string, Record<string, QueryableField>> = ${JSON.stringify(resolvedQueryableMap, null, 2)};\n\nexport interface ResolvedRelation {\n collection: string;\n groupBy: string;\n groups: Record<string, string>; // shortName → full token value\n}\n\nexport const resolvedRelationsMap: Record<string, Record<string, ResolvedRelation>> = ${JSON.stringify(resolvedRelationsMap, null, 2)};\n`;
459
+
writeFileSync(join(rootDir, "src", "core", "queryable.generated.ts"), queryableContent);
460
+
log("Generated src/core/queryable.generated.ts");
461
+
}
462
+
463
+
log("\nDone!");
464
+
return generated;
465
+
}
466
+
467
+
// --- Helpers used by writeRuntimeFiles ---
468
+
469
+
function analyzeProperties(
470
+
defs: Record<string, any>,
471
+
properties: Record<string, any>,
472
+
prefix: string
473
+
): Record<string, QueryableField> {
474
+
const result: Record<string, QueryableField> = {};
475
+
for (const [field, def] of Object.entries(properties)) {
476
+
const path = prefix ? `${prefix}.${field}` : field;
477
+
if (def.type === "string") {
478
+
if (def.format === "datetime") result[path] = { type: "range" };
479
+
else if (def.format !== "uri" && def.format !== "at-uri") result[path] = {};
480
+
} else if (def.type === "integer" || def.type === "number") {
481
+
result[path] = { type: "range" };
482
+
} else if (def.type === "ref" && def.ref === "com.atproto.repo.strongRef") {
483
+
result[`${path}.uri`] = {};
484
+
} else if (def.type === "union" && Array.isArray(def.refs) && def.refs.includes("com.atproto.repo.strongRef")) {
485
+
result[`${path}.uri`] = {};
486
+
} else if (def.type === "ref" && def.ref) {
487
+
const refId = def.ref.includes("#") ? def.ref.split("#")[1] : null;
488
+
if (refId && defs[refId]?.type === "string") result[path] = {};
489
+
}
490
+
}
491
+
return result;
492
+
}
493
+
494
+
function scanLexiconsDir(dirs: string[]): string[] {
495
+
const files: string[] = [];
496
+
for (const dir of dirs) {
497
+
if (!existsSync(dir)) continue;
498
+
for (const entry of readdirSync(dir, { withFileTypes: true })) {
499
+
const fullPath = join(dir, entry.name);
500
+
if (entry.isDirectory()) files.push(...scanLexiconsDir([fullPath]));
501
+
else if (entry.name.endsWith(".json")) files.push(fullPath);
502
+
}
503
+
}
504
+
return files;
505
+
}
506
+
507
+
function findRefsInLexicon(filePath: string): string[] {
508
+
try {
509
+
const content = readFileSync(filePath, "utf-8");
510
+
const refs: string[] = [];
511
+
const refPattern = /"ref":\s*"([a-z][a-zA-Z0-9]*(?:\.[a-zA-Z0-9]+)+)(?:#\w+)?"/g;
512
+
let match;
513
+
while ((match = refPattern.exec(content)) !== null) refs.push(match[1]);
514
+
const refsArrayPattern = /"refs":\s*\[([^\]]+)\]/g;
515
+
while ((match = refsArrayPattern.exec(content)) !== null) {
516
+
const inner = match[1];
517
+
const nsidPattern = /"([a-z][a-zA-Z0-9]*(?:\.[a-zA-Z0-9]+)+)(?:#\w+)?"/g;
518
+
let innerMatch;
519
+
while ((innerMatch = nsidPattern.exec(inner)) !== null) refs.push(innerMatch[1]);
520
+
}
521
+
return refs;
522
+
} catch {
523
+
return [];
524
+
}
525
+
}
+265
tests/generate.test.ts
+265
tests/generate.test.ts
···
1
+
import { describe, it, expect, beforeAll } from "vitest";
2
+
import { join } from "path";
3
+
import { generateLexicons } from "../src/generate";
4
+
import type { ContrailConfig } from "../src/core/types";
5
+
6
+
const ROOT_DIR = join(__dirname, "..");
7
+
8
+
function getParams(lexicon: any): Record<string, any> {
9
+
return lexicon?.defs?.main?.parameters?.properties ?? {};
10
+
}
11
+
12
+
function getInputSchema(lexicon: any): any {
13
+
return lexicon?.defs?.main?.input?.schema;
14
+
}
15
+
16
+
function getOutputSchema(lexicon: any): any {
17
+
return lexicon?.defs?.main?.output?.schema;
18
+
}
19
+
20
+
function generate(config: ContrailConfig) {
21
+
return generateLexicons({
22
+
config,
23
+
rootDir: ROOT_DIR,
24
+
lexiconDirs: [],
25
+
quiet: true,
26
+
});
27
+
}
28
+
29
+
// --- Test configs ---
30
+
31
+
const BASIC_CONFIG: ContrailConfig = {
32
+
namespace: "test.app",
33
+
collections: {
34
+
"com.example.post": {
35
+
queryable: {
36
+
title: {},
37
+
body: {},
38
+
createdAt: { type: "range" },
39
+
},
40
+
},
41
+
},
42
+
};
43
+
44
+
const RELATIONS_CONFIG: ContrailConfig = {
45
+
namespace: "test.app",
46
+
collections: {
47
+
"com.example.post": {
48
+
queryable: { title: {} },
49
+
relations: {
50
+
likes: {
51
+
collection: "com.example.like",
52
+
},
53
+
},
54
+
},
55
+
"com.example.like": {
56
+
queryable: { status: {} },
57
+
references: {
58
+
post: {
59
+
collection: "com.example.post",
60
+
field: "subject.uri",
61
+
},
62
+
},
63
+
},
64
+
},
65
+
};
66
+
67
+
const SEARCH_EXPLICIT_CONFIG: ContrailConfig = {
68
+
namespace: "test.app",
69
+
collections: {
70
+
"com.example.post": {
71
+
queryable: {
72
+
title: {},
73
+
body: {},
74
+
category: {},
75
+
createdAt: { type: "range" },
76
+
},
77
+
searchable: ["title", "body"],
78
+
},
79
+
},
80
+
};
81
+
82
+
const SEARCH_DISABLED_CONFIG: ContrailConfig = {
83
+
namespace: "test.app",
84
+
collections: {
85
+
"com.example.post": {
86
+
queryable: { title: {}, body: {} },
87
+
searchable: false,
88
+
},
89
+
},
90
+
};
91
+
92
+
const SEARCH_AUTO_CONFIG: ContrailConfig = {
93
+
namespace: "test.app",
94
+
collections: {
95
+
"com.example.post": {
96
+
queryable: {
97
+
title: {},
98
+
body: {},
99
+
score: { type: "range" },
100
+
},
101
+
},
102
+
},
103
+
};
104
+
105
+
describe("basic generation", () => {
106
+
let lexicons: Record<string, any>;
107
+
108
+
beforeAll(() => {
109
+
lexicons = generate(BASIC_CONFIG);
110
+
});
111
+
112
+
it("generates admin endpoints", () => {
113
+
expect(lexicons["test.app.admin.getCursor"]).toBeDefined();
114
+
expect(lexicons["test.app.admin.getOverview"]).toBeDefined();
115
+
expect(lexicons["test.app.admin.sync"]).toBeDefined();
116
+
expect(lexicons["test.app.admin.reset"]).toBeDefined();
117
+
});
118
+
119
+
it("generates getProfile", () => {
120
+
const lex = lexicons["test.app.getProfile"];
121
+
expect(lex).toBeDefined();
122
+
const params = getParams(lex);
123
+
expect(params.actor).toBeDefined();
124
+
expect(params.actor.format).toBe("at-identifier");
125
+
});
126
+
127
+
it("generates notifyOfUpdate as a procedure", () => {
128
+
const lex = lexicons["test.app.notifyOfUpdate"];
129
+
expect(lex).toBeDefined();
130
+
expect(lex.defs.main.type).toBe("procedure");
131
+
132
+
const input = getInputSchema(lex);
133
+
expect(input.properties.uri).toBeDefined();
134
+
expect(input.properties.uri.format).toBe("at-uri");
135
+
expect(input.properties.uris.type).toBe("array");
136
+
expect(input.properties.uris.maxLength).toBe(25);
137
+
138
+
const output = getOutputSchema(lex);
139
+
expect(output.required).toContain("indexed");
140
+
expect(output.required).toContain("deleted");
141
+
expect(output.properties.errors.type).toBe("array");
142
+
});
143
+
144
+
it("generates listRecords with standard params", () => {
145
+
const params = getParams(lexicons["com.example.post.listRecords"]);
146
+
expect(params.limit).toBeDefined();
147
+
expect(params.cursor).toBeDefined();
148
+
expect(params.actor).toBeDefined();
149
+
expect(params.profiles).toBeDefined();
150
+
});
151
+
152
+
it("generates queryable field params", () => {
153
+
const params = getParams(lexicons["com.example.post.listRecords"]);
154
+
expect(params.title).toBeDefined();
155
+
expect(params.title.type).toBe("string");
156
+
expect(params.body).toBeDefined();
157
+
expect(params.createdAtMin).toBeDefined();
158
+
expect(params.createdAtMax).toBeDefined();
159
+
});
160
+
161
+
it("generates sort and order params", () => {
162
+
const params = getParams(lexicons["com.example.post.listRecords"]);
163
+
expect(params.sort).toBeDefined();
164
+
expect(params.sort.knownValues).toContain("title");
165
+
expect(params.sort.knownValues).toContain("body");
166
+
expect(params.sort.knownValues).toContain("createdAt");
167
+
expect(params.order.knownValues).toEqual(["asc", "desc"]);
168
+
});
169
+
170
+
it("generates getRecord with uri param", () => {
171
+
const params = getParams(lexicons["com.example.post.getRecord"]);
172
+
expect(params.uri).toBeDefined();
173
+
expect(params.uri.format).toBe("at-uri");
174
+
});
175
+
176
+
it("does not include search on getRecord", () => {
177
+
const params = getParams(lexicons["com.example.post.getRecord"]);
178
+
expect(params.search).toBeUndefined();
179
+
});
180
+
});
181
+
182
+
describe("relations and references", () => {
183
+
let lexicons: Record<string, any>;
184
+
185
+
beforeAll(() => {
186
+
lexicons = generate(RELATIONS_CONFIG);
187
+
});
188
+
189
+
it("generates count filter params for relations", () => {
190
+
const params = getParams(lexicons["com.example.post.listRecords"]);
191
+
expect(params.likesCountMin).toBeDefined();
192
+
expect(params.likesCountMin.type).toBe("integer");
193
+
});
194
+
195
+
it("generates hydrate params for relations", () => {
196
+
const params = getParams(lexicons["com.example.post.listRecords"]);
197
+
expect(params.hydrateLikes).toBeDefined();
198
+
expect(params.hydrateLikes.type).toBe("integer");
199
+
});
200
+
201
+
it("generates hydrate params for references", () => {
202
+
const params = getParams(lexicons["com.example.like.listRecords"]);
203
+
expect(params.hydratePost).toBeDefined();
204
+
expect(params.hydratePost.type).toBe("boolean");
205
+
});
206
+
207
+
it("includes count fields in record def", () => {
208
+
const recordDef = lexicons["com.example.post.listRecords"].defs.record;
209
+
expect(recordDef.properties.likesCount).toBeDefined();
210
+
expect(recordDef.properties.likesCount.type).toBe("integer");
211
+
});
212
+
213
+
it("includes relation shape in record def (ungrouped → array)", () => {
214
+
const recordDef = lexicons["com.example.post.listRecords"].defs.record;
215
+
expect(recordDef.properties.likes).toBeDefined();
216
+
expect(recordDef.properties.likes.type).toBe("array");
217
+
});
218
+
219
+
it("includes reference shape in record def", () => {
220
+
const recordDef = lexicons["com.example.like.listRecords"].defs.record;
221
+
expect(recordDef.properties.post).toBeDefined();
222
+
expect(recordDef.properties.post.type).toBe("ref");
223
+
});
224
+
225
+
it("sort knownValues includes count fields", () => {
226
+
const params = getParams(lexicons["com.example.post.listRecords"]);
227
+
expect(params.sort.knownValues).toContain("likesCount");
228
+
});
229
+
});
230
+
231
+
describe("search: explicit fields", () => {
232
+
let lexicons: Record<string, any>;
233
+
234
+
beforeAll(() => {
235
+
lexicons = generate(SEARCH_EXPLICIT_CONFIG);
236
+
});
237
+
238
+
it("includes search param listing only explicit fields", () => {
239
+
const params = getParams(lexicons["com.example.post.listRecords"]);
240
+
expect(params.search).toBeDefined();
241
+
expect(params.search.description).toContain("title");
242
+
expect(params.search.description).toContain("body");
243
+
expect(params.search.description).not.toContain("category");
244
+
expect(params.search.description).not.toContain("createdAt");
245
+
});
246
+
});
247
+
248
+
describe("search: disabled", () => {
249
+
it("does not include search param", () => {
250
+
const lexicons = generate(SEARCH_DISABLED_CONFIG);
251
+
const params = getParams(lexicons["com.example.post.listRecords"]);
252
+
expect(params.search).toBeUndefined();
253
+
});
254
+
});
255
+
256
+
describe("search: auto-detect", () => {
257
+
it("includes search param with non-range fields only", () => {
258
+
const lexicons = generate(SEARCH_AUTO_CONFIG);
259
+
const params = getParams(lexicons["com.example.post.listRecords"]);
260
+
expect(params.search).toBeDefined();
261
+
expect(params.search.description).toContain("title");
262
+
expect(params.search.description).toContain("body");
263
+
expect(params.search.description).not.toContain("score");
264
+
});
265
+
});
+471
tests/notify.test.ts
+471
tests/notify.test.ts
···
1
+
import { describe, it, expect, beforeEach, vi, afterEach } from "vitest";
2
+
import type { Database } from "../src/core/types";
3
+
import { createTestDbWithSchema, makeEvent, TEST_CONFIG } from "./helpers";
4
+
import { parseAtUri } from "../src/core/router/notify";
5
+
import { createApp } from "../src/core/router/index";
6
+
import { applyEvents, queryRecords } from "../src/core/db/records";
7
+
import type { Hono } from "hono";
8
+
9
+
let db: Database;
10
+
let app: Hono;
11
+
12
+
beforeEach(async () => {
13
+
db = await createTestDbWithSchema();
14
+
app = createApp(db, TEST_CONFIG);
15
+
});
16
+
17
+
afterEach(() => {
18
+
vi.restoreAllMocks();
19
+
});
20
+
21
+
describe("parseAtUri", () => {
22
+
it("parses a valid AT URI", () => {
23
+
const result = parseAtUri(
24
+
"at://did:plc:abc123/community.lexicon.calendar.event/rkey1"
25
+
);
26
+
expect(result).toEqual({
27
+
did: "did:plc:abc123",
28
+
collection: "community.lexicon.calendar.event",
29
+
rkey: "rkey1",
30
+
});
31
+
});
32
+
33
+
it("parses did:web URIs", () => {
34
+
const result = parseAtUri(
35
+
"at://did:web:example.com/app.bsky.feed.post/abc"
36
+
);
37
+
expect(result).toEqual({
38
+
did: "did:web:example.com",
39
+
collection: "app.bsky.feed.post",
40
+
rkey: "abc",
41
+
});
42
+
});
43
+
44
+
it("returns null for invalid URIs", () => {
45
+
expect(parseAtUri("")).toBeNull();
46
+
expect(parseAtUri("https://example.com")).toBeNull();
47
+
expect(parseAtUri("at://did:plc:abc")).toBeNull(); // missing collection and rkey
48
+
expect(parseAtUri("at://did:plc:abc/collection")).toBeNull(); // missing rkey
49
+
expect(parseAtUri("at://did:plc:abc/collection/rkey/extra")).toBeNull(); // too many segments
50
+
});
51
+
});
52
+
53
+
describe("POST notifyOfUpdate", () => {
54
+
const endpoint = `/xrpc/${TEST_CONFIG.namespace}.notifyOfUpdate`;
55
+
56
+
function mockFetch(
57
+
records: Record<string, { value: unknown; cid: string }>
58
+
) {
59
+
vi.stubGlobal(
60
+
"fetch",
61
+
vi.fn(async (url: string) => {
62
+
const u = new URL(url);
63
+
const repo = u.searchParams.get("repo");
64
+
const collection = u.searchParams.get("collection");
65
+
const rkey = u.searchParams.get("rkey");
66
+
const uri = `at://${repo}/${collection}/${rkey}`;
67
+
68
+
if (records[uri]) {
69
+
return new Response(JSON.stringify(records[uri]), {
70
+
status: 200,
71
+
headers: { "Content-Type": "application/json" },
72
+
});
73
+
}
74
+
return new Response("not found", { status: 404 });
75
+
})
76
+
);
77
+
}
78
+
79
+
/** Seed the identities table so getPDS resolves without hitting slingshot. */
80
+
async function seedIdentity(did: string, pds: string) {
81
+
await db
82
+
.prepare(
83
+
"INSERT OR REPLACE INTO identities (did, handle, pds, resolved_at) VALUES (?, ?, ?, ?)"
84
+
)
85
+
.bind(did, "test.handle", pds, Date.now())
86
+
.run();
87
+
}
88
+
89
+
it("returns 400 when no uri provided", async () => {
90
+
const res = await app.request(endpoint, {
91
+
method: "POST",
92
+
headers: { "Content-Type": "application/json" },
93
+
body: JSON.stringify({}),
94
+
});
95
+
expect(res.status).toBe(400);
96
+
const body = await res.json();
97
+
expect(body.error).toMatch(/uri/);
98
+
});
99
+
100
+
it("returns 400 when too many URIs", async () => {
101
+
const uris = Array.from({ length: 26 }, (_, i) =>
102
+
`at://did:plc:test/community.lexicon.calendar.event/r${i}`
103
+
);
104
+
const res = await app.request(endpoint, {
105
+
method: "POST",
106
+
headers: { "Content-Type": "application/json" },
107
+
body: JSON.stringify({ uris }),
108
+
});
109
+
expect(res.status).toBe(400);
110
+
const body = await res.json();
111
+
expect(body.error).toMatch(/max 25/);
112
+
});
113
+
114
+
it("reports error for invalid AT URI", async () => {
115
+
const res = await app.request(endpoint, {
116
+
method: "POST",
117
+
headers: { "Content-Type": "application/json" },
118
+
body: JSON.stringify({ uri: "not-a-valid-uri" }),
119
+
});
120
+
expect(res.status).toBe(200);
121
+
const body = await res.json();
122
+
expect(body.errors).toContain("invalid AT URI: not-a-valid-uri");
123
+
expect(body.indexed).toBe(0);
124
+
});
125
+
126
+
it("reports error for untracked collection", async () => {
127
+
const res = await app.request(endpoint, {
128
+
method: "POST",
129
+
headers: { "Content-Type": "application/json" },
130
+
body: JSON.stringify({
131
+
uri: "at://did:plc:test/app.bsky.feed.post/abc",
132
+
}),
133
+
});
134
+
expect(res.status).toBe(200);
135
+
const body = await res.json();
136
+
expect(body.errors).toContain("collection not tracked: app.bsky.feed.post");
137
+
});
138
+
139
+
it("fetches and indexes a record from PDS", async () => {
140
+
const did = "did:plc:test";
141
+
const uri = `at://${did}/community.lexicon.calendar.event/evt1`;
142
+
const record = { name: "Test Event", startsAt: "2026-04-01T10:00:00Z" };
143
+
144
+
await seedIdentity(did, "https://pds.example.com");
145
+
mockFetch({ [uri]: { value: record, cid: "bafytest" } });
146
+
147
+
const res = await app.request(endpoint, {
148
+
method: "POST",
149
+
headers: { "Content-Type": "application/json" },
150
+
body: JSON.stringify({ uri }),
151
+
});
152
+
153
+
expect(res.status).toBe(200);
154
+
const body = await res.json();
155
+
expect(body.indexed).toBe(1);
156
+
expect(body.deleted).toBe(0);
157
+
expect(body.errors).toBeUndefined();
158
+
159
+
// Verify the record is in the database
160
+
const result = await queryRecords(db, TEST_CONFIG, {
161
+
collection: "community.lexicon.calendar.event",
162
+
});
163
+
expect(result.records).toHaveLength(1);
164
+
expect(result.records[0].uri).toBe(uri);
165
+
expect(result.records[0].cid).toBe("bafytest");
166
+
expect(JSON.parse(result.records[0].record!)).toEqual(record);
167
+
});
168
+
169
+
it("deletes locally when record not found on PDS", async () => {
170
+
const did = "did:plc:test";
171
+
const uri = `at://${did}/community.lexicon.calendar.event/evt1`;
172
+
173
+
// Pre-populate a record
174
+
await applyEvents(db, [
175
+
makeEvent({ uri, did, rkey: "evt1", record: { name: "Old" } }),
176
+
]);
177
+
178
+
await seedIdentity(did, "https://pds.example.com");
179
+
mockFetch({}); // PDS returns 404 for everything
180
+
181
+
const res = await app.request(endpoint, {
182
+
method: "POST",
183
+
headers: { "Content-Type": "application/json" },
184
+
body: JSON.stringify({ uri }),
185
+
});
186
+
187
+
expect(res.status).toBe(200);
188
+
const body = await res.json();
189
+
expect(body.indexed).toBe(0);
190
+
expect(body.deleted).toBe(1);
191
+
192
+
// Verify the record is gone
193
+
const result = await queryRecords(db, TEST_CONFIG, {
194
+
collection: "community.lexicon.calendar.event",
195
+
});
196
+
expect(result.records).toHaveLength(0);
197
+
});
198
+
199
+
it("handles batch of URIs", async () => {
200
+
const did = "did:plc:test";
201
+
const uri1 = `at://${did}/community.lexicon.calendar.event/e1`;
202
+
const uri2 = `at://${did}/community.lexicon.calendar.event/e2`;
203
+
204
+
await seedIdentity(did, "https://pds.example.com");
205
+
mockFetch({
206
+
[uri1]: { value: { name: "Event 1" }, cid: "cid1" },
207
+
[uri2]: { value: { name: "Event 2" }, cid: "cid2" },
208
+
});
209
+
210
+
const res = await app.request(endpoint, {
211
+
method: "POST",
212
+
headers: { "Content-Type": "application/json" },
213
+
body: JSON.stringify({ uris: [uri1, uri2] }),
214
+
});
215
+
216
+
expect(res.status).toBe(200);
217
+
const body = await res.json();
218
+
expect(body.indexed).toBe(2);
219
+
220
+
const result = await queryRecords(db, TEST_CONFIG, {
221
+
collection: "community.lexicon.calendar.event",
222
+
});
223
+
expect(result.records).toHaveLength(2);
224
+
});
225
+
226
+
it("updates an existing record (upsert)", async () => {
227
+
const did = "did:plc:test";
228
+
const uri = `at://${did}/community.lexicon.calendar.event/evt1`;
229
+
230
+
// Insert original
231
+
await applyEvents(db, [
232
+
makeEvent({ uri, did, rkey: "evt1", record: { name: "V1" } }),
233
+
]);
234
+
235
+
await seedIdentity(did, "https://pds.example.com");
236
+
mockFetch({
237
+
[uri]: { value: { name: "V2" }, cid: "newcid" },
238
+
});
239
+
240
+
const res = await app.request(endpoint, {
241
+
method: "POST",
242
+
headers: { "Content-Type": "application/json" },
243
+
body: JSON.stringify({ uri }),
244
+
});
245
+
246
+
expect(res.status).toBe(200);
247
+
const body = await res.json();
248
+
expect(body.indexed).toBe(1);
249
+
250
+
const result = await queryRecords(db, TEST_CONFIG, {
251
+
collection: "community.lexicon.calendar.event",
252
+
});
253
+
expect(result.records).toHaveLength(1);
254
+
expect(JSON.parse(result.records[0].record!).name).toBe("V2");
255
+
expect(result.records[0].cid).toBe("newcid");
256
+
});
257
+
258
+
it("updates counts when notifying about a relation record", async () => {
259
+
const did = "did:plc:test";
260
+
const eventUri = `at://${did}/community.lexicon.calendar.event/evt1`;
261
+
const rsvpUri = `at://${did}/community.lexicon.calendar.rsvp/r1`;
262
+
263
+
// Insert parent event
264
+
await applyEvents(db, [
265
+
makeEvent({ uri: eventUri, did, rkey: "evt1", record: { name: "Event" } }),
266
+
]);
267
+
268
+
await seedIdentity(did, "https://pds.example.com");
269
+
mockFetch({
270
+
[rsvpUri]: {
271
+
value: { subject: { uri: eventUri }, status: "going" },
272
+
cid: "rsvpcid",
273
+
},
274
+
});
275
+
276
+
await app.request(endpoint, {
277
+
method: "POST",
278
+
headers: { "Content-Type": "application/json" },
279
+
body: JSON.stringify({ uri: rsvpUri }),
280
+
});
281
+
282
+
// Check that the event now has an RSVP count
283
+
const result = await queryRecords(db, TEST_CONFIG, {
284
+
collection: "community.lexicon.calendar.event",
285
+
});
286
+
expect(result.records).toHaveLength(1);
287
+
expect(
288
+
result.records[0].counts?.["community.lexicon.calendar.rsvp"]
289
+
).toBe(1);
290
+
});
291
+
292
+
it("skips when record already exists with same CID (no double-counting)", async () => {
293
+
const did = "did:plc:test";
294
+
const eventUri = `at://${did}/community.lexicon.calendar.event/evt1`;
295
+
const rsvpUri = `at://${did}/community.lexicon.calendar.rsvp/r1`;
296
+
const rsvpRecord = { subject: { uri: eventUri }, status: "going" };
297
+
298
+
// Insert parent event and RSVP via normal ingestion
299
+
await applyEvents(db, [
300
+
makeEvent({ uri: eventUri, did, rkey: "evt1", record: { name: "Event" } }),
301
+
]);
302
+
await applyEvents(
303
+
db,
304
+
[
305
+
makeEvent({
306
+
uri: rsvpUri,
307
+
did,
308
+
collection: "community.lexicon.calendar.rsvp",
309
+
rkey: "r1",
310
+
cid: "rsvpcid",
311
+
record: rsvpRecord,
312
+
time_us: 2000000,
313
+
}),
314
+
],
315
+
TEST_CONFIG
316
+
);
317
+
318
+
// Verify count is 1
319
+
let result = await queryRecords(db, TEST_CONFIG, {
320
+
collection: "community.lexicon.calendar.event",
321
+
});
322
+
expect(result.records[0].counts?.["community.lexicon.calendar.rsvp"]).toBe(1);
323
+
324
+
// Now notify with the same RSVP (same CID) — should be a no-op
325
+
await seedIdentity(did, "https://pds.example.com");
326
+
mockFetch({
327
+
[rsvpUri]: { value: rsvpRecord, cid: "rsvpcid" },
328
+
});
329
+
330
+
const res = await app.request(endpoint, {
331
+
method: "POST",
332
+
headers: { "Content-Type": "application/json" },
333
+
body: JSON.stringify({ uri: rsvpUri }),
334
+
});
335
+
336
+
const body = await res.json();
337
+
expect(body.indexed).toBe(0); // skipped, nothing changed
338
+
expect(body.deleted).toBe(0);
339
+
340
+
// Count should still be 1, not 2
341
+
result = await queryRecords(db, TEST_CONFIG, {
342
+
collection: "community.lexicon.calendar.event",
343
+
});
344
+
expect(result.records[0].counts?.["community.lexicon.calendar.rsvp"]).toBe(1);
345
+
});
346
+
347
+
it("uses update (not create) when record exists with different CID", async () => {
348
+
const did = "did:plc:test";
349
+
const eventUri = `at://${did}/community.lexicon.calendar.event/evt1`;
350
+
const rsvpUri = `at://${did}/community.lexicon.calendar.rsvp/r1`;
351
+
352
+
// Insert parent event and RSVP via normal ingestion
353
+
await applyEvents(db, [
354
+
makeEvent({ uri: eventUri, did, rkey: "evt1", record: { name: "Event" } }),
355
+
]);
356
+
await applyEvents(
357
+
db,
358
+
[
359
+
makeEvent({
360
+
uri: rsvpUri,
361
+
did,
362
+
collection: "community.lexicon.calendar.rsvp",
363
+
rkey: "r1",
364
+
cid: "oldcid",
365
+
record: { subject: { uri: eventUri }, status: "going" },
366
+
time_us: 2000000,
367
+
}),
368
+
],
369
+
TEST_CONFIG
370
+
);
371
+
372
+
// Now notify with updated record (different CID)
373
+
await seedIdentity(did, "https://pds.example.com");
374
+
mockFetch({
375
+
[rsvpUri]: {
376
+
value: { subject: { uri: eventUri }, status: "interested" },
377
+
cid: "newcid",
378
+
},
379
+
});
380
+
381
+
const res = await app.request(endpoint, {
382
+
method: "POST",
383
+
headers: { "Content-Type": "application/json" },
384
+
body: JSON.stringify({ uri: rsvpUri }),
385
+
});
386
+
387
+
const body = await res.json();
388
+
expect(body.indexed).toBe(1);
389
+
390
+
// Record should be updated
391
+
const rsvpResult = await queryRecords(db, TEST_CONFIG, {
392
+
collection: "community.lexicon.calendar.rsvp",
393
+
});
394
+
expect(rsvpResult.records).toHaveLength(1);
395
+
expect(rsvpResult.records[0].cid).toBe("newcid");
396
+
397
+
// Count should still be 1 (not double-counted)
398
+
const result = await queryRecords(db, TEST_CONFIG, {
399
+
collection: "community.lexicon.calendar.event",
400
+
});
401
+
expect(result.records[0].counts?.["community.lexicon.calendar.rsvp"]).toBe(1);
402
+
});
403
+
404
+
it("does nothing when record not on PDS and not local", async () => {
405
+
const did = "did:plc:test";
406
+
const uri = `at://${did}/community.lexicon.calendar.event/nonexistent`;
407
+
408
+
await seedIdentity(did, "https://pds.example.com");
409
+
mockFetch({}); // 404 for everything
410
+
411
+
const res = await app.request(endpoint, {
412
+
method: "POST",
413
+
headers: { "Content-Type": "application/json" },
414
+
body: JSON.stringify({ uri }),
415
+
});
416
+
417
+
const body = await res.json();
418
+
expect(body.indexed).toBe(0);
419
+
expect(body.deleted).toBe(0);
420
+
});
421
+
422
+
it("decrements counts when deleting a relation record", async () => {
423
+
const did = "did:plc:test";
424
+
const eventUri = `at://${did}/community.lexicon.calendar.event/evt1`;
425
+
const rsvpUri = `at://${did}/community.lexicon.calendar.rsvp/r1`;
426
+
427
+
// Insert event + RSVP
428
+
await applyEvents(db, [
429
+
makeEvent({ uri: eventUri, did, rkey: "evt1", record: { name: "Event" } }),
430
+
]);
431
+
await applyEvents(
432
+
db,
433
+
[
434
+
makeEvent({
435
+
uri: rsvpUri,
436
+
did,
437
+
collection: "community.lexicon.calendar.rsvp",
438
+
rkey: "r1",
439
+
record: { subject: { uri: eventUri }, status: "going" },
440
+
time_us: 2000000,
441
+
}),
442
+
],
443
+
TEST_CONFIG
444
+
);
445
+
446
+
// Verify count is 1
447
+
let result = await queryRecords(db, TEST_CONFIG, {
448
+
collection: "community.lexicon.calendar.event",
449
+
});
450
+
expect(result.records[0].counts?.["community.lexicon.calendar.rsvp"]).toBe(1);
451
+
452
+
// Notify with RSVP URI — PDS returns 404 (deleted)
453
+
await seedIdentity(did, "https://pds.example.com");
454
+
mockFetch({});
455
+
456
+
const res = await app.request(endpoint, {
457
+
method: "POST",
458
+
headers: { "Content-Type": "application/json" },
459
+
body: JSON.stringify({ uri: rsvpUri }),
460
+
});
461
+
462
+
const body = await res.json();
463
+
expect(body.deleted).toBe(1);
464
+
465
+
// Count should be back to 0
466
+
result = await queryRecords(db, TEST_CONFIG, {
467
+
collection: "community.lexicon.calendar.event",
468
+
});
469
+
expect(result.records[0].counts?.["community.lexicon.calendar.rsvp"] ?? 0).toBe(0);
470
+
});
471
+
});
+321
tests/search.test.ts
+321
tests/search.test.ts
···
1
+
import { describe, it, expect, beforeEach } from "vitest";
2
+
import type { Database, ContrailConfig } from "../src/core/types";
3
+
import { createTestDb, makeEvent } from "./helpers";
4
+
import { initSchema } from "../src/core/db/schema";
5
+
import { applyEvents, queryRecords } from "../src/core/db/records";
6
+
7
+
const SEARCH_CONFIG: ContrailConfig = {
8
+
namespace: "com.example",
9
+
collections: {
10
+
"community.lexicon.calendar.event": {
11
+
queryable: {
12
+
mode: {},
13
+
name: {},
14
+
description: {},
15
+
startsAt: { type: "range" },
16
+
},
17
+
// searchable omitted → auto-detect non-range: mode, name, description
18
+
},
19
+
"test.explicit.collection": {
20
+
queryable: {
21
+
title: {},
22
+
body: {},
23
+
category: {},
24
+
},
25
+
searchable: ["title", "body"], // explicit: only title and body
26
+
},
27
+
"test.disabled.collection": {
28
+
queryable: {
29
+
name: {},
30
+
},
31
+
searchable: false, // disabled
32
+
},
33
+
},
34
+
};
35
+
36
+
let db: Database;
37
+
38
+
beforeEach(async () => {
39
+
db = createTestDb();
40
+
await initSchema(db, SEARCH_CONFIG);
41
+
});
42
+
43
+
describe("FTS auto-detect (searchable omitted)", () => {
44
+
const collection = "community.lexicon.calendar.event";
45
+
46
+
beforeEach(async () => {
47
+
await applyEvents(
48
+
db,
49
+
[
50
+
makeEvent({
51
+
uri: "at://did:plc:a/community.lexicon.calendar.event/1",
52
+
did: "did:plc:a",
53
+
collection,
54
+
rkey: "1",
55
+
record: { name: "Rust Meetup", mode: "in-person", description: "A gathering of Rustaceans" },
56
+
time_us: 3000,
57
+
}),
58
+
makeEvent({
59
+
uri: "at://did:plc:b/community.lexicon.calendar.event/2",
60
+
did: "did:plc:b",
61
+
collection,
62
+
rkey: "2",
63
+
record: { name: "TypeScript Workshop", mode: "online", description: "Learn advanced TypeScript" },
64
+
time_us: 2000,
65
+
}),
66
+
makeEvent({
67
+
uri: "at://did:plc:c/community.lexicon.calendar.event/3",
68
+
did: "did:plc:c",
69
+
collection,
70
+
rkey: "3",
71
+
record: { name: "Rust and TypeScript", mode: "hybrid", description: "Best of both worlds" },
72
+
time_us: 1000,
73
+
}),
74
+
],
75
+
SEARCH_CONFIG
76
+
);
77
+
});
78
+
79
+
it("finds records matching a search term", async () => {
80
+
const result = await queryRecords(db, SEARCH_CONFIG, {
81
+
collection,
82
+
search: "Rust",
83
+
});
84
+
expect(result.records).toHaveLength(2);
85
+
const names = result.records.map((r) => JSON.parse(r.record!).name);
86
+
expect(names).toContain("Rust Meetup");
87
+
expect(names).toContain("Rust and TypeScript");
88
+
});
89
+
90
+
it("searches across multiple fields", async () => {
91
+
const result = await queryRecords(db, SEARCH_CONFIG, {
92
+
collection,
93
+
search: "Rustaceans",
94
+
});
95
+
expect(result.records).toHaveLength(1);
96
+
expect(JSON.parse(result.records[0].record!).name).toBe("Rust Meetup");
97
+
});
98
+
99
+
it("returns nothing for non-matching search", async () => {
100
+
const result = await queryRecords(db, SEARCH_CONFIG, {
101
+
collection,
102
+
search: "Python",
103
+
});
104
+
expect(result.records).toHaveLength(0);
105
+
});
106
+
107
+
it("supports prefix search", async () => {
108
+
const result = await queryRecords(db, SEARCH_CONFIG, {
109
+
collection,
110
+
search: "Type*",
111
+
});
112
+
expect(result.records).toHaveLength(2);
113
+
});
114
+
115
+
it("combines search with filters", async () => {
116
+
const result = await queryRecords(db, SEARCH_CONFIG, {
117
+
collection,
118
+
search: "Rust",
119
+
filters: { mode: "in-person" },
120
+
});
121
+
expect(result.records).toHaveLength(1);
122
+
expect(JSON.parse(result.records[0].record!).name).toBe("Rust Meetup");
123
+
});
124
+
125
+
it("does not search range fields (startsAt)", async () => {
126
+
// startsAt is range, so not included in FTS. Searching for its value should not match.
127
+
await applyEvents(
128
+
db,
129
+
[
130
+
makeEvent({
131
+
uri: "at://did:plc:d/community.lexicon.calendar.event/4",
132
+
did: "did:plc:d",
133
+
collection,
134
+
rkey: "4",
135
+
record: { name: "Date Event", startsAt: "2026-04-01T10:00:00Z", mode: "online", description: "Nothing special" },
136
+
time_us: 500,
137
+
}),
138
+
],
139
+
SEARCH_CONFIG
140
+
);
141
+
// "T10" would appear in the startsAt value but not in any searchable field
142
+
const result = await queryRecords(db, SEARCH_CONFIG, {
143
+
collection,
144
+
search: "T10",
145
+
});
146
+
expect(result.records).toHaveLength(0);
147
+
});
148
+
});
149
+
150
+
describe("FTS sync", () => {
151
+
const collection = "community.lexicon.calendar.event";
152
+
153
+
it("updates FTS on record update", async () => {
154
+
await applyEvents(
155
+
db,
156
+
[
157
+
makeEvent({
158
+
uri: "at://did:plc:a/community.lexicon.calendar.event/1",
159
+
collection,
160
+
rkey: "1",
161
+
record: { name: "Old Name", mode: "online", description: "test" },
162
+
time_us: 1000,
163
+
}),
164
+
],
165
+
SEARCH_CONFIG
166
+
);
167
+
168
+
// Update the record
169
+
await applyEvents(
170
+
db,
171
+
[
172
+
makeEvent({
173
+
uri: "at://did:plc:a/community.lexicon.calendar.event/1",
174
+
collection,
175
+
rkey: "1",
176
+
record: { name: "New Name", mode: "online", description: "test" },
177
+
operation: "update",
178
+
time_us: 2000,
179
+
}),
180
+
],
181
+
SEARCH_CONFIG
182
+
);
183
+
184
+
const oldResult = await queryRecords(db, SEARCH_CONFIG, { collection, search: "Old" });
185
+
expect(oldResult.records).toHaveLength(0);
186
+
187
+
const newResult = await queryRecords(db, SEARCH_CONFIG, { collection, search: "New" });
188
+
expect(newResult.records).toHaveLength(1);
189
+
});
190
+
191
+
it("removes from FTS on delete", async () => {
192
+
await applyEvents(
193
+
db,
194
+
[
195
+
makeEvent({
196
+
uri: "at://did:plc:a/community.lexicon.calendar.event/1",
197
+
collection,
198
+
rkey: "1",
199
+
record: { name: "Deletable", mode: "online", description: "test" },
200
+
time_us: 1000,
201
+
}),
202
+
],
203
+
SEARCH_CONFIG
204
+
);
205
+
206
+
await applyEvents(
207
+
db,
208
+
[
209
+
makeEvent({
210
+
uri: "at://did:plc:a/community.lexicon.calendar.event/1",
211
+
collection,
212
+
rkey: "1",
213
+
operation: "delete",
214
+
record: { name: "Deletable", mode: "online", description: "test" },
215
+
time_us: 2000,
216
+
}),
217
+
],
218
+
SEARCH_CONFIG
219
+
);
220
+
221
+
const result = await queryRecords(db, SEARCH_CONFIG, { collection, search: "Deletable" });
222
+
expect(result.records).toHaveLength(0);
223
+
});
224
+
});
225
+
226
+
describe("explicit searchable fields", () => {
227
+
const collection = "test.explicit.collection";
228
+
229
+
beforeEach(async () => {
230
+
await applyEvents(
231
+
db,
232
+
[
233
+
makeEvent({
234
+
uri: "at://did:plc:a/test.explicit.collection/1",
235
+
did: "did:plc:a",
236
+
collection,
237
+
rkey: "1",
238
+
record: { title: "Interesting Article", body: "Some content here", category: "tech" },
239
+
time_us: 1000,
240
+
}),
241
+
],
242
+
SEARCH_CONFIG
243
+
);
244
+
});
245
+
246
+
it("searches in explicitly listed fields", async () => {
247
+
const result = await queryRecords(db, SEARCH_CONFIG, { collection, search: "Interesting" });
248
+
expect(result.records).toHaveLength(1);
249
+
250
+
const result2 = await queryRecords(db, SEARCH_CONFIG, { collection, search: "content" });
251
+
expect(result2.records).toHaveLength(1);
252
+
});
253
+
254
+
it("does not search non-listed fields", async () => {
255
+
// "tech" is in category, which is not in searchable
256
+
const result = await queryRecords(db, SEARCH_CONFIG, { collection, search: "tech" });
257
+
expect(result.records).toHaveLength(0);
258
+
});
259
+
});
260
+
261
+
describe("searchable: false", () => {
262
+
const collection = "test.disabled.collection";
263
+
264
+
it("search param is ignored when FTS is disabled", async () => {
265
+
await applyEvents(
266
+
db,
267
+
[
268
+
makeEvent({
269
+
uri: "at://did:plc:a/test.disabled.collection/1",
270
+
did: "did:plc:a",
271
+
collection,
272
+
rkey: "1",
273
+
record: { name: "Should Not Be Searchable" },
274
+
time_us: 1000,
275
+
}),
276
+
],
277
+
SEARCH_CONFIG
278
+
);
279
+
280
+
// Search is a no-op — returns all records (no FTS join)
281
+
const result = await queryRecords(db, SEARCH_CONFIG, { collection, search: "Searchable" });
282
+
expect(result.records).toHaveLength(1); // returned because no FTS filtering applied
283
+
});
284
+
});
285
+
286
+
describe("search pagination", () => {
287
+
const collection = "community.lexicon.calendar.event";
288
+
289
+
beforeEach(async () => {
290
+
const events = Array.from({ length: 5 }, (_, i) =>
291
+
makeEvent({
292
+
uri: `at://did:plc:a/community.lexicon.calendar.event/e${i}`,
293
+
did: "did:plc:a",
294
+
collection,
295
+
rkey: `e${i}`,
296
+
record: { name: `Rust Event ${i}`, mode: "online", description: "test" },
297
+
time_us: (i + 1) * 1000,
298
+
})
299
+
);
300
+
await applyEvents(db, events, SEARCH_CONFIG);
301
+
});
302
+
303
+
it("paginates search results", async () => {
304
+
const page1 = await queryRecords(db, SEARCH_CONFIG, {
305
+
collection,
306
+
search: "Rust",
307
+
limit: 3,
308
+
});
309
+
expect(page1.records).toHaveLength(3);
310
+
expect(page1.cursor).toBeDefined();
311
+
312
+
const page2 = await queryRecords(db, SEARCH_CONFIG, {
313
+
collection,
314
+
search: "Rust",
315
+
limit: 3,
316
+
cursor: page1.cursor,
317
+
});
318
+
expect(page2.records).toHaveLength(2);
319
+
expect(page2.cursor).toBeUndefined();
320
+
});
321
+
});