Version Packages

fix(search): make FTS sync idempotent to stop duplicate rows on backfill

The idempotent FTS sync returned early when buildFtsContent produced no
content, skipping the delete. An update that cleared every searchable field
therefore left the prior FTS row in place, so old terms kept matching through
the search JOIN. Run the delete unconditionally and gate only the re-insert on
content.

buildFtsStatements only deleted the existing FTS row before inserting when the record was already in existingMap. Backfill runs with skipReplayDetection, leaving existingMap empty, so re-applied records looked new and appended duplicate FTS rows. The FTS5 virtual table has no uniqueness constraint, so duplicates accumulated and the search JOIN fanned each event out into one row per duplicate, breaking keyed lists in the appview. Delete-then-insert is now unconditional.

Version Packages

A collection can be keyed directly by its NSID, with the `collection` field
omitted (the map key is itself the NSID). That shape was only
half-supported: records landed in the records table, but the rest of the
pipeline silently skipped these collections.

- Config: make CollectionConfig.collection optional. resolveConfig
normalizes an omitted value to the map key once; validateConfig accepts
NSID-keyed entries (dotted keys with no collection field). Every
collection-list and lookup helper resolves `collection ?? key`, so
behavior is correct on both raw and resolved configs.
- Ingest: getCollectionNsids / getDiscoverableNsids / getDependentNsids no
longer return undefined, so Jetstream subscription and backfill now
include NSID-keyed collections, and notify resolves shortNameForNsid
instead of rejecting "collection not tracked".
- Search: buildFtsStatements and lookupExistingRecords resolve the storage
key the same way, so FTS is populated and replay/update detection works
for NSID-keyed collections (previously full-text search stayed empty for
them).
- A resolveCollectionKey(config, nsid) helper in contrail-base returns the
storage key (short alias, or the NSID itself when keyed directly), or
null when unknown, de-duplicating the fallback the insert path inlined.

Every change is a type-widening or a fallback that only fires when
`collection` is absent, so existing alias-keyed configs are unchanged.

Tests: config-layer unit tests (resolveConfig / validateConfig for
NSID-keyed entries), the resolve-collection-key helper, an end-to-end
notify-to-ingest test, and the existing search.test.ts (which uses an
NSID-keyed config and was the failing regression). Full suite green.

Changeset included.

@atcute/jetstream rolls the cursor back 10s on first connect when given an
array url, to absorb clock skew across a randomly-selected pool of
instances. It is meant to fire once per session. But contrail's cron
ingestion rebuilds the JetstreamSubscription every cycle, so for a
single-instance config that "first-connect" rollback fires every tick and
redundantly re-delivers the last 10s of events each cycle.

Add a jetstreamUrlOption(jetstreams) helper in contrail-base that maps the
configured list onto @atcute's url shape by topology:

- one instance -> a string (one fixed instance, no skew, no rollback)
- a real pool (2+) -> an array (cross-instance rollback preserved)

Applied at both construction sites (the cron ingestEvents path and the
runPersistent daemon). Setting two jetstreams does not sidestep this: an
array keeps the per-cycle rollback, and with genuine cross-instance skew
that rollback is then actually needed. A string for the single-instance
case is the only shape that removes the redundant re-ingest.

Also fix a misleading per-cycle reconnect log that claimed every reconnect
picks a URL at random and rolls back 10s, which is only true for a
multi-instance pool. The warning is now gated on urls.length > 1 and
reports the actual rolled_back value; single-instance reconnects log at
info level.

Tests: a helper unit test (single->string, pool->array, empty->empty), an
ingestEvents wiring test, and a characterization test that drives the real
JetstreamSubscription (mocking only the injected socket) to pin the
array-rolls-back / string-does-not behavior the fix relies on, so a future
@atcute change fails loudly here.

The feed-items prune ran on every ingest tick (cron) and every sweep
interval (persistent), walking every actor to issue a cutoff DELETE that
almost always deleted nothing. On an operated deployment this no-op sweep
was the #1 D1 query by count and runtime (~825k/day, reading ~6.3M rows to
delete ~0). It is not an indexing problem; the cost is running the sweep
too often.

A feed can only exceed its cap right after a feed-mutating record is applied
(an event fanning out to followers, or a follow backfilling a feed). Gate
the sweep on that:

- Cron (runIngestCycle): sweep only when the tick ingested a feed-mutating
collection, or a 6h recovery interval has elapsed. The recovery clock is
persisted in _contrail_meta since the isolate recycles each tick.
- Persistent (streamAndFlush): a state.feedDirty flag is set when a flushed
batch contains a feed-mutating record; the sweep is gated on feedDirty or
the recovery interval.
- getFeedMutatingNsids(config) derives the feed-mutating NSID set from the
configured feed targets and follow collections, so the gate cannot drift
from what actually writes feed_items.
- Recovery completes a full pass per interval (tracking the last completed
pass, not a single slice) and runs on no-op notify calls, so over-cap rows
from a lowered cap or bulk import still drain with no live activity.

Idle deployments drop from a sweep every tick (~1,440/day on a 1-min cron)
to ~4/day; correctness is unchanged (feeds are caches, and the recovery
interval bounds how long a feed can stay briefly over cap).

fix(contrail-appview): persist ingest cursor before identity-refresh tail

saveCursor ran after refreshStaleIdentities in runIngestCycle. The
identity refresh makes per-DID network calls and can run long; if the
ingest isolate was aborted before the save (e.g. a scheduled-invocation
deadline), the cursor never advanced and the next cycle re-drained the
same jetstream window indefinitely.

Records are durably applied before this point, so the cursor's forward
progress is committed first. Identity refresh is idempotent and
staleness-driven, so deferring it past the save is safe.

Version Packages

feat: write-only `sinks` for applied records

fix(ingest): bound the ingest cycle by its safety timeout

A Sink observes every applied record to build derived state (search
index, audit log, webhook) without masquerading as a realtime pubsub.
Fanned out inside applyEvents() after commit, on both the live and
backfill paths, with one deduplicated RecordEvent per record and
isolated failures. Distinct from realtime: no subscribers, no ticket
secret, fires on backfill. Public records only; purely additive.

Add regression tests locking the ingestEvents loop behavior reviewed for the
quiet-stream fix:
- caught-up break collects the batch + cursor when a kept event reaches the
live edge (the exit path neither existing test exercised)
- the caught-up check now fires on a filtered live event too (early return),
deferring a following kept event to the next cycle rather than collecting it
- fast-flowing events within the deadline are all collected (the
next()/timeout race drops nothing)
- #identity events surface as handle updates through the ingest path

ingestEvents drove the Jetstream subscription with `for await` and checked its
exit conditions (caught-up and safety timeout) only inside the loop body, after
an event arrived. On a low-traffic stream the async iterator blocks forever once
history is replayed: the safety timeout never fires, the caller's hard timeout
kills the cycle (on Workers: "Exceeded CPU Limit"), and the collected batch and
cursor are never written. The per-event filter paths (unknown DID, subject
filter, recordFilter) used `continue`, which also skipped the exit checks, so a
flood of all-filtered events ran past the deadline too.

Drive the iterator manually: check the deadline before awaiting each event, race
iterator.next() against the remaining deadline so a silent stream can't outlast
the safety timeout, and run the caught-up and deadline checks regardless of
whether the event was filtered. Close the subscription socket best-effort on
exit (not awaited, since awaiting a quiet stream's close could reintroduce the
hang).

Add tests covering the quiet-stream and all-filtered-flood cases.

Version Packages

getRecord: allow passing in handle in at uri

Version Packages

performance: gate tables init, optimize filtered queries with multipl…

Version Packages

fix: avoid race condition while resolving PDS

# Conflicts:
# packages/contrail/src/core/backfill.ts

Version Packages

Following feeds fix unbounded pruning

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* feat(contrail-community): provision schema, types, adapter CRUD

* feat(contrail-community): PDS + PLC + service-auth helpers

* feat(contrail-community): provision orchestrator + XRPC route + session cache

* feat(contrail-community): reap CLI for stuck provisioning rows

* test+docs(contrail): provision e2e, docs, changeset, deps

* fix(contrail-community): harden provision allowlist + reap (PR #31 review)

Security/review follow-ups on the community DID provisioning feature:

- Provisioning fails closed: with allowProvisioning=true, an empty/undefined
allowlist no longer accepts any caller pdsEndpoint (fail-open). Require a
non-empty allowlist, or the explicit, loud allowAnyProvisionPdsEndpoint.
Renamed allowedPdsEndpoints -> allowedProvisionPdsEndpoints to scope it to
provisioning (Contrail still reads/indexes from every PDS).

- reap --all-stuck gains a mandatory age floor (listStuckAttempts(olderThanMs)
+ --older-than <minutes>, default 30) so a bulk run can't tombstone an
in-flight provision. --attempt-id still targets a known row regardless.

- reap is reachable: ship it as a contrail-community bin (under pnpm the core
contrail CLI can't resolve the community package; the dynamic import there
now logs instead of an empty catch). Adds Postgres support via
--db/DATABASE_URL alongside the D1 binding.

- archiveStuckAttempt is now atomic (db.batch) + idempotent (ON CONFLICT DO
NOTHING), so a retry after a partial failure can't PK-conflict.

- Minor: export bytesToB64url from plc and reuse in service-auth; drop the
duplicate reap mutual-exclusion validator.

mint-route allowlist gap (pre-existing on main) tracked separately in om-vyok.

* test(contrail-community): drop tautological builder-shape assertions

Remove the 'buildTombstoneOp produces the expected shape' case and the two
pass-through type/prev assertions in the update-op test: both assert literals
the builder just set or args just passed in, exercising object-literal
semantics rather than runtime behavior. The real signing behavior
(signTombstoneOp / signUpdateOp base64url sig) and the live-PLC CID round-trip
remain covered.

* docs(contrail-community): document SSRF exposure of allowAnyProvisionPdsEndpoint

The accept-any escape hatch bypasses endpoint validation: the caller-supplied
pdsEndpoint reaches describeServer + createAccount fetches with no
private-IP/metadata guard. Note inline that it must be paired with trusted auth
+ egress network policy + IMDSv2; the default allowlist path contains this by
construction. No behavior change.

* feat(contrail-base): networkOverrides config for private-network deployments

Add optional ContrailConfig.networkOverrides with three subfields:
plcUrl, slingshotUrl, additionalAllowedHosts. Plumb config through
resolvePDS, getPDSViaDidDoc (inlined; module-level didResolver removed),
getPDS, resolvePDSCached, getClient, and all five identity.ts functions
(fetchAndSave, resolveIdentity, resolveIdentities, resolveActor,
refreshStaleIdentities). Pass config through backfill.ts call sites
(getClient + getPDS) and the persistent.ts:160 refreshStaleIdentities
call. additionalAllowedHosts is exact + case-insensitive + port-agnostic;
the default SSRF validator still runs for non-listed hosts.

All changes are additive — omitting networkOverrides preserves current
public-internet defaults. validatePdsUrl remains internal (no new
exports beyond the networkOverrides field itself).

* test(contrail-base): network overrides + SSRF regression coverage

Add three test files exercising the networkOverrides plumbing:
- client.test.ts: ContrailConfig.networkOverrides type, validatePdsUrl
regression baseline (public HTTPS accepted; non-HTTPS + private CIDRs
+ localhost + link-local rejected), additionalAllowedHosts allowlist
(allowed host accepted; non-listed hosts still rejected; port-agnostic
match), getClient + getPDS + resolvePDS plumb-through with mocked fetch.
- identity-config.test.ts: resolveIdentity, resolveIdentities, resolveActor,
refreshStaleIdentities all route to override slingshot URL.
- network-overrides.test.ts: integration test against node:http stub PLC
and Slingshot servers. Asserts resolvePDS, getClient, and
refreshStaleIdentities actually hit the configured override URLs end to
end, exercising the full chain.

The SSRF regression cases give the existing validator implicit baseline
explicit coverage in this PR.

* feat(contrail-base): apply networkOverrides.resolver to spaces credential verifier

* feat(contrail-appview): apply networkOverrides to label-endpoint resolution

* fix(contrail-appview): use IF NOT EXISTS on schema migrations to remove init race

Concurrent `initSchema` calls previously raced on `ALTER TABLE ADD COLUMN`
because the SQL had no `IF NOT EXISTS` clause, and the two consumer-site
`try { ... } catch { /* Column already exists — ignore */ }` blocks silently
swallowed *every* DDL error — masking missing tables, syntax errors, and
type mismatches alongside the intended duplicate-column case.

L3 replaces both swallows with a dialect-aware `addColumnIfNotExists` helper:
- Postgres: emits `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` (atomic in PG).
- SQLite: pre-checks `PRAGMA table_info`, then narrowly absorbs only the
literal "duplicate column name" error from the still-possible PRAGMA->ALTER
race. Every other error surfaces.

The `MIGRATIONS` array is now structured `{ table, column, columnDef, target }`
data routed through the same helper. `target: "spaces"` routes to `spacesDb`
when one is configured (split-DB deployments), `target: "feeds"` is gated on
feeds being configured so the absent `feed_backfills` table no longer
silently fails.

Tests at `packages/contrail/tests/schema-idempotency.test.ts` cover:
- sequential and concurrent (3-way) `initSchema` calls
- repeated init does not leave duplicate count columns
- helper is a no-op when column already exists
- helper surfaces "no such table" errors (the L3-removed swallow case)
- `initSchema` propagates errors thrown from extension schema modules

OpenMeet's consumer-side Postgres advisory lock around `contrail.init()`
exists only to work around this race; with L3 in place that lock can be
deleted in a follow-up.

* test: L3 concurrent CREATE race on Postgres

* chore: add changeset for private-network deployment support

* fix(contrail-appview): thread networkOverrides config through all resolver/identity call sites

The networkOverrides config param was threaded into the resolver/identity
functions but dropped at ~8 appview call sites (live-ingest refresh cycle,
on-demand refresh, and router getProfile/getFeed/collection/profiles/notify),
so private-network deploys silently fell back to the public resolver and the
un-widened SSRF guard on those paths.

Pass the in-scope config at each site. Add a regression test driving an
override config through the live-ingest refresh path (runIngestCycle) and a
router actor-resolution path (getProfile), asserting the override slingshot
is hit instead of the default.

* refactor(contrail-base): dedup SSRF validator into shared validateExternalUrl

labels/resolve.ts validateEndpointUrl duplicated contrail-base validatePdsUrl,
and this PR had edited the additionalAllowedHosts allowlist into both copies —
a future SSRF-rule fix applied to one copy would leave the other exploitable.

Export a single validateExternalUrl(url, allowedHosts?) from contrail-base and
consume it from both the PDS client and labeler-endpoint resolution.
validateEndpointUrl stays exported as a thin alias for backward compat.
Behavior is identical (covered by existing SSRF tests).

Also document the PDS resolve cache's process-wide-config assumption (keyed by
DID only; safe now that all callers thread the same config).

* docs(contrail-base): document validateExternalUrl threat-model scope

Note inline that the SSRF guard is best-effort literal-match only: no DNS
resolution, partial IPv6 / encoded-IP coverage, egress network policy
expected for fully untrusted resolver inputs. No behavior change.

another big refactor

I noticed about 1/4 of identities in my app did not have a handle after
backfill. Looks like getPDS over dids run twice over the same set of
dids creates race condition. One is from this loop, and the other getPDS
is invoked from getClient function.

I tested with my set of data and edited variant always gives 100%
of handles in identities table.

fix(spaces): owner-signed enroll, enrollment-only default binding, declaration validation

merge: resolve conflicts with main

Two trust assumptions in the binding layer cannot be closed at the
Contrail layer alone:

- DID-doc service-entry rebind: deployments using
createDidDocBindingResolver inherit PLC's rotation-key authorization
model. Any rotation key on the owner's account can rewrite the
#atproto_space_authority service entry.

- PDS app password held for managed communities: community.provision
and community.adopt hold unscoped app passwords that can write any
record on the user-owned PDS, including the binding declaration
read by createPdsBindingResolver.

Both are constraints of their respective binding sources and
onboarding flows, not bugs. Document each so operators picking a
binding strategy can size them against their own threat model, with
pointers to the canonical issues for the upstream/protocol-level
work that would close them: #38 (DID-doc), #39 (PDS app password).

The PDS binding resolver trusted body.value.authority on any record
returned for the space's URI, with only a startsWith('did:') sniff. A
malformed or attacker-shaped record at the same NSID could rebind
authority — record content alone, no signature verification.

Validate before trusting:
- $type must equal the URI's type (the space-type lexicon embeds the
declaration fields per tools.atmo.space.declaration's contract)
- createdAt must be a string
- authority must match a strict did:plc | did:web regex

Fail closed on any deviation. Adds three negative tests covering each
deviation path.

The default in-process credential verifier composed
[Enrollment, Local], so a never-enrolled space still resolved its
authority via LocalBindingResolver — collapsing the host-side consent
layer the enrollment table is meant to provide.

Default to enrollment-only. createLocalBindingResolver remains exported
for explicit single-tenant wiring via options.credentialVerifier.

The recordHost.enroll endpoint accepted either an owner-signed call OR
a self-attested authority call. The latter let any DID claim to be the
authority for any space's URI and rebind that space's authority.

Drop the self-attesting-authority branch. Require the caller to be the
space owner (sa.issuer === parts.ownerDid). The owner is still free to
designate any authority via body.authority — only the legitimacy of the
caller changes.

Updates the existing test that codified the old behavior to assert the
new rejection, and adds a positive test for the canonical owner-driven
split-deployment enrollment flow.

Main added record-filter, identity-events, follow-feed, and lex-gen-fix
features in `packages/contrail/src/core/*` that PR #30 had reorganized
into the new package split. Each conflict resolved by:

1. Keeping PR #30's re-export shims in `packages/contrail/src/core/*`
(these files now `export * from "@atmo-dev/contrail-appview"`)
2. Porting main's additions to the corresponding files in
`packages/contrail-appview/src/core/*`, which is the new home of the
actual implementation.

Files re-shimmed:
- backfill.ts, db/records.ts, db/schema.ts, identity.ts, jetstream.ts,
persistent.ts, router/feed.ts, types.ts

Ports landed in contrail-appview/src/core (or contrail-base where types
and identity helpers actually live):
- backfill.ts, db/records.ts, db/schema.ts, jetstream.ts, persistent.ts,
router/feed.ts (+1.0K lines from main)
- constellation.ts (new 181-line module)
- contrail-base/src/types.ts: FeedTargetConfig, ConstellationConfig,
buildFeedTargetCaps, normalizeFeedTarget, feedTargetMaxItems,
CollectionConfig.{timeField, subjectField, recordFilter}, and
resolveConfig defaulting `discover: false` for `app.bsky.*`
- contrail-base/src/identity.ts: applyIdentityEvent (handle change handler)

Test mock for `persistent.test.ts` updated to mock
`@atmo-dev/contrail-base` (post-split home of identity helpers) and
include both `refreshStaleIdentities` and `applyIdentityEvent`.

Test status post-merge: 314 passed, 7 failed in
`packages/contrail/tests/search.test.ts` — these 7 failures pre-exist on
plain `origin/main` and are unrelated to the merge.

No behavior changes intended beyond what main and PR #30 already deliver.

Version Packages

test(contrail-e2e): adopt post-PR30 spaces split + community integration

Bring the e2e suite back to green under the package split:

- makeSpacesConfig() wraps the old flat { type, serviceDid, resolver }
shape under authority / recordHost and generates a fresh signing key
per call. The config validator now requires spaces.authority whenever
community is set, so this is the minimum viable shape for any test
that uses spaces.
- setupCommunityContrail() wires the community module via
createCommunityIntegration({ db, config: resolveConfig(...) }) and
passes the result as communityIntegration to the Contrail constructor.
The community config field is now opaque to contrail core; routes are
registered through the integration option.
- 7 test files updated to use the helpers; 4 community tests pick up a
workspace dep on @atmo-dev/contrail-community.

35/35 e2e tests pass against devnet. No source-package changes.

Version Packages