at main
6 folders
6 files
fix(contrail): ingest and index NSID-keyed collections
A collection can be keyed directly by its NSID, with the `collection` field
omitted (the map key is itself the NSID). That shape was only
half-supported: records landed in the records table, but the rest of the
pipeline silently skipped these collections.
- Config: make CollectionConfig.collection optional. resolveConfig
normalizes an omitted value to the map key once; validateConfig accepts
NSID-keyed entries (dotted keys with no collection field). Every
collection-list and lookup helper resolves `collection ?? key`, so
behavior is correct on both raw and resolved configs.
- Ingest: getCollectionNsids / getDiscoverableNsids / getDependentNsids no
longer return undefined, so Jetstream subscription and backfill now
include NSID-keyed collections, and notify resolves shortNameForNsid
instead of rejecting "collection not tracked".
- Search: buildFtsStatements and lookupExistingRecords resolve the storage
key the same way, so FTS is populated and replay/update detection works
for NSID-keyed collections (previously full-text search stayed empty for
them).
- A resolveCollectionKey(config, nsid) helper in contrail-base returns the
storage key (short alias, or the NSID itself when keyed directly), or
null when unknown, de-duplicating the fallback the insert path inlined.
Every change is a type-widening or a fallback that only fires when
`collection` is absent, so existing alias-keyed configs are unchanged.
Tests: config-layer unit tests (resolveConfig / validateConfig for
NSID-keyed entries), the resolve-collection-key helper, an end-to-end
notify-to-ingest test, and the existing search.test.ts (which uses an
NSID-keyed config and was the failing regression). Full suite green.
Changeset included.
feat(contrail): private-network deployment + concurrent-init hardening (#44)
* feat(contrail-base): networkOverrides config for private-network deployments
Add optional ContrailConfig.networkOverrides with three subfields:
plcUrl, slingshotUrl, additionalAllowedHosts. Plumb config through
resolvePDS, getPDSViaDidDoc (inlined; module-level didResolver removed),
getPDS, resolvePDSCached, getClient, and all five identity.ts functions
(fetchAndSave, resolveIdentity, resolveIdentities, resolveActor,
refreshStaleIdentities). Pass config through backfill.ts call sites
(getClient + getPDS) and the persistent.ts:160 refreshStaleIdentities
call. additionalAllowedHosts is exact + case-insensitive + port-agnostic;
the default SSRF validator still runs for non-listed hosts.
All changes are additive — omitting networkOverrides preserves current
public-internet defaults. validatePdsUrl remains internal (no new
exports beyond the networkOverrides field itself).
* test(contrail-base): network overrides + SSRF regression coverage
Add three test files exercising the networkOverrides plumbing:
- client.test.ts: ContrailConfig.networkOverrides type, validatePdsUrl
regression baseline (public HTTPS accepted; non-HTTPS + private CIDRs
+ localhost + link-local rejected), additionalAllowedHosts allowlist
(allowed host accepted; non-listed hosts still rejected; port-agnostic
match), getClient + getPDS + resolvePDS plumb-through with mocked fetch.
- identity-config.test.ts: resolveIdentity, resolveIdentities, resolveActor,
refreshStaleIdentities all route to override slingshot URL.
- network-overrides.test.ts: integration test against node:http stub PLC
and Slingshot servers. Asserts resolvePDS, getClient, and
refreshStaleIdentities actually hit the configured override URLs end to
end, exercising the full chain.
The SSRF regression cases give the existing validator implicit baseline
explicit coverage in this PR.
* feat(contrail-base): apply networkOverrides.resolver to spaces credential verifier
* feat(contrail-appview): apply networkOverrides to label-endpoint resolution
* fix(contrail-appview): use IF NOT EXISTS on schema migrations to remove init race
Concurrent `initSchema` calls previously raced on `ALTER TABLE ADD COLUMN`
because the SQL had no `IF NOT EXISTS` clause, and the two consumer-site
`try { ... } catch { /* Column already exists — ignore */ }` blocks silently
swallowed *every* DDL error — masking missing tables, syntax errors, and
type mismatches alongside the intended duplicate-column case.
L3 replaces both swallows with a dialect-aware `addColumnIfNotExists` helper:
- Postgres: emits `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` (atomic in PG).
- SQLite: pre-checks `PRAGMA table_info`, then narrowly absorbs only the
literal "duplicate column name" error from the still-possible PRAGMA->ALTER
race. Every other error surfaces.
The `MIGRATIONS` array is now structured `{ table, column, columnDef, target }`
data routed through the same helper. `target: "spaces"` routes to `spacesDb`
when one is configured (split-DB deployments), `target: "feeds"` is gated on
feeds being configured so the absent `feed_backfills` table no longer
silently fails.
Tests at `packages/contrail/tests/schema-idempotency.test.ts` cover:
- sequential and concurrent (3-way) `initSchema` calls
- repeated init does not leave duplicate count columns
- helper is a no-op when column already exists
- helper surfaces "no such table" errors (the L3-removed swallow case)
- `initSchema` propagates errors thrown from extension schema modules
OpenMeet's consumer-side Postgres advisory lock around `contrail.init()`
exists only to work around this race; with L3 in place that lock can be
deleted in a follow-up.
* test: L3 concurrent CREATE race on Postgres
* chore: add changeset for private-network deployment support
* fix(contrail-appview): thread networkOverrides config through all resolver/identity call sites
The networkOverrides config param was threaded into the resolver/identity
functions but dropped at ~8 appview call sites (live-ingest refresh cycle,
on-demand refresh, and router getProfile/getFeed/collection/profiles/notify),
so private-network deploys silently fell back to the public resolver and the
un-widened SSRF guard on those paths.
Pass the in-scope config at each site. Add a regression test driving an
override config through the live-ingest refresh path (runIngestCycle) and a
router actor-resolution path (getProfile), asserting the override slingshot
is hit instead of the default.
* refactor(contrail-base): dedup SSRF validator into shared validateExternalUrl
labels/resolve.ts validateEndpointUrl duplicated contrail-base validatePdsUrl,
and this PR had edited the additionalAllowedHosts allowlist into both copies —
a future SSRF-rule fix applied to one copy would leave the other exploitable.
Export a single validateExternalUrl(url, allowedHosts?) from contrail-base and
consume it from both the PDS client and labeler-endpoint resolution.
validateEndpointUrl stays exported as a thin alias for backward compat.
Behavior is identical (covered by existing SSRF tests).
Also document the PDS resolve cache's process-wide-config assumption (keyed by
DID only; safe now that all callers thread the same config).
* docs(contrail-base): document validateExternalUrl threat-model scope
Note inline that the SSRF guard is best-effort literal-match only: no DNS
resolution, partial IPv6 / encoded-IP coverage, egress network policy
expected for fully untrusted resolver inputs. No behavior change.
fix(contrail): ingest and index NSID-keyed collections
A collection can be keyed directly by its NSID, with the `collection` field
omitted (the map key is itself the NSID). That shape was only
half-supported: records landed in the records table, but the rest of the
pipeline silently skipped these collections.
- Config: make CollectionConfig.collection optional. resolveConfig
normalizes an omitted value to the map key once; validateConfig accepts
NSID-keyed entries (dotted keys with no collection field). Every
collection-list and lookup helper resolves `collection ?? key`, so
behavior is correct on both raw and resolved configs.
- Ingest: getCollectionNsids / getDiscoverableNsids / getDependentNsids no
longer return undefined, so Jetstream subscription and backfill now
include NSID-keyed collections, and notify resolves shortNameForNsid
instead of rejecting "collection not tracked".
- Search: buildFtsStatements and lookupExistingRecords resolve the storage
key the same way, so FTS is populated and replay/update detection works
for NSID-keyed collections (previously full-text search stayed empty for
them).
- A resolveCollectionKey(config, nsid) helper in contrail-base returns the
storage key (short alias, or the NSID itself when keyed directly), or
null when unknown, de-duplicating the fallback the insert path inlined.
Every change is a type-widening or a fallback that only fires when
`collection` is absent, so existing alias-keyed configs are unchanged.
Tests: config-layer unit tests (resolveConfig / validateConfig for
NSID-keyed entries), the resolve-collection-key helper, an end-to-end
notify-to-ingest test, and the existing search.test.ts (which uses an
NSID-keyed config and was the failing regression). Full suite green.
Changeset included.