atproto pds in zig
pds.zat.dev
pds
atproto
docs: note getRepo export concerns
+27
+27
GETREPO_NOTES.md
+27
GETREPO_NOTES.md
···
1
+
# getRepo notes
2
+
3
+
Context: on 2026-06-29, operators called out `com.atproto.sync.getRepo` as an expensive route for PDS hosts. The main concerns were compression cost, large repo export reads, avoiding disclosure of stale/deleted blocks, and isolating backup/backfill traffic from user-facing work.
4
+
5
+
ZDS is directly relevant because it serves `com.atproto.sync.getRepo` from `src/atproto/sync.zig`.
6
+
7
+
Current shape:
8
+
9
+
- ZDS serves raw CAR bytes and does not appear to add application-layer gzip, so it avoids the Node compression issue described in the operator thread.
10
+
- Full repo export is built from SQLite `repo_blocks`, ordered by CID.
11
+
- Incremental export with `since` filters by `repo_rev`.
12
+
- Repo writes already use lazy MST loading backed by `repo_blocks`, so ZDS is closer to a Hubble-style block/CAR-serving shape than an eager full-repo rebuild path.
13
+
- `subscribeRepos` has a connection cap, but `getRepo` does not appear to have route-specific backpressure.
14
+
15
+
Open questions:
16
+
17
+
- Should full `getRepo` walk the current commit/MST and include only reachable blocks, instead of exporting every retained block for the DID?
18
+
- Do retained old MST or record blocks make the current full export disclose deleted or stale record data?
19
+
- Should `getRepo` have route-specific concurrency or rate limits separate from normal public reads?
20
+
- Should the benchmark suite compare the current range-scan export against a reachability-based export for small, medium, and large repos?
21
+
22
+
Suggested next pass:
23
+
24
+
1. Add a regression test for delete/update followed by full `getRepo`, checking whether stale record blocks are exported.
25
+
2. If stale blocks are visible, change full export to current-root reachability rather than whole-DID block scan.
26
+
3. Add lightweight route backpressure for full `getRepo`.
27
+
4. Re-run `bench repo-size` and keep raw range-scan numbers separate from reachable-export numbers.