# Architecture

The solution follows the [AT Protocol](https://atproto.com/articles/atproto-for-distsys-engineers) based on [Kappa Architecture](https://milinda.pathirage.org/kappa-architecture.com/). Data Repo services are canonical storage, Even Log services provide federation, View services index the network, and App services are how users interact with the system.

## End-to-end flow

```
Browser (SPA)
  │
  ├─[1] ATProto OAuth flow ──────────────────► ATProto Authorization Server
  │       (PKCE + DPoP, handled by library)         (user's PDS)
  │       Sessions stored in IndexedDB
  │
  ├─[2] com.atproto.repo.putRecord ────────► PDS  (site.standard.publication)
  │       (via session.fetchHandler, DPoP-bound)
  │
  ├─[3] com.atproto.repo.createRecord ────► PDS  (app.bsky.feed.post — sidecar)
  │       On publish: creates a link-card Bluesky post referencing the article.
  │       Returned { uri, cid } stored as bskyPostRef in the document record.
  │       (non-fatal; sidecar errors do not block article creation)
  │
  ├─[4] com.atproto.repo.putRecord ────────► PDS  (site.standard.document)
  │       (includes bskyPostRef if sidecar was created)
  │
  ├─[5] GET /xrpc/com.atproto.server.getServiceAuth ──► PDS
  │       (via session.fetchHandler, DPoP-bound)
  │       ◄── { token: <service JWT> }
  │
  └─[6] POST /posts ──────────────────────────► API Gateway + Lambda
          Authorization: Bearer <service JWT>        │
                                                     ├─ verify JWT sig via IdResolver
                                                     ├─ check aud == API_DID, exp
                                                     └─ PutItem to DynamoDB (index)

Engagement (signed-in users reading an article):
  │
  ├─[1] app.bsky.feed.getPostThread ──────► public.api.bsky.app  (unauthenticated)
  │       Fetches like/reply counts and nested replies for the sidecar post.
  │
  ├─[2] com.atproto.repo.createRecord ────► PDS  (app.bsky.feed.like)
  │       Like: creates a like record whose subject is the sidecar post { uri, cid }.
  │       Unlike: deleteRecord on the like's rkey.
  │
  └─[3] com.atproto.repo.createRecord ────► PDS  (app.bsky.feed.post, reply)
          Reply: creates a post with reply.root/parent pointing to the sidecar post.

ATProto Jetstream (external firehose)
  │
  └─[1] wss://jetstream2.us-east.bsky.network ──► Indexer (EC2)
          subscribe?wantedCollections=               │
            site.standard.publication                ├─ PutItem / DeleteItem (publications)
            site.standard.document                   ├─ PutItem / DeleteItem (documents)
            site.standard.graph.subscription         └─ PutItem / DeleteItem (subscriptions) + fan-in
                                                         → DynamoDB table `blogs`
```

ATProto records (publications, documents, subscriptions) are written directly to the user's own PDS — they own the data. DynamoDB holds a secondary index for fast querying by the API. The indexer populates that index asynchronously by tailing the Jetstream firehose.

## Infrastructure

Both local and production use **API Gateway V1 (REST) API**. V2 (HTTP) API has minimal features at a lower cost.

The indexer runs on an **EC2 instance in a public subnet** (no NAT gateway, which would cost more than the EC2 instance itself). The security group has zero inbound rules, so the public IP is unreachable from the internet. Outbound-only access is sufficient to consume the Jetstream WebSocket. Shell access is via **SSM Session Manager** — no open ports or key pair required.

## ACM certificate

The certificate covers `example.com` (apex) and `*.example.com` (wildcard SAN). The wildcard covers every subdomain with one certificate, so new subdomains never require a cert update.

**`*.example.com` does not match the bare apex** — both names are always required.

Theoretical downsides of a wildcard (not concerns here):
- **Blast radius on compromise** — a leaked private key covers all subdomains. With ACM the private key never leaves AWS, so this is moot.
- **Revocation scope** — revoking a wildcard cert takes down all subdomains at once. ACM auto-renews, so manual revocation is rare.
- **PCI-DSS / some compliance frameworks** — prohibit wildcards. Unlikely to apply here.
- **Single-level only** — `*.example.com` does not cover `foo.api.example.com`. Not a factor for this architecture.

## DynamoDB

Table name: `blogs`. All entities share the table in a [single-table design](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/data-modeling-schema-social-network.html).

PK  = Partition Key (hash)  
SK  = Sort Key (range)  
GSI = Global Secondary Index


| Entity | PK | SK | GSI1PK | GSI1SK | Notes |
|--------|----|----|--------|--------|-------|
| Publication | `u#<did>#pub` | `pub#<rkey>` | — | — | Written on `site.standard.publication` create/update/delete. |
| Post | `u#<did>#post` | `p#<rkey>` | `blog-post#published` | `<publishedAt>#<rkey>` | Written on `site.standard.document` create/update/delete. `GSI1SK` clamps `publishedAt` to index time to prevent future-dated records from dominating the feed. Optional attributes: `bskyPostRef` (`{ uri, cid }` of the sidecar Bluesky post), `description` (from the document's `description` field), `textContent` (plain-text excerpt, max 300 chars, derived from markdown content). |
| Subscription | `u#<readerDid>#subscriptions` | `sub#<rkey>` | — | — | Written when a reader subscribes. `authorDid` and `publicationAtUri` attributes identify the target publication. |
| Subscriber | `u#<authorDid>#subscribers` | `u#<readerDid>` | — | — | Written by the indexer/backfill on subscribe; used to enumerate an author's subscribers for fan-out. |
| Timeline item | `u#<readerDid>#timeline` | `<publishedAt>#<rkey>` | — | — | Fan-in/fan-out items written when a subscription is created or a new post is published. Deleted (all items for that author) when the subscription is removed. Expires via `ttl` (30 days). Carries `description` and `textContent` (max 300 chars) from the source post so the following feed can render excerpts without a secondary lookup. |
| Subscriber count | `u#<did>` | `count` | — | — | `subscribers` attribute incremented on each new subscription. Counted per DID, not per publication — if a reader subscribes to two publications by the same author, the count increments twice. Used by the API to identify celebrity authors (threshold: 1000) who bypass fan-out in favour of live post queries at read time. |
| User info | `u#<did>` | `info` | — | — | Written on first sign-in via `POST /users`. Attributes: `entityType` (`user-info`), `acceptedTermsAt` (ISO timestamp, set once), `updatedAt`. |
| Suggestion | `u#<did>#sugg` | `pub#<rkey>` | — | — | One item per suggested publication. `show` (boolean) controls visibility: `true` = active suggestion, `false` = dismissed tombstone. Attributes: `pubAtUri`, `authorDid`, `pubName`, `pubUrl`, `pubDescription`. Written by `POST /suggestions`; updated to `show=false` by `PATCH /suggestions`. Never deleted — tombstones prevent re-adding dismissed or subscribed-to publications on future "Find suggestions" runs. |
| iframely cache | `iframely#cache` | SHA-256 hex of the embed URL | — | — | Expires via `ttl` attribute (DynamoDB TTL enabled) |

Global secondary index `FeedIndex` (GSI1PK=`blog-post#published`, GSI1SK=`<publishedAt>#<rkey>`) enables a time-ordered global feed across all users.

The API lambda and the indexer both write post records. The API lambda handles writes from the authenticated owner (drafts, edits, visibility changes). The indexer handles writes from the public Jetstream firehose (published documents from any user). Both use `PutCommand` (upsert semantics).

## Following feed

The `/following` page shows a reader their personalised timeline. There is no fan-in at read time — it reads pre-written `timeline` items and merges them with a live celebrity query.

### Fan-out (new post published)

When the indexer sees a new `site.standard.document` commit on the firehose it checks the author's subscriber count (`u#<authorDid>` / `count`):

- **Below the celebrity threshold (1 000)** — the indexer queries `u#<authorDid>#subscribers` to enumerate every reader, then writes a `timeline` item into each reader's `u#<readerDid>#timeline` partition. Each item copies `description` and `textContent` (max 300 chars) from the post so the following feed can render excerpts without a secondary lookup. Timeline items expire after 30 days.
- **Above the threshold** — fan-out is skipped entirely; those readers receive the author's posts via the live celebrity query instead (see below).

### Fan-in (new subscription created)

When a subscription is created (via the indexer processing a Jetstream event, a `POST /subscriptions` call from the client backfill, or the backfill Lambda), `fanInExistingPosts` is called. It paginates through the author's published posts filtered to the subscribed publication and writes up to 20 timeline items for the reader. If the author's posts are not yet in DynamoDB at the time of the subscription (e.g. the author has never signed in and hasn't been included in a backfill), the fan-in finds nothing and the timeline stays empty until the author's posts are indexed later.

### Celebrity read path

At read time, `GET /timeline` fetches the reader's subscription list and batch-reads the subscriber count for each author. Authors above the threshold are queried live: the 10 most recent published posts are fetched directly from `u#<authorDid>#post` and merged into the response. This avoids the cost of fan-out writes to potentially millions of timeline partitions.

### Pagination

The `/following` route uses infinite scroll — an `IntersectionObserver` on a sentinel element triggers the next page fetch as the reader scrolls. The cursor is a base64url-encoded DynamoDB `LastEvaluatedKey` and only covers the pre-written timeline items. Celebrity posts are always fetched as a fixed batch (up to 10 per celebrity) on every page load and are not paginated.

### Client-side backfill on first sign-in

On first sign-in the home screen checks whether the user's data is already in DynamoDB and, if not, backfills it directly from the PDS:

1. Publications — if `GET /publications` returns empty, all `site.standard.publication` records are fetched from the PDS and posted to the API.
2. Posts — if `GET /posts` returns empty, all `site.standard.document` records are fetched and posted to the API.
3. Subscriptions — if `GET /subscriptions` returns empty, all `site.standard.graph.subscription` records are fetched and posted to the API, which triggers fan-in for each.

This only covers the signed-in user's own data. If subscribed authors have not yet been indexed, fan-in finds no posts. Running the backfill Lambda with both the reader's and the author's DIDs resolves this.

## Subscriptions page

The `/subscriptions` route (`app/routes/subscriptions.tsx`) lets a signed-in reader view and manage all publication subscriptions in one place.

### Data flow

`DataProvider` (`app/lib/data-context.tsx`) holds two layers of subscriptions state, both scoped to the session and cleared on sign-out:

- **`subscriptions: Subscription[]`** — raw subscription records loaded once per session via `GET /subscriptions`. A single lightweight API call; no publication details are fetched here.
- **`resolvedPubs: ResolvedPub[]`** — publication details (name, URL, description) resolved lazily the first time the user visits `/subscriptions`. `DataProvider` owns this state but has no fetch effect for it — all fetching happens inside the route component and the results are written back via `setResolvedPubs` / `appendResolvedPubs`.

This means navigating away and back is instant: the initial-load effect in the route component checks `resolvedPubs !== undefined` and skips the fetch entirely on return visits.

### Infinite scroll

The page loads publication details for 20 subscriptions at a time:

1. The initial `useEffect` fires once when `subscriptions` first becomes defined in context (`initializedRef` prevents re-triggering on subsequent context updates from unsubscribes).
2. For each page slice, subscriptions are grouped by `authorDid` and `listPublicPublications` is called for each unique author — at most 5 concurrent requests at a time (`BATCH = 5`) using `Promise.allSettled`. Failed author lookups are silently dropped; the rest are rendered.
3. An `IntersectionObserver` on a sentinel `<div>` at the bottom of the list triggers the next page fetch when it scrolls into view. The observer captures `offset` and `subscriptions` at setup time so stale closure values are avoided.

### Unsubscribe

Clicking "Subscribed" on a card calls `DELETE /subscriptions/:rkey` on the API and then calls `removeSubscription(rkey)` on the context, which filters the record out of the shared `subscriptions` array. The card is also removed from the local `pubs` state. The `initializedRef` guard ensures the removed subscription does not re-trigger the initial fetch loop.

## Suggestions page

The `/suggestions` route (`app/routes/suggestions.tsx`) surfaces publications from people the signed-in user follows on Bluesky, helping them discover content to subscribe to.

### Data model

Suggestions are stored in DynamoDB under `PK=u#<did>#sugg`, `SK=pub#<rkey>` (where `rkey` is the rkey of the publication on the author's PDS). The `show` boolean controls whether an item appears in the list:

- `show=true` — active suggestion, returned by `GET /suggestions`
- `show=false` — dismissed tombstone, excluded from query results but kept in the table to prevent the same publication from being re-added on future "Find suggestions" runs

Records are never deleted; the tombstone is the permanent record of a user's decision to dismiss or subscribe.

### "Find suggestions" flow

1. Fetch all Bluesky follows for the signed-in user via `app.bsky.graph.getFollows` on the public Bluesky API (paginated until exhausted).
2. For each followed DID, call `GET /publications?did=<did>` to look up their publications in the Lemma DynamoDB index (5 concurrent requests at a time).
3. Filter out the user's own publications and any pubs they are already subscribed to (checked client-side against `subscriptions` from `DataContext`).
4. Bulk-upsert candidates in batches of 100 via `POST /suggestions`. Each item is written with `ConditionExpression: 'attribute_not_exists(PK)'` so already-present items (including `show=false` tombstones) are silently skipped — the `added` count in the response reflects only net-new items.
5. Reload the suggestions list from the API and update `DataContext`.

### Filtering

Two layers of filtering keep the list clean:

- **Server-side** — `GET /suggestions` queries only items where `show=true` (via `FilterExpression`), paginating through all DynamoDB pages to avoid under-fetching.
- **Client-side** — items whose `pubAtUri` matches any entry in the user's `subscriptions` list are hidden at render time, catching publications subscribed to outside the suggestions page without waiting for a server round-trip.

### Subscribe and dismiss

- **Subscribe** — calls `POST /subscriptions` (writes the subscription to the PDS and to DynamoDB), then immediately calls `PATCH /suggestions` to set `show=false` for that item. The card is removed from `DataContext` state at the same time so it disappears from the list without a reload. This prevents the subscribed publication from reappearing on the next "Find suggestions" run.
- **Dismiss (X button)** — calls `PATCH /suggestions` directly to set `show=false` and removes the item from local state.

### Stale index entries

Suggestions are sourced from the Lemma DynamoDB index, not the live PDS. If a publication has been deleted from the PDS but the indexer missed the delete event (e.g. the indexer was down beyond the Jetstream 72-hour replay window), the stale DynamoDB record will still surface as a suggestion candidate. Such orphaned records must be removed manually by verifying the record against `com.atproto.repo.getRecord` on the author's PDS and then deleting the DynamoDB item.

### State caching

The loaded suggestions list is stored in `DataContext` (`suggestions: SuggestionItem[] | undefined`). The initial fetch fires once per session on first visit; subsequent navigation to `/suggestions` skips the fetch and renders the cached list instantly. All items are loaded in a single API call; infinite scroll reveals them 20 at a time client-side using an `IntersectionObserver` on a sentinel element.

## Jetstream indexer

### What it does

The indexer (`infra/indexer/`) is a long-running Node.js process that subscribes to the [ATProto Jetstream](https://github.com/bluesky-social/jetstream) WebSocket firehose. It filters for `site.standard.publication`, `site.standard.document`, and `site.standard.graph.subscription` commits and upserts or deletes the corresponding records in DynamoDB, keeping the index consistent with the wider ATProto network without any user-initiated action. Subscription events additionally trigger fan-in of the author's existing posts into the reader's timeline.

### Cursor

Jetstream delivers events with a `time_us` microsecond timestamp. The indexer writes the most recent `time_us` to a local file (`/var/lib/indexer/cursor`) on every event. On startup (or after a reconnect) the cursor is passed as a URL parameter so no events are missed or double-processed. The cursor file survives process restarts but not instance replacement.

**Replay window limitation:** Jetstream retains events for approximately 72 hours. If the indexer is down for longer than that window, the cursor becomes stale and Jetstream resumes from the current time, permanently skipping any events (including deletes) that occurred during the gap. Delete events missed this way leave orphaned records in DynamoDB that must be removed manually.

### Reconnection

The WebSocket connection uses exponential backoff (1 s → 30 s) on disconnect. The `SIGINT` handler closes the socket cleanly before the process exits.

The indexer is built with **esbuild** into a single CJS bundle (`dist/index.js`) with no external runtime dependencies. See `DEV.md` for local deployment and `PROD.md` for AWS deployment.

### Configuration

| Variable | Default | Description |
|---|---|---|
| `DYNAMODB_TABLE` | `blogs` | DynamoDB table name |
| `AWS_REGION` | `us-east-1` | AWS region |
| `CURSOR_FILE` | `./cursor` | Path to the cursor file |
| `JETSTREAM_HOST` | `jetstream2.us-east.bsky.network` | Jetstream WebSocket host |
| `ACCEPT_LOOSE_DOCUMENTS` | `false` | Index `site.standard.document` records whose `site` is a plain URL rather than a publication `at://` URI. Set to `true` to accept them. Defaults to `false` to reduce noise from apps that write documents without a linked publication. |

See `DEV.md` for how to opt in locally and `PROD.md` for EC2.

## Backfill Lambda

### What it does

The backfill Lambda (`infra/backfill/`) performs a one-time bulk import of ATProto data for a given set of DIDs. It resolves each DID's PDS, fetches all `site.standard.publication`, `site.standard.document`, and `site.standard.graph.subscription` records, and upserts them into DynamoDB in two passes:

1. **Publications & documents** — ensures posts are indexed before subscriptions are processed.
2. **Subscriptions** — for each DID in the file, fetches their subscription records from their own PDS (the DID is the reader), upserts each subscription, writes a subscriber item and increments the author's subscriber count, then fans in up to 20 of the author's existing posts into that DID's timeline.

The backfill Lambda is triggered by an `s3:ObjectCreated` event on the dedicated backfill trigger bucket (`atproto-blog-backfill`). Uploading a `dids.txt` file (one DID per line, `#` comments ignored) starts the job. See `DEV.md` for local usage and `PROD.md` for AWS usage including timeouts and CloudWatch logs.

### Configuration

| Variable | Default | Description |
|---|---|---|
| `DYNAMODB_TABLE` | `blogs` | DynamoDB table name |
| `CONCURRENCY` | `5` | Number of DIDs processed in parallel |
| `ACCEPT_LOOSE_DOCUMENTS` | `false` | Index documents whose `site` is a plain URL rather than a publication `at://` URI |

## iframely embed pipeline

Markdown posts support `embed` fenced code blocks:

````
```embed
https://example.com
```
````

**Unified pipeline** (`rehype-iframely` plugin): replaces the code block with a `div.iframely-embed-wrapper` carrying `data-embed-url`, `data-embed-width`, and `data-embed-height` attributes. No network calls at this stage.

**Client-side** (`PreviewRenderer` → `IframelyEmbed`): hydrates those wrapper divs into React components. Each component fetches `GET /iframely?url=<encoded>` from the API Gateway, receives iframely JSON, sanitizes the `html` field with DOMPurify (strips event handlers and `javascript:` URLs, preserves iframes), and renders it via `dangerouslySetInnerHTML`.

### `/iframely` endpoint

Public (no auth required) — the API key is kept server-side in AWS Secrets Manager.

**Abuse protection:**
- **API Gateway throttling** — 50 req/s steady, 100 burst; returns 429 before the API Lambda is invoked.
- **DynamoDB cache** — successful iframely responses are cached for 24 hours, keyed by SHA-256 hash of the URL. Repeated requests for the same URL cost a DynamoDB read instead of an iframely API call. Cache failures are non-fatal.

**Client-side security:**
- DOMPurify sanitizes the iframely HTML before rendering. `<script>` tags are stripped (DOMPurify removes them; React's `dangerouslySetInnerHTML` would not execute them anyway). DOMPurify additionally strips inline event handlers and `javascript:` hrefs.
- After the sanitized HTML is in the DOM, `IframelyEmbed` programmatically injects `cdn.iframe.ly/embed.js` once per page (subsequent embeds call `window.iframely.load()` to trigger a re-scan). The script tag carries an SRI `integrity` attribute so the browser verifies the fetched bytes match a known hash before executing — a tampered CDN response is rejected.
- URL validation and SSRF protection (private IP blocking) are applied by the `rehype-iframely` plugin before the URL is ever sent to the API.

## ATProto lexicons

The app uses [Standard.site](https://standard.site) lexicons for publishing and social graph:

- `site.standard.publication` — written to the user's PDS to represent their blog. Required before any documents can be created.
- `site.standard.document` — written to the user's PDS for each post. The `site` field holds the publication's AT-URI; `path` is set to `/<rkey>` so the document URL can be reconstructed from the publication's base URL. Optional `description` field holds a short summary displayed in feed cards.
- `site.standard.graph.subscription` — written to the reader's PDS when they subscribe to a publication. The `publication` field holds the publication's AT-URI.

Publications and documents are written by the browser via `com.atproto.repo.putRecord` (upsert semantics). Subscriptions are written via `com.atproto.repo.createRecord`. All use the user's DPoP-bound session. The API Lambda only receives the resulting AT-URIs for indexing.

`site.standard.document` includes an optional `bskyPostRef: { uri: string; cid: string }` field that points to the sidecar `app.bsky.feed.post` created at publish time.

## Front End

React Router v7 in **SPA mode** (`ssr: false`, `prerender: true`). The build emits a fully static `build/client/` directory with no server runtime — every route is pre-rendered to a static HTML shell and hydrated in the browser.

Key config files:
- `react-router.config.ts` — `ssr: false, prerender: true` emits a fully static `build/client/` with no server runtime.
- `vite.config.ts` — Tailwind CSS v4, React Router, `vite-plugin-pwa` (auto-update, standalone display).
- `app/routes.ts` — single index route → `app/routes/home.tsx`.

Route types are auto-generated into `.react-router/types/` by `react-router typegen`. Import them as `import type { Route } from "./+types/home"`.

Dependencies:
- **Tailwind CSS v4** for styling
- **vite-plugin-pwa** for service worker / offline support (standalone display mode)
- **CodeMirror** for the markdown editor
- **Shiki** for syntax highlighting in post previews


## PWA / bundle size

The publish page bundles CodeMirror and Shiki (syntax highlighting grammars for many languages), producing chunks well over Workbox's default 2 MiB precache limit. `vite.config.ts` raises `workbox.maximumFileSizeToCacheInBytes` to 10 MiB to allow the service worker to precache these assets.

If additional large dependencies are added and the build fails with a Workbox precache error, increase that limit or exclude the offending chunks with `workbox.globIgnores`.

## S3 hosting

Sync `build/client/` to the bucket. Client-side routing on direct URL access or refresh is handled by the CloudFront SPA router function (rewrites extensionless paths to `/index.html`) and custom error response (403 → 200 `index.html`). No S3 website configuration or error document is needed.

The bucket has no public access; reads are gated through the CloudFront OAC.

## `did:web` production requirements

Two files must be present in the S3 bucket / served from the site origin for production ATProto OAuth to work:

| File | Purpose |
|------|---------|
| `/.well-known/did.json` | Makes `did:web:<domain>` resolvable. Must contain a `service` entry with `id: #blog-api` and `serviceEndpoint` pointing to the API URL. See `docs/api-auth.md` for the full document. |
| `/oauth-client-metadata.json` | ATProto OAuth client metadata. Generated by the SPA build from `public/oauth-client-metadata.json`. |

**OAuth scope** — `oauth-client.ts` derives the scope from `VITE_API_DID` at build time. When `VITE_API_DID` is `did:web:localhost` (local dev) it uses `transition:generic`; for any other value it uses the granular scope `atproto rpc:*?aud=<VITE_API_DID>%23blog-api`. No manual change is needed — setting `VITE_API_DID` correctly is sufficient. The granular scope requires `did:web:<domain>` to be resolvable (i.e. `did.json` served from the site origin) so the PDS can validate the `aud` claim against the DID document's service entries. `setup-aws-domain` updates `did.json` automatically.

## Environment variables

| Variable | Description |
|----------|-------------|
| `VITE_API_URL` | Base URL of the blog API (e.g. `https://api.example.com`). All API paths (`/posts`, `/publications`, `/iframely`, `/users`, etc.) are appended to this. Written to `.env.local` by `deploy-local.sh` / `deploy-aws.sh`. |
| `VITE_API_DID` | DID of the API service, used as the `aud` claim in ATProto service JWTs. For local dev: `did:web:localhost`. For production: `did:web:<domain>` (without fragment; the API Lambda appends `#blog-api` for the `rpc:` scope check). |

Vite inlines `VITE_`-prefixed variables at build time. If either value changes, rebuild and redeploy the SPA.

## Security

Security measures covering XSS prevention, API authorization, DynamoDB isolation, SSRF, IAM, and known trade-offs are documented in `docs/security.md`.

