# Production

## Additional Production Prerequisites

- A registered domain name with Route 53 as the DNS service.
- An AWS profile configured (default: `production`). Override with `AWS_PROFILE=<profile>`.
- [`jq`](https://jqlang.org) for updating `did.json` during custom domain setup.
- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) for the AWS CLI.

## Workflow

### Full solution deployment

Make sure your shell is not logging these in your command [history](https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html#index-HISTCONTROL).
```sh
 export IFRAMELY_API_KEY=your-api-key-client-hash
make aws AWS_PROFILE=production
```

### Custom domain setup

Run once after the core stacks are deployed. Requires `jq` (for updating `did.json`).

```sh
export DOMAIN_NAME=example.com
export SITE_SUBDOMAIN=www   # optional — omit for apex-only
export API_SUBDOMAIN=api    # optional — omit to skip API subdomain
make -C infra setup-aws-domain
```

This runs in sequence: issues the ACM certificate (waits for DNS validation, ~1–2 min), deploys WAF and CloudFront, creates Route 53 records, re-deploys the API with the correct `AllowedOrigin`, updates `public/.well-known/did.json`, rebuilds the frontend, and syncs the site to S3.

### Indexer

**First deploy** — creates the EC2 instance. The `UserData` cloud-init script runs once on first boot: installs Node.js 24, downloads `indexer.zip` from S3, writes the systemd unit with `DYNAMODB_TABLE`, `AWS_REGION`, and `CURSOR_FILE` baked in, then enables and starts the service. **Idempotent** — CF deploy uses `--no-fail-on-empty-changeset`; re-running does not update code on a running instance.

```sh
make deploy-aws-indexer
```

**Code updates** — use `make update-aws-indexer` to push new code without replacing the instance. **Idempotent.**

```sh
make update-aws-indexer
```

1. Builds and uploads the new bundle to S3 (`upload-indexer-aws`)
2. Fetches the instance ID from the CloudFormation stack output
3. Uses SSM `send-command` with `AWS-RunShellScript` to run on the instance:
   - `systemctl stop indexer`
   - Downloads new bundle from S3
   - Unzips it into `/opt/indexer/`
   - Deletes the zip
   - `systemctl start indexer`
4. Waits for the command to finish before returning

The cursor file at `/var/lib/indexer/cursor` lives outside `/opt/indexer/` and is untouched, so the indexer resumes from where it left off.

**Force instance replacement** — if you need to replace the EC2 instance (e.g. after changing cloud-init config), pass a timestamped `S3Key` so CloudFormation detects a `UserData` change. **Not idempotent by design.**

```sh
make create-aws-indexer INDEXER_S3_KEY=indexer/indexer-$(date +%Y%m%d%H%M%S).zip
```

**Checking the indexer:**

```sh
# View setup log (written by cloud-init on first boot)
sudo cat /var/log/indexer-init.log

# View live service logs
sudo journalctl -u indexer -f
```

**Opting in to loose documents** — to index `site.standard.document` records whose `site` is a plain URL rather than a publication `at://` URI, add `ACCEPT_LOOSE_DOCUMENTS=true` to the `[Service]` section of the systemd unit in `infra/cf/indexer.yaml`:

```yaml
          Environment=CURSOR_FILE=/var/lib/indexer/cursor
          Environment=ACCEPT_LOOSE_DOCUMENTS=true
```

Then redeploy the stack (`make deploy-aws-indexer`) and restart the service via `make update-aws-indexer`.

### Iframely

**Updating the embed.js SRI hash** (run after an iframely CDN update breaks embeds):

```sh
make -C infra update-iframely-sri
```

This fetches the current `embed.js`, computes the hash, and updates `EMBED_JS_INTEGRITY` in `app/components/IframelyEmbed.tsx` and the corresponding assertion in `IframelyEmbed.test.tsx`. Then rebuild and redeploy the SPA:

```sh
pnpm build
make -C infra sync-site
```

### Troubleshooting

#### Inspect failed CloudFormation changesets

`aws cloudformation deploy` always deletes the changeset it creates, so no accumulation occurs. If you want to verify:

```sh
for stack in $(aws cloudformation list-stacks --profile "$AWS_PROFILE" \
  --query 'StackSummaries[?StackStatus!=`DELETE_COMPLETE`].StackName' \
  --output text | tr '\t' '\n' | grep atproto-blog); do
  echo "=== $stack ==="
  aws cloudformation list-change-sets --profile "$AWS_PROFILE" \
    --stack-name "$stack" \
    --query 'Summaries[?Status==`FAILED`].[ChangeSetName,StatusReason]' \
    --output table
done
```

To delete a specific failed changeset:

```sh
aws cloudformation delete-change-set --profile "$AWS_PROFILE" \
  --stack-name <stack-name> --change-set-name <change-set-name>
```

### Backfill

```sh
make deploy-aws-backfill     # First deploy: build, upload bundle, deploy CF stack. Idempotent.
make update-aws-backfill     # Code-only update. Idempotent.
```

```sh
make run-aws-backfill DIDS_FILE=/tmp/dids.txt   # Upload dids.txt to S3 trigger bucket — backfill Lambda fires automatically
```

**Data-idempotent** — all DynamoDB writes are upserts and the subscriber count increment is guarded by `attribute_not_exists`, so re-running with the same file leaves the database in the same state. S3 `ObjectCreated` fires on every upload, triggering a new backfill Lambda execution each time.

The backfill Lambda timeout is 900 seconds (15 minutes). For very large DID lists, split the file into chunks and upload each separately — each upload triggers a separate Lambda invocation with its own 15-minute window.

To check how long a run took:

```sh
aws logs filter-log-events \
  --profile "$AWS_PROFILE" \
  --log-group-name /aws/lambda/backfill \
  --filter-pattern "REPORT" \
  --query 'events[*].message' \
  --output text
```

Each `REPORT` line includes `Duration`, `Billed Duration`, and `Max Memory Used`.

### Mute a user

Makes the indexer ignore records from an account. Useful for bots and spam that flood the network with document records.

Example
```sh
aws dynamodb put-item \
    --profile production \
    --table-name blogs \
    --item '{
      "PK": {"S": "muted#user"},
      "SK": {"S": "u#did:plc:abcd1234"}
    }'
```

After muting a user restart the index with
```sh
AWS_PROFILE=production make -C infra restart-aws-indexer
```