-214

PLAN.md

Reviewed

··· 1 1 - # Flux GitOps Migration Plan 2 2 - 3 3 - ## Final Step-By-Step Plan 4 4 - 5 5 - 1. **Bootstrap Flux With Option B** 6 6 - - Use `flux bootstrap github` now. 7 7 - - Bootstrap path: `k8s/flux/system`. 8 8 - - Do not change NixOS during this migration. 9 9 - - Treat future NixOS/k3s auto-deploy bootstrap as optional hardening later. 10 10 - - Ensure there is one root model: `k8s/flux/system` owns the Flux root and references child Flux `Kustomization`s. 11 11 - 12 12 - 2. **Add Flux SOPS Key** 13 13 - - Generate a dedicated Flux age keypair. 14 14 - - Commit only the public key as `keys/flux_k8s.pub`. 15 15 - - Store the private key out-of-band. 16 16 - - Create `flux-system/sops-age` manually as a bootstrap artifact. 17 17 - - Do not rely on Flux to decrypt/manage its own decryption key. 18 18 - - Disaster recovery procedure will be: rebuild k3s, bootstrap Flux, apply `sops-age`, then Flux reconciles encrypted resources. 19 19 - 20 20 - 3. **Update SOPS Policy** 21 21 - - Extend `.sops.yaml` generation so `k8s/flux/secrets/*.sops.yaml` encrypts to: 22 22 - - existing user keys 23 23 - - existing host keys 24 24 - - new `flux_k8s` key 25 25 - - Keep existing `secrets/*.yaml` encrypted only to user/host keys. 26 26 - - Flux must not be able to decrypt host/Terraform/local secrets. 27 27 - - Run `just sops-rekey` after updating recipients. 28 28 - 29 29 - 4. **Create Flux Directory Layout** 30 30 - - Create: 31 31 - 32 32 - ```text 33 33 - k8s/flux/ 34 34 - ├── system/ 35 35 - ├── sources/ 36 36 - ├── infra-crds/ 37 37 - ├── infra-core/ 38 38 - ├── platform/ 39 39 - ├── apps/ 40 40 - ├── external-routes/ 41 41 - └── secrets/ 42 42 - ``` 43 43 - 44 44 - - Keep all first-class k8s Secret manifests in flat `k8s/flux/secrets/`. 45 45 - 46 46 - 5. **Define Flux Layer Graph** 47 47 - - Use Flux `Kustomization.dependsOn` between layers: 48 48 - 49 49 - ```text 50 50 - system -> sources -> infra-crds -> infra-core -> platform -> apps -> external-routes 51 51 - ``` 52 52 - 53 53 - - Use `HelmRelease.dependsOn` only for same-layer release ordering. 54 54 - - Set all Flux intervals to `15m`. 55 55 - - Use explicit `flux reconcile ...` for immediate deploys. 56 56 - - Empty layers carry temporary placeholder ConfigMaps so Flux/Kustomize can 57 57 - reconcile them before real resources are migrated. Remove each placeholder 58 58 - when that layer gets its first real resource. 59 59 - 60 60 - 6. **Create Flux Sources** 61 61 - - Add `GitRepository cute-haus` pointing at the monorepo. 62 62 - - Add `HelmRepository` resources for: 63 63 - - `grafana` 64 64 - - `prometheus-community` 65 65 - - `traefik` 66 66 - - `longhorn` 67 67 - - `cnpg` 68 68 - - `tailscale` 69 69 - - `jetstack` 70 70 - - `nfd` 71 71 - - `intel` 72 72 - 73 73 - 7. **Convert Global Values** 74 74 - - Convert `k8s/values/global.yaml` into non-secret `ConfigMap cute-haus-global`. 75 75 - - Reference it from HelmReleases with `valuesFrom`. 76 76 - - Document that this ConfigMap must stay non-secret. 77 77 - - Any secret-like value must go into `k8s/flux/secrets/*.sops.yaml`. 78 78 - 79 79 - 8. **Migrate Secrets To First-Class Manifests** 80 80 - - For charts that currently render Secrets, create equivalent SOPS-encrypted Kubernetes Secret manifests in `k8s/flux/secrets/`. 81 81 - - Delete chart secret templates after the first-class Secret exists. 82 82 - - Ensure each Secret is applied in the same or earlier Flux `Kustomization` than its consuming `HelmRelease`. 83 83 - 84 84 - 9. **Apply Secret-Specific Migrations** 85 85 - - `longhorn-creds`: move `longhorn-b2` to first-class Secret; keep chart for recurring job, UI ingress, and proxy class. 86 86 - - `cert-manager-issuers`: move `cloudflare-api-token` to first-class Secret. 87 87 - - `cluster-tls`: move TLS material to first-class `kubernetes.io/tls` Secrets. 88 88 - - `pg-shared`: move `pg-shared-b2` and `<role>-pg-creds` to first-class Secrets. 89 89 - - `tranquil-pds`: move `tranquil-pds-env` and `atcr-pull` to first-class Secrets. 90 90 - - `forward-auth`: move per-app `forward-auth-<app>-env` Secrets to first-class Secrets. 91 91 - - App env Secrets: move `<app>-env` Secrets to first-class Secrets. 92 92 - - `tailscale-operator`: use `valuesFrom: Secret` because the upstream chart consumes secret values directly. 93 93 - 94 94 - 10. **Preserve Forward-Auth Check** 95 95 - - Keep the forward-auth `apps:` declaration plaintext. 96 96 - - Move only signing secrets into first-class encrypted Secret manifests. 97 97 - - Keep `check-forward-auth.ts` conceptually unchanged. 98 98 - 99 99 - 11. **Convert Helmfile Releases To Flux HelmReleases** 100 100 - - Convert releases layer-by-layer. 101 101 - - Store Flux `HelmRelease` objects in `flux-system` and set 102 102 - `spec.targetNamespace` to the workload namespace. This keeps shared 103 103 - `valuesFrom` objects such as `cute-haus-global` in one namespace. 104 104 - - External charts use `HelmRepository`. 105 105 - - In-tree charts use `GitRepository` and `chart: ./k8s/charts/<name>`. 106 106 - - Inline non-secret values in `spec.values`. 107 107 - - Use `valuesFrom` for `cute-haus-global` and the `tailscale-operator` secret values. 108 108 - - Do not let Helmfile and Flux manage the same release at the same time. 109 109 - 110 110 - 12. **Smoke Test First** 111 111 - - Migrate `watsup` first. 112 112 - - Since `cluster-tls` is still Helmfile-managed at that point, omit temporary Flux `dependsOn` for `cluster-tls`. 113 113 - - Validate: 114 114 - - Flux source reconciliation 115 115 - - local chart HelmRelease 116 116 - - `helm-controller` 117 117 - - manual `just k8s` workflow 118 118 - - rollback path 119 119 - 120 120 - 13. **Migrate Layer-By-Layer** 121 121 - - After `watsup`, migrate: 122 122 - 1. `infra-crds`: `cert-manager`, `cnpg` 123 123 - 2. `infra-core`: `traefik`, `longhorn-creds`, `longhorn`, `tailscale-operator`, NFD, Intel device plugins, `cert-manager-issuers` 124 124 - 3. `platform`: `pg-shared`, `valkey`, `tika`, `gotenberg`, `cluster-tls`, monitoring 125 125 - 4. `apps`: app releases, env-only first, multi-secret apps later 126 126 - 5. `external-routes`: last 127 127 - 128 128 - 14. **Per-Release Cutover Checklist** 129 129 - - Create required first-class SOPS Secret manifests. 130 130 - - Add or update the `HelmRelease`. 131 131 - - Remove the release from `k8s/helmfile.yaml`. 132 132 - - Remove obsolete `k8s/values/secrets/<release>.yaml`. 133 133 - - Push. 134 134 - - Run: 135 135 - 136 136 - ```bash 137 137 - flux reconcile source git cute-haus -n flux-system 138 138 - flux reconcile kustomization <layer> -n flux-system 139 139 - ``` 140 140 - 141 141 - - Verify: 142 142 - 143 143 - ```bash 144 144 - flux get kustomizations -A 145 145 - flux get helmreleases -A 146 146 - helm list -A 147 147 - kubectl get all -n <namespace> 148 148 - ``` 149 149 - 150 150 - 15. **Add `just k8s` Manual Workflow** 151 151 - - Add one short helper: 152 152 - 153 153 - ```bash 154 154 - just k8s apply <release> 155 155 - just k8s diff <release> 156 156 - just k8s suspend <release> 157 157 - just k8s resume <release> 158 158 - just k8s reconcile <release> 159 159 - ``` 160 160 - 161 161 - - Scope: local in-tree charts only where release name equals `k8s/charts/<release>`. 162 162 - - Flux `HelmRelease` objects live in `flux-system`; the optional namespace 163 163 - argument remains the Helm target namespace. 164 164 - - `apply` suspends Flux first, then runs `helm upgrade --install`. 165 165 - - `diff` uses `helm diff upgrade --allow-unreleased`. 166 166 - - External charts are managed through Flux only. 167 167 - 168 168 - 16. **Update CI And Checks** 169 169 - - Keep existing chart render checks. 170 170 - - Update `check-release-names.ts` to read Flux `HelmRelease` resources instead of `helmfile.yaml`. 171 171 - - Update `check-pinned-images.ts` to derive deployed local charts from Flux `HelmRelease`s. 172 172 - - Keep `check-forward-auth.ts` reading plaintext forward-auth app declarations. 173 173 - - Add Flux/Kustomize validation: 174 174 - 175 175 - ```bash 176 176 - kustomize build k8s/flux/system 177 177 - kustomize build k8s/flux/sources 178 178 - kustomize build k8s/flux/secrets 179 179 - kustomize build k8s/flux/infra-crds 180 180 - kustomize build k8s/flux/infra-core 181 181 - kustomize build k8s/flux/platform 182 182 - kustomize build k8s/flux/apps 183 183 - kustomize build k8s/flux/external-routes 184 184 - ``` 185 185 - 186 186 - 17. **Rollback Strategy** 187 187 - - Keep `helmfile.yaml` usable until the final cleanup. 188 188 - - If one release fails: 189 189 - 190 190 - ```bash 191 191 - flux suspend helmrelease <release> -n <namespace> 192 192 - ``` 193 193 - 194 194 - - Re-add the release to Helmfile if needed and apply it manually. 195 195 - - For a bad Flux commit, revert the commit and reconcile Flux. 196 196 - - Do not delete Helmfile until all releases have been verified under Flux. 197 197 - 198 198 - 18. **Final Cleanup** 199 199 - - Delete `k8s/helmfile.yaml`. 200 200 - - Delete obsolete `k8s/values/secrets/*.yaml`. 201 201 - - Audit for `vals` and `ref+sops://`. 202 202 - - Remove `vals` from the devShell if unused. 203 203 - - Update docs: 204 204 - - `AGENTS.md` 205 205 - - `README.md` 206 206 - - `k8s/charts/README.md` 207 207 - - Clarify that `longhorn-creds` no longer owns credential material despite retaining the release name. 208 208 - 209 209 - 19. **Post-Migration Hardening** 210 210 - - Rotate duplicated shared credentials toward one credential per consumer. 211 211 - - Optionally migrate Flux bootstrap from Option B to Option A: 212 212 - - NixOS writes Flux controller manifests into k3s auto-deploy dir. 213 213 - - Flux continues managing app/platform resources from Git. 214 214 - - Optionally reorganize legacy `secrets/` later, outside this migration.

+1 -1

k8s/charts/README.md

Reviewed

··· 13 13 charts/ 14 14 ├── aly-codes/ # Static site (aly.codes) 15 15 ├── audiobookshelf/ # Audiobook library with rclone-mounted media 16 16 - ├── bluesky-pds/ # Reference atproto Personal Data Server 16 16 + ├── tranquil-pds/ # Reference atproto Personal Data Server 17 17 ├── cert-manager-issuers/ # Let's Encrypt ClusterIssuer + wildcard Certificates 18 18 ├── cluster-tls/ # Cloudflare Origin TLS Secrets per domain 19 19 ├── external-routes/ # Ingress + Service + EndpointSlice for off-cluster targets

-5

k8s/charts/bluesky-pds/Chart.yaml

Reviewed

··· 1 1 - apiVersion: v2 2 2 - name: bluesky-pds 3 3 - description: Bluesky PDS — atproto Personal Data Server 4 4 - type: application 5 5 - version: 0.1.0

-103

k8s/charts/bluesky-pds/templates/deployment.yaml

Reviewed

··· 1 1 - apiVersion: apps/v1 2 2 - kind: Deployment 3 3 - metadata: 4 4 - name: {{ .Chart.Name }} 5 5 - labels: 6 6 - app: {{ .Chart.Name }} 7 7 - spec: 8 8 - replicas: 1 9 9 - strategy: 10 10 - type: Recreate 11 11 - selector: 12 12 - matchLabels: 13 13 - app: {{ .Chart.Name }} 14 14 - template: 15 15 - metadata: 16 16 - labels: 17 17 - app: {{ .Chart.Name }} 18 18 - spec: 19 19 - automountServiceAccountToken: false 20 20 - enableServiceLinks: false 21 21 - terminationGracePeriodSeconds: 15 22 22 - securityContext: 23 23 - seccompProfile: 24 24 - type: RuntimeDefault 25 25 - tolerations: 26 26 - - key: node.kubernetes.io/not-ready 27 27 - operator: Exists 28 28 - effect: NoExecute 29 29 - tolerationSeconds: {{ $.Values.global.failover.fastTolerationSeconds | default 60 }} 30 30 - - key: node.kubernetes.io/unreachable 31 31 - operator: Exists 32 32 - effect: NoExecute 33 33 - tolerationSeconds: {{ $.Values.global.failover.fastTolerationSeconds | default 60 }} 34 34 - containers: 35 35 - - name: {{ .Chart.Name }} 36 36 - image: ghcr.io/bluesky-social/pds:latest@sha256:1fa8bbceabb65d8e1710749b1ea92c1c20a7489ca38da4a0a5f64c0c10a70c29 37 37 - imagePullPolicy: IfNotPresent 38 38 - env: 39 39 - - name: LOG_ENABLED 40 40 - value: "true" 41 41 - - name: PDS_BSKY_APP_VIEW_DID 42 42 - value: "did:web:api.bsky.app" 43 43 - - name: PDS_BSKY_APP_VIEW_URL 44 44 - value: "https://api.bsky.app" 45 45 - - name: PDS_BLOBSTORE_DISK_LOCATION 46 46 - value: "/pds/blocks" 47 47 - - name: PDS_BLOB_UPLOAD_LIMIT 48 48 - value: "104857600" 49 49 - - name: PDS_CRAWLERS 50 50 - value: "https://bsky.network" 51 51 - - name: PDS_DATA_DIRECTORY 52 52 - value: "/pds" 53 53 - - name: PDS_DID_PLC_URL 54 54 - value: "https://plc.directory" 55 55 - - name: PDS_HOSTNAME 56 56 - value: "aly.social" 57 57 - - name: PDS_PORT 58 58 - value: "3000" 59 59 - - name: PDS_RATE_LIMITS_ENABLED 60 60 - value: "true" 61 61 - - name: PDS_REPORT_SERVICE_DID 62 62 - value: "did:plc:ar7c4by46qjdydhdevvrndac" 63 63 - - name: PDS_REPORT_SERVICE_URL 64 64 - value: "https://mod.bsky.app" 65 65 - envFrom: 66 66 - - secretRef: 67 67 - name: bluesky-pds-env 68 68 - ports: 69 69 - - name: http 70 70 - containerPort: 3000 71 71 - resources: 72 72 - requests: 73 73 - cpu: 100m 74 74 - memory: 256Mi 75 75 - limits: 76 76 - cpu: "1" 77 77 - memory: 1Gi 78 78 - volumeMounts: 79 79 - - name: data 80 80 - mountPath: /pds 81 81 - startupProbe: 82 82 - httpGet: 83 83 - path: /xrpc/_health 84 84 - port: http 85 85 - periodSeconds: 5 86 86 - failureThreshold: 60 87 87 - livenessProbe: 88 88 - httpGet: 89 89 - path: /xrpc/_health 90 90 - port: http 91 91 - periodSeconds: 60 92 92 - timeoutSeconds: 15 93 93 - readinessProbe: 94 94 - httpGet: 95 95 - path: /xrpc/_health 96 96 - port: http 97 97 - periodSeconds: 10 98 98 - timeoutSeconds: 10 99 99 - failureThreshold: 2 100 100 - volumes: 101 101 - - name: data 102 102 - persistentVolumeClaim: 103 103 - claimName: {{ .Chart.Name }}-data

-23

k8s/charts/bluesky-pds/templates/ingress.yaml

Reviewed

··· 1 1 - apiVersion: networking.k8s.io/v1 2 2 - kind: Ingress 3 3 - metadata: 4 4 - name: {{ .Chart.Name }} 5 5 - labels: 6 6 - app: {{ .Chart.Name }} 7 7 - spec: 8 8 - ingressClassName: traefik 9 9 - tls: 10 10 - - hosts: 11 11 - - "aly.social" 12 12 - secretName: aly-social-tls 13 13 - rules: 14 14 - - host: "aly.social" 15 15 - http: 16 16 - paths: 17 17 - - path: / 18 18 - pathType: Prefix 19 19 - backend: 20 20 - service: 21 21 - name: {{ .Chart.Name }} 22 22 - port: 23 23 - number: 80

-15

k8s/charts/bluesky-pds/templates/pvc.yaml

Reviewed

··· 1 1 - apiVersion: v1 2 2 - kind: PersistentVolumeClaim 3 3 - metadata: 4 4 - name: {{ .Chart.Name }}-data 5 5 - labels: 6 6 - app: {{ .Chart.Name }} 7 7 - annotations: 8 8 - helm.sh/resource-policy: keep 9 9 - spec: 10 10 - accessModes: 11 11 - - ReadWriteOnce 12 12 - storageClassName: longhorn 13 13 - resources: 14 14 - requests: 15 15 - storage: 10Gi

-15

k8s/charts/bluesky-pds/templates/secret.yaml

Reviewed

··· 1 1 - {{- if .Values.secret }} 2 2 - apiVersion: v1 3 3 - kind: Secret 4 4 - metadata: 5 5 - name: bluesky-pds-env 6 6 - labels: 7 7 - app: {{ .Chart.Name }} 8 8 - type: Opaque 9 9 - stringData: 10 10 - PDS_ADMIN_PASSWORD: {{ .Values.secret.PDS_ADMIN_PASSWORD | quote }} 11 11 - PDS_EMAIL_FROM_ADDRESS: {{ .Values.secret.PDS_EMAIL_FROM_ADDRESS | quote }} 12 12 - PDS_EMAIL_SMTP_URL: {{ .Values.secret.PDS_EMAIL_SMTP_URL | quote }} 13 13 - PDS_JWT_SECRET: {{ .Values.secret.PDS_JWT_SECRET | quote }} 14 14 - PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX: {{ .Values.secret.PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX | quote }} 15 15 - {{- end }}

-14

k8s/charts/bluesky-pds/templates/service.yaml

Reviewed

··· 1 1 - apiVersion: v1 2 2 - kind: Service 3 3 - metadata: 4 4 - name: {{ .Chart.Name }} 5 5 - labels: 6 6 - app: {{ .Chart.Name }} 7 7 - spec: 8 8 - type: ClusterIP 9 9 - selector: 10 10 - app: {{ .Chart.Name }} 11 11 - ports: 12 12 - - name: http 13 13 - port: 80 14 14 - targetPort: 3000

+4 -4

k8s/flux/system/README.md

Reviewed

··· 1 1 # Flux System 2 2 3 3 - `flux bootstrap github --path=k8s/flux/system` installs Flux and may generate 4 4 - or update files in this directory. Preserve `layers.yaml` and keep it included 5 5 - from `kustomization.yaml`; it wires the child reconciliation layers for this 6 6 - repo. 3 3 + Flux was bootstrapped from exported manifests (Option B), not via 4 4 + `flux bootstrap github`. The manifests in this directory were applied manually 5 5 + to the cluster. Preserve `layers.yaml` and keep it included from 6 6 + `kustomization.yaml`; it wires the child reconciliation layers for this repo. 7 7 8 8 Under the current bootstrap model, `flux-system/sops-age` is a bootstrap 9 9 artifact applied out-of-band, not an encrypted Flux-managed Secret.