Hybrid cloud cluster monorepo with Ansible, K8s, NixOS, and Terraform.
cute.haus
ansible
terraform
nix
k8s
k8s: complete flux migration
Aly Raffauf
(Jul 1, 2026, 4:19 PM EDT)
27a83d7a
04696073
+82
-123
+11
-9
AGENTS.md
+11
-9
AGENTS.md
···
1
1
# AGENTS.md
2
2
3
3
NixOS + k3s cluster IaC for cute.haus. Single operator. Flake-parts
4
-
NixOS configs on bare-metal/VPS hosts; an in-tree helm chart per app on k3s;
5
-
SOPS+age secrets; Cloudflare/B2 via Terraform.
4
+
NixOS configs on bare-metal/VPS hosts; Flux-managed in-tree Helm charts on
5
+
k3s; SOPS+age secrets; Cloudflare/B2 via Terraform.
6
6
7
7
## Environment
8
8
···
59
59
`flake.modules.nixos.<name>` (see `nix/nixos/base/default.nix`). Disk
60
60
profiles in `nix/disko.nix`. k3s node config in `nix/nixos/profiles/k3s.nix`
61
61
(flannel on `tailscale0`; startup blocks until tailscale0 has an IP).
62
-
- `k8s/` — `helmfile.yaml` is the ordered release graph (uses `needs:`).
63
-
`charts/<name>/` are in-tree charts with **explicit manifests, no shared
64
-
helpers** (see `k8s/charts/README.md`). `values/global.yaml` holds shared
65
-
image pins + constants; `values/secrets/<name>.yaml` holds `ref+sops://`
66
-
vals refs into `secrets/`. Secret flow: `secrets/*.yaml` → vals → helmfile
67
-
→ chart `templates/secret.yaml` → app `envFrom`.
62
+
- `k8s/` — `flux/` is the ordered release graph. `flux/system/layers.yaml`
63
+
defines the Flux Kustomization DAG; `flux/*/*.yaml` hold HelmReleases;
64
+
`flux/secrets/*.sops.yaml` are first-class Kubernetes Secrets decrypted by
65
+
Flux. `charts/<name>/` are in-tree charts with **explicit manifests, no
66
+
shared helpers** (see `k8s/charts/README.md`). `values/global.yaml` holds
67
+
shared non-secret image pins + constants, mirrored into the Flux
68
+
`cute-haus-global` ConfigMap.
68
69
- `secrets/` — SOPS, multi-recipient age. `.sops.yaml` is generated; never
69
70
hand-edit either.
70
71
- `keys/` — `aly_<host>.pub` (user key) + `root_<host>.pub` (host key) age
···
92
93
in `k8s/values/global.yaml`. Renovate keeps digests fresh (skips
93
94
`tranquil`, which is on private atcr.io — use `just bump-tranquil`).
94
95
- **forward-auth OIDC slugs** must be `forward-auth-<app>` and match a key in
95
-
`k8s/values/secrets/forward-auth.yaml` (enforced by `check-forward-auth.ts`).
96
+
the plaintext `apps:` map in `k8s/flux/apps/releases.yaml` (enforced by
97
+
`check-forward-auth.ts`).
96
98
- k3s node labels: `topology.kubernetes.io/zone` (cloud/home),
97
99
`cute.haus/ingress=true`, and `cute.haus/intel-gpu=true` (NixOS module).
98
100
For GPU pods request the `gpu.intel.com/i915` resource, not a nodeSelector.
+3
-3
README.md
+3
-3
README.md
···
18
18
├── nix/ # NixOS + flake modules
19
19
│ ├── hosts/ # NixOS host configurations
20
20
│ └── modules/ # NixOS / flake modules
21
-
├── k8s/ # k3s: helmfile + in-tree charts + vals overlays
22
-
│ ├── helmfile.yaml # release graph (helmfile + helm + vals)
21
+
├── k8s/ # k3s: Flux + in-tree Helm charts
22
+
│ ├── flux/ # GitOps release graph (Flux Kustomizations/HelmReleases)
23
23
│ ├── charts/ # In-tree helm charts (see k8s/charts/README.md)
24
-
│ └── values/ # Per-chart vals refs into ../secrets/
24
+
│ └── values/ # Shared non-secret Helm values
25
25
├── secrets/ # SOPS-encrypted yaml (multi-recipient age)
26
26
├── keys/ # Per-host + per-user age recipients
27
27
├── terraform/ # Cloudflare DNS, etc.
+14
-17
k8s/charts/README.md
+14
-17
k8s/charts/README.md
···
1
1
# Charts
2
2
3
-
In-tree Helm charts deployed by [`../helmfile.yaml`](../helmfile.yaml).
3
+
In-tree Helm charts deployed by Flux HelmReleases under [`../flux`](../flux).
4
4
5
5
Most app charts use explicit Kubernetes manifests. Helm is used for light
6
-
substitution, mostly `.Chart.Name` and secret values passed by helmfile. Avoid
7
-
shared Deployment/Service/PVC helpers; app-specific behavior should stay visible
8
-
in the app chart.
6
+
substitution, mostly `.Chart.Name` and shared non-secret values passed by Flux.
7
+
Avoid shared Deployment/Service/PVC helpers; app-specific behavior should stay
8
+
visible in the app chart.
9
9
10
10
## Layout
11
11
···
79
79
Keep these:
80
80
81
81
- `.Chart.Name` for names, labels, selectors, and PVC names.
82
-
- `if .Values.secret` around Secret manifests populated by vals/SOPS.
83
82
- Small `range` loops for charts that are naturally data-driven.
84
83
85
84
Avoid these:
···
130
129
## Secret Flow
131
130
132
131
```text
133
-
secrets/foo.yaml SOPS-encrypted, multi-recipient age
132
+
k8s/flux/secrets/foo.sops.yaml SOPS-encrypted Kubernetes Secret
134
133
|
135
134
v
136
-
values/foo.yaml ref+sops://... refs read by vals
135
+
Flux kustomize-controller decrypts with flux-system/sops-age
137
136
|
138
137
v
139
-
helmfile values: passes plaintext values during render
138
+
Kubernetes Secret first-class object in target namespace
140
139
|
141
140
v
142
-
templates/secret.yaml renders a chart-local Secret
143
-
|
144
-
v
145
-
deployment envFrom app consumes the Secret
141
+
Deployment envFrom/secretRef app consumes the Secret
146
142
```
147
143
148
-
`vals` resolves `ref+sops://` URLs at render time using the local age key.
144
+
Flux decrypts only `k8s/flux/secrets/*.sops.yaml`; host/Terraform secrets under
145
+
`secrets/*.yaml` remain user/host-key scoped.
149
146
150
147
## Adding An App
151
148
···
154
151
`deployment.yaml`, `service.yaml`, `ingress.yaml`, optional `pvc.yaml`, and
155
152
optional `secret.yaml`.
156
153
3. Put app-specific behavior directly in those manifests.
157
-
4. If the app needs secrets, create `secrets/<name>.yaml` and
158
-
`values/<name>.yaml`, then add that values file to the helmfile release.
159
-
5. Add the release to `../helmfile.yaml`.
154
+
4. If the app needs secrets, create first-class SOPS Secret manifests under
155
+
`../flux/secrets/` and reference those Secret names from the chart.
156
+
5. Add a HelmRelease under `../flux/apps/`.
160
157
6. Render before applying:
161
158
162
159
```bash
163
160
helm template <name> charts/<name>
164
-
helmfile -l name=<name> apply
161
+
flux reconcile kustomization apps -n flux-system
165
162
```
+1
-3
k8s/charts/cluster-tls/values.yaml
+1
-3
k8s/charts/cluster-tls/values.yaml
+1
-1
k8s/charts/forward-auth/values.yaml
+1
-1
k8s/charts/forward-auth/values.yaml
+2
-9
k8s/charts/longhorn-creds/values.yaml
+2
-9
k8s/charts/longhorn-creds/values.yaml
···
1
-
# Filled by values/secrets/longhorn-creds.yaml via vals + sops.
2
-
# Longhorn's helm release references the rendered Secret via
3
-
# defaultBackupStore.backupTargetCredentialSecret = longhorn-b2.
4
-
secret:
5
-
name: longhorn-b2
6
-
endpoint: https://s3.us-east-005.backblazeb2.com
7
-
virtualHostedStyle: "true"
8
-
keyId: ""
9
-
appKey: ""
1
+
# Longhorn's helm release references the first-class longhorn-b2 Secret via
2
+
# defaultBackupStore.backupTargetCredentialSecret.
10
3
11
4
# Daily backup of every volume in the implicit `default` group (i.e. all of
12
5
# them). retain is per-volume; concurrency caps parallel jobs cluster-wide.
+22
k8s/flux/external-routes/external-routes.yaml
+22
k8s/flux/external-routes/external-routes.yaml
···
1
+
apiVersion: helm.toolkit.fluxcd.io/v2
2
+
kind: HelmRelease
3
+
metadata:
4
+
name: external-routes
5
+
namespace: flux-system
6
+
spec:
7
+
interval: 15m
8
+
releaseName: external-routes
9
+
targetNamespace: default
10
+
chart:
11
+
spec:
12
+
chart: ./k8s/charts/external-routes
13
+
interval: 15m
14
+
reconcileStrategy: Revision
15
+
sourceRef:
16
+
kind: GitRepository
17
+
name: cute-haus
18
+
namespace: flux-system
19
+
valuesFrom:
20
+
- kind: ConfigMap
21
+
name: cute-haus-global
22
+
valuesKey: values.yaml
+1
-1
k8s/flux/external-routes/kustomization.yaml
+1
-1
k8s/flux/external-routes/kustomization.yaml
-9
k8s/flux/external-routes/placeholder.yaml
-9
k8s/flux/external-routes/placeholder.yaml
-43
k8s/helmfile.yaml
-43
k8s/helmfile.yaml
···
1
-
# Low history; CRD-heavy charts (cert-manager, traefik, cnpg) push each
2
-
# revision Secret to ~1MB, and the default of 10 stalls the apiserver.
3
-
helmDefaults:
4
-
historyMax: 3
5
-
6
-
repositories:
7
-
- name: grafana
8
-
url: https://grafana.github.io/helm-charts
9
-
- name: prometheus-community
10
-
url: https://prometheus-community.github.io/helm-charts
11
-
- name: traefik
12
-
url: https://traefik.github.io/charts
13
-
- name: longhorn
14
-
url: https://charts.longhorn.io
15
-
- name: cnpg
16
-
url: https://cloudnative-pg.github.io/charts
17
-
- name: tailscale
18
-
url: https://pkgs.tailscale.com/helmcharts
19
-
- name: jetstack
20
-
url: https://charts.jetstack.io
21
-
- name: nfd
22
-
url: https://kubernetes-sigs.github.io/node-feature-discovery/charts
23
-
- name: intel
24
-
url: https://intel.github.io/helm-charts/
25
-
26
-
releases:
27
-
# ─── Cluster infra ──────────────────────────────────────────────────────
28
-
# Ingress, storage, postgres operator, tailscale operator. These come
29
-
# before any app release that depends on their CRDs / StorageClasses.
30
-
31
-
# ─── Apps ───────────────────────────────────────────────────────────────
32
-
33
-
# Ingress + Service + EndpointSlice for services that live OUTSIDE the
34
-
# cluster (on tailnet IPs). e.g. plex/immich on jubilife, audiobookshelf
35
-
# on eterna. See charts/external-routes/values.yaml for the route list.
36
-
- name: external-routes
37
-
namespace: default
38
-
chart: ./charts/external-routes
39
-
needs:
40
-
- default/cluster-tls
41
-
- tailscale/tailscale-operator
42
-
values:
43
-
- values/global.yaml
+5
-8
k8s/values/README.md
+5
-8
k8s/values/README.md
···
1
1
# 📋 values/
2
2
3
-
Helmfile-provided value overrides.
3
+
Shared non-secret Helm values.
4
4
5
5
```
6
6
values/
7
-
└── secrets/ # vals refs (ref+sops://) into ../../secrets/
7
+
└── global.yaml # image pins and operational constants mirrored into Flux
8
8
```
9
9
10
-
Each file under `secrets/` is plain yaml containing only `vals` refs that
11
-
point into [`../../secrets/`](../../secrets/) for the actual encrypted values.
12
-
helmfile passes these as `values:` to a release; `vals` resolves the refs
13
-
at render time (SOPS-decrypts → splices into the values stream → helm
14
-
renders).
10
+
Secret values are first-class SOPS-encrypted Kubernetes Secret manifests under
11
+
[`../flux/secrets`](../flux/secrets). Do not put secret-like values here.
15
12
16
13
The chart's own `values.yaml` (under `charts/<name>/values.yaml`) holds
17
-
the in-tree, non-secret defaults. This dir holds only the secret overlay.
14
+
the in-tree, non-secret defaults. This dir holds only shared non-secret values.
18
15
19
16
See [`../charts/README.md`](../charts/README.md) for the full secret flow.
-6
k8s/values/secrets/bluesky-pds.yaml
-6
k8s/values/secrets/bluesky-pds.yaml
···
1
-
secret:
2
-
PDS_ADMIN_PASSWORD: ref+sops://../secrets/bluesky-pds.yaml#/secret/PDS_ADMIN_PASSWORD
3
-
PDS_EMAIL_FROM_ADDRESS: ref+sops://../secrets/bluesky-pds.yaml#/secret/PDS_EMAIL_FROM_ADDRESS
4
-
PDS_EMAIL_SMTP_URL: ref+sops://../secrets/bluesky-pds.yaml#/secret/PDS_EMAIL_SMTP_URL
5
-
PDS_JWT_SECRET: ref+sops://../secrets/bluesky-pds.yaml#/secret/PDS_JWT_SECRET
6
-
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX: ref+sops://../secrets/bluesky-pds.yaml#/secret/PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX
-2
nix/devShells.nix
-2
nix/devShells.nix
-1
nix/nixos/profiles/k3s.nix
-1
nix/nixos/profiles/k3s.nix
+9
-4
scripts/check-pinned-images.ts
+9
-4
scripts/check-pinned-images.ts
···
26
26
}
27
27
28
28
async function deployedChartNames(): Promise<Set<string>> {
29
-
const helmfile = Bun.YAML.parse(await Bun.file(HELMFILE).text()) as Helmfile;
30
29
const names = new Set<string>();
31
-
for (const release of helmfile.releases) {
32
-
if (release.chart.startsWith("./charts/")) {
33
-
names.add(release.chart.replace(/^\.\/charts\//, ""));
30
+
31
+
if (await Bun.file(HELMFILE).exists()) {
32
+
const helmfile = Bun.YAML.parse(
33
+
await Bun.file(HELMFILE).text(),
34
+
) as Helmfile;
35
+
for (const release of helmfile.releases) {
36
+
if (release.chart.startsWith("./charts/")) {
37
+
names.add(release.chart.replace(/^\.\/charts\//, ""));
38
+
}
34
39
}
35
40
}
36
41
+10
-4
scripts/check-release-names.ts
+10
-4
scripts/check-release-names.ts
···
51
51
return releases;
52
52
}
53
53
54
+
async function helmfileLocalReleases(): Promise<Release[]> {
55
+
if (!(await Bun.file(HELMFILE).exists())) return [];
56
+
57
+
const helmfile = Bun.YAML.parse(await Bun.file(HELMFILE).text()) as Helmfile;
58
+
return helmfile.releases.filter((release) =>
59
+
release.chart.startsWith("./charts/"),
60
+
);
61
+
}
62
+
54
63
function chartYamlPath(chart: string): string {
55
64
return (
56
65
chart.replace(/^\.\/charts\//, "k8s/charts/").replace(/^\.\//, "") +
···
59
68
}
60
69
61
70
export async function checkReleaseNames(): Promise<string[]> {
62
-
const helmfile = Bun.YAML.parse(await Bun.file(HELMFILE).text()) as Helmfile;
63
71
const errors: string[] = [];
64
72
65
73
const releases = [
66
-
...helmfile.releases.filter((release) =>
67
-
release.chart.startsWith("./charts/"),
68
-
),
74
+
...(await helmfileLocalReleases()),
69
75
...(await fluxLocalReleases()),
70
76
];
71
77