App Conventions#

Every App in this directory follows these conventions. Internal Apps (in this repo) source apps/lib/common.sh for shared deploy/teardown logic. External Apps (in other repos) must follow the same contract by convention.

Directory structure#

apps/<name>/
├── deploy.sh          # server-side, idempotent, never destroys
├── teardown.sh        # server-side, explicit, checks for dependents
├── deploy-to.sh       # local-side, rsyncs to server then runs deploy.sh
├── depends.conf       # optional: one dependency per line as app:resource-name
├── quadlets/          # .container files, one per Service
├── config/            # template-only: .example templates + static scaffolding
├── hosts/             # optional: per-Host user data overlay
│   ├── example/       # committed skeleton (real filenames + CHANGE_ME)
│   └── <host>/        # gitignored real data, mirrors config/ paths, overwrites by path
├── timers/            # optional: .timer + .service pairs for maintenance
├── scripts/           # optional: maintenance scripts called by timer services
├── .env.example       # optional: template for secrets
└── .gitignore         # ignores .env and real host dirs (keeps hosts/example/)

Service naming#

Every Service is named <app>-<service> (e.g. proxy-caddy-tls). This name is used consistently for:

• The Quadlet filename: proxy-caddy-tls.container
• The systemd service: proxy-caddy-tls.service
• The Podman container name: ContainerName=proxy-caddy-tls

The app prefix prevents cross-App collisions and makes ownership visible in podman ps.

Ownership labels#

Every .container file MUST set:

Label=infra.app=<app-name>
Label=infra.service=<service-name>

These are runtime labels (set by the Quadlet, not baked into the image). They are the source of truth for which App owns a container. teardown.sh uses them to detect dependents. An omitted label breaks both this App's teardown and other Apps' ability to detect it as a dependent.

Deploy script (deploy.sh)#

Runs on the server, in the App root (~/apps/<name>/). Takes no arguments. Does:

1. Source ~/apps/lib/common.sh
2. Create networks (idempotent: podman network create <name> || true)
3. Validate dependencies from depends.conf (fail with clear message if missing)
4. Copy Quadlet files to ~/.config/containers/systemd/
5. systemctl --user daemon-reload
6. Start/restart every Service (systemctl --user restart <name>)
7. Print status

Invariant: deploy.sh never destroys. It never removes networks, volumes, or containers. Redeploy is always safe for dependents.

Teardown script (teardown.sh)#

Runs on the server, in the App root. Takes no arguments. Prompts [y/N] for confirmation. Does:

1. Source ~/apps/lib/common.sh
2. Stop and disable all Services
3. Remove Quadlet files from ~/.config/containers/systemd/
4. systemctl --user daemon-reload
5. For each network this App owns:
   • List all containers on the network
   • If any container has infra.app != <this-app> → refuse ("App X still depends on network Y")
   • If only this App's containers or none → podman network rm <name>
6. Print confirmation

Never removes volumes. Volumes are permanent data loss. Reclaim disk space manually with podman volume rm. Does not podman rm containers. Stopping the systemd service is the clean break; podman GC handles the rest.

Dependencies (depends.conf)#

Plain text, one dependency per line as <app>:<resource-name>. The resource name is a Podman resource name (e.g. a network name). deploy.sh checks each exists before proceeding. Absent or empty file means no dependencies.

Example:

# apps/splint/depends.conf
proxy:caddy

Maintenance timers#

An App may include user-level systemd timers for periodic maintenance: backups, health watchdogs, cert refreshes. Optional — most Apps have none.

Timers live in timers/ as .timer + .service pairs. The service files call maintenance scripts in scripts/ via ExecStart. All units are user-level (installed to ~/.config/systemd/user/, not the Quadlet directory).

deploy.sh installs and enables timers after starting Services:

1. install_timers timers — copies .timer and .service files to ~/.config/systemd/user/
2. reload_systemd — systemctl --user daemon-reload
3. enable_timers timers — enables and starts each .timer unit

teardown.sh disables and removes timers before removing networks:

1. disable_timers timers — stops and disables each timer
2. remove_timers timers — removes the unit files
3. reload_systemd

Timer and service unit names SHOULD be app-prefixed (e.g. mail-backup.timer, mail-watchdog.timer) to avoid collisions with other Apps' units.

Secrets#

Secrets are local, gitignored, and synced to the VPS by the deploy wrapper (deploy-to.sh) — the posture amended in ADR 0003. There is no longer a "created on the server, never synced" class.

1. .env — the App-root environment file, loaded by Quadlets via EnvironmentFile=%h/apps/<name>/.env and sourced by maintenance scripts. .env.example is the committed template; .env is gitignored. The sync wrapper sources apps/lib/common.sh and runs ensure_env locally before syncing: it copies .env.example → .env when .env is missing and auto-generates any *_SECRET values on the operator's machine, so first deploy produces a working .env without a manual openssl ritual. The wrapper is agnostic to how .env got its contents — hand-edit it, copy from .env.example, or provision it from a secret store such as Skate (skate get … > apps/<name>/.env) before deploy. It simply syncs whatever .env exists.

2. Secret configs — config files with embedded credentials that are not KEY=value pairs (e.g. frps.toml with a dashboard password). These are per-Host application data: they live in hosts/<host>/ (gitignored), reach the server via the hosts/<host>/ → config/ overlay, and are bind-mounted from config/ by Quadlets. See "Host-specific config" below.

deploy.sh (server-side, post-sync) verifies completeness: check_secret_files fails if a required file is missing, and check_placeholders fails if any synced file (.env or a file under config/) still contains a CHANGE_ME placeholder. Run grep -r CHANGE_ME locally before deploying to self-check; see the repo README's "Getting started → The CHANGE_ME convention" for the marker's contract.

Secrets never live in the repo. Back up the local .env and the hosts/<host>/ reals — they are not recoverable from the repo.

Sync wrapper (deploy-to.sh)#

Runs locally. Takes one argument: the target host. It first sources apps/lib/common.sh and runs ensure_env locally — provisioning .env from .env.example (auto-generating *_SECRET values) when .env is missing. Then it syncs files to the server and SSHs in to run deploy.sh:

1. rsync config/ → ~/apps/<name>/config/
2. rsync hosts/<host>/ → ~/apps/<name>/config/ (overlays same-named files)
3. rsync quadlets/, timers/, scripts/, deploy.sh, teardown.sh, depends.conf → ~/apps/<name>/
4. rsync .env → ~/apps/<name>/.env (guarded by local existence)
5. rsync apps/lib/ → ~/apps/lib/
6. ssh <host> 'cd ~/apps/<name> && ./deploy.sh'

The wrapper syncs .env only if it exists locally; it does not fail when .env is absent (the post-sync check_secret_files existence check handles a truly missing required file). It no longer ships .example templates to the server as references — templates are local copy-sources only. It remains agnostic to how .env got its contents.

Host-specific config#

Most Apps have no hosts/ directory. For Apps with per-Host data (different domains, distinct upstreams per VPS) the split is by role, not just by file:

• config/ is template-only. It holds .example templates and static scaffolding (empty bind-mount directories with .gitkeep). It no longer holds real bind-mounted user data.
• Real bind-mounted data files (Caddyfiles, frps.toml, etc.) live in hosts/<host>/, mirroring the config/ path structure. They reach the server exclusively via the hosts/<host>/ → config/ overlay applied by the sync wrapper; on the server the App sees a flat config/ and Quadlet files stay host-agnostic.
• hosts/example/ is the committed skeleton: a template directory containing one copy of every data file a host needs, using real filenames (Caddyfile, frps.toml — not .example) with CHANGE_ME placeholder content. To stand up a new host: cp -r hosts/example hosts/<my-vps> then fill every CHANGE_ME.
• A data file that carries embedded credentials (e.g. frps.toml with a dashboard password) is treated as per-Host data — it lives in hosts/<host>/ and is synced by the wrapper, not as a config/-local secret config.
• Real host dirs are gitignored. Each App's .gitignore ignores hosts/* while negating the skeleton: !hosts/example/. Operator-supplied data never gets committed.
• The sync wrapper merges: config/ first, then hosts/<host>/ overwrites same-named files.

Invariant: a file in config/ cannot be removed for a specific host via overlay — it always syncs. Because real data now lives in hosts/<host>/, "removing" a data file for a host means simply not placing it in that host dir (scaffolding-only config/ files still sync to every host). A leftover CHANGE_ME in a synced file fails deploy.sh with a clear message.