caravan#

Declarative provisioning that converges a provider to a spec.

Overview#

caravan reconciles what an infrastructure provider reports against a declared target. A pure planner compares observed state with the spec and returns the ordered actions that close the gap -- or refuses with a conflict when closing it would destroy something. A provider backend then executes the plan, step by step, waiting for each resource to settle.

Three rules shape every plan:

• The data volume is never destroyed. The one resource whose loss is unrecoverable can never be deleted, shrunk, or reformatted by a plan. Any mismatch on it surfaces as a conflict to resolve explicitly, never as a synthesized destructive action.
• Compute is disposable. Reconciling immutable bootstrap state -- a new boot image, a new instance type -- stops and replaces the instance, but first detaches and preserves its data volume and routed address.
• Routine updates are out of scope. An instance whose running system image drifted by itself -- an over-the-air update -- keeps its provision tag and is left alone. In-place system updates belong to the update channel (uplink); caravan only replaces compute when the declared provision input changes.

caravan sits between two companion tools: uniboot builds the immutable image and publishes it, caravan creates the provider resources that run it and keeps them matching the declaration, and uplink moves signed updates onto machines that already run.

The planner is I/O-free and provider-neutral. Backends implement the Caravan.PROVIDER contract; Scaleway (caravan.scaleway, over scaleway) is the first.

Installation#

Install with opam:

$ opam install caravan

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install caravan

Usage#

Declare a target in YAML:

name: camel
provider: scaleway
instance-type: STANDARD2-A2C-8G
image: 11111111-2222-4222-8222-222222222222
volume-gib: 50
inbound-tcp: [80, 443]
registry: camel
bootstrap-file: ./bootstrap.yaml

Then:

$ caravan plan -c camel.yaml      # the actions that would run, billable marked
$ caravan apply -c camel.yaml     # show the plan, confirm, execute
$ caravan status -c camel.yaml    # what the provider reports; converged or not

plan prints the numbered actions that would close the gap. Each step that starts recurring charges is marked with its price when the provider's catalog exposes one ([EUR 7.99/month], from Scaleway's server-type catalog) and with [provider charge] otherwise; a total: line sums the priced steps and says when unpriced charged steps remain. apply executes the plan after confirmation (--yes skips it), printing each step as it runs and finishing with the routed address and registry endpoint.

Some drift cannot be closed without destroying data: the observed volume is smaller than the spec asks for, a stale attachment from a deleted server still holds it, a volume of the wrong storage class sits under the expected name. The planner never turns these into delete or resize steps that would ride into an apply -- it stops with a conflict naming the resource and the mismatch. Resolving one is explicit, in three tiers:

• apply alone never resolves a conflict.
• apply --force runs the lossless resolutions, printing each step: detaching a data volume from a stale holder, adopting a same-named resource caravan did not create by tagging it.
• apply --destroy-data runs the destructive one: the volume that cannot be reconciled -- too small, wrong class, or foreign -- is detached from every holder, deleted, and recreated empty on the next pass. An interactive run must retype the volume's name (--yes does not skip that; without a terminal, --yes plus --destroy-data is the authorization), and the same name is checked again server-side by the provider call. When one conflict has both a lossless and a destructive fix, --force wins.

The provider console is never required for a resource caravan owns; plan shows which flag applies when it reports a conflict. A plan by itself never contains a destroy step -- destruction only ever comes from the explicit resolution path. The remaining alternative is always to change the target to match reality (for example, declare the size the volume actually has) and run plan again.

Credentials resolve the way the scw CLI resolves them (environment, then profile); --profile selects a named profile.

caravan keeps no state file; the provider is the only source of truth. Two mechanisms make that work:

• Identity by name and provenance tag. A target named camel looks up its resources by the names the planner assigns -- instance camel, data volume camel-data, firewall camel-fw -- but a name alone is not proof of ownership: provider names are not unique, and a same-named resource may be a stale leftover or a plant. So caravan writes the identity tag caravan:camel on every taggable resource it creates, and resolution requires name and tag together. A same-named resource without the tag is a Foreign_resource conflict, never a silent adoption -- for the data volume, adoption decides what the node's persistent state is, so it must be a decision. apply --force adopts one explicitly by recording the tag (how a pre-caravan volume is taken over, once, on purpose). Routed addresses have no provider name at all, so for them the tag is also the only handle. Registry namespaces take no tags; their names are region-unique at the provider and image integrity rides on digest pinning, not registry identity.
• Provision state as an instance tag. When caravan creates an instance, it writes one more tag on it: caravan:provision:<sha256-prefix of the target's image reference>. On every plan, observe reads the tags back and compares that fingerprint against the one computed from the current target file. Equal means the instance is the realization of the current declaration, and the image the instance actually runs is ignored -- self-applied updates are legitimate drift. Different means the declaration changed since the instance was provisioned, and the plan replaces it: stop, detach the data volume and the address, delete, recreate from the new image, reattach. A managed instance with no provision tag at all is adopted in place: Record_provision writes the tag without a restart.

The comparison is declared-fingerprint against recorded-fingerprint, never declared-image against observed-image -- that is what lets in-place updates over uplink coexist with declarative replacement.

Library#

The same engine is a library. Caravan.plan is pure -- spec in, observed state in, actions or conflict out -- so reconciliation logic tests without a provider. Caravan.Make (P) executes plans over any Caravan.PROVIDER:

module Engine = Caravan.Make (Caravan_scaleway)

let converge backend spec =
  match Caravan_scaleway.observe backend spec with
  | Error e -> Error (`Provider e)
  | Ok observed -> (
      match Caravan.plan spec observed with
      | Error conflict -> Error (`Conflict conflict)
      | Ok actions -> (
          match Engine.apply backend spec actions with
          | Ok applied -> Ok applied
          | Error (action, e) -> Error (`Step (action, e))))

Bootstrap documents are opaque Caravan.Bootstrap.t values: every printer in the library redacts them, so a plan rendered into a log never leaks the payload.

Related work and credits#

• InfraKit (Docker, archived) is where the concept comes from: pair an immutable-image builder with a toolkit that actively converges infrastructure to a declared spec. InfraKit played that role next to LinuxKit; caravan plays it next to uniboot, which is itself LinuxKit-inspired. The architecture differs -- caravan is a one-shot observe/plan/apply with explicit conflicts and no plugin daemons, and shares no code with InfraKit -- but the pairing is InfraKit's idea, and so is recording the provisioned configuration as a tag on the instance instead of in a state file: InfraKit's group plugin tagged every instance with its config SHA and reconciled by comparing tags, which is exactly what the provision tag does here. Kubernetes uses the same mechanism as the pod-template-hash label.
• Terraform established the plan-then-apply workflow with a human-reviewed plan; caravan's CLI follows that shape, scoped to one target.

License#

ISC License. See LICENSE.md for details.