Skip to content
A personal experiment · alpha · code not public yet · no offer, no support Project status →

Operations processes

Teams think in processes: patching, releases, incidents, restores. Vaicoa expresses them as signed catalog entries and rules the core and the nodes enforce. Vaicoa is the approval layer, not a ticket system: it refers to your tickets.

A playbook is a signed sequence of catalog actions with checks between the steps, approved once as a whole plan.

  • The core expands the full plan in advance: every step, every target, every check, the stop rule. You approve exactly that expansion, by digest.
  • Every step runs as its own op carrying the playbook reference; the nodes check each step against the approved plan and refuse a step that already ran.
  • A red check stops the run. A rollback that was part of the plan runs at once, without a new approval.
  • maxShare limits the share of a cluster a run touches (per batch for rolling playbooks); a larger plan is refused before anyone is asked.
PlaybookWhat it does
patch.windowper node: cordon, drain, OS update, reboot, wait until Ready with a new boot id, uncordon; one node at a time
release.rolloutrelease tag, Flux reconcile, a canary share of the replicas, a check; on red the replicas go back
incident.first_responsereads events, logs and metrics of a workload, acknowledges the alert, writes the findings into the incident
restore.drillchecks the core’s backup, restores your newest backups into throw-away objects, checks and deletes them
gameday.runkills a pod, isolates a node, adds latency, each with a check; never in production

Agents see playbooks as vaicoa-run tools named after them.

Normal changeproposed, approved by the quorum, run
Standard changea template in the signed roster: which routine actions run without their own approval, within which bounds (a CEL expression), in which environment, for how long. Allowed under starter, an audited weakening under standard, never under strict.
Emergency changewith an incident reference, self-confirmed, reviewed by two owners within 24 hours; until then owners are woken and the change shows red. Nothing destructive; off under strict.
Freeze and maintenance windowsin the roster, binding for the nodes too. During a freeze normal and standard changes are refused; incidents and alerts stay possible.
Error budgetfrom SLO files and metrics.query. When a service’s budget is spent its changes are frozen (or need one more approval); changes labelled reliability pass and are counted. If the metric source is unreachable the core refuses rather than waves through.
Readiness reviewan environment becomes production only through env.production, whose plan is a checklist the core checks: SLO files, alert routes, a backup target, a passed restore drill no older than 90 days, playbooks.
  • Incidents are references to your ticket (for example jira:OPS-42). Ops proposed with that reference belong to it; results can be written back as a Forgejo comment or a signed webhook.
  • Alerts can be silenced in Alertmanager for at most 24 hours, only with specific matchers.
  • Postmortems (report.create kind=postmortem) build the timeline from the audit and name roles, not people.
  • Restores of CloudNativePG databases and volumes go into new objects; replacing a cluster is destructive and needs the typed name.
  • Restore drills and game days end with a report the core signs with its audit key, with measured RPO and RTO.
  • cert.renew, secret.rotate (without anyone seeing the value), host.isolate and its release, host.packages.
  • Access review every 90 days: members, devices, roles, tunnel grants, session templates. What is not confirmed within the deadline lapses through a roster change the core drafts; nobody can be locked out by it.
  • Reports from the audit: DORA metrics, toil, on-call; each key figure next to its counter-metric.