A personal experiment · alpha · code not public yet · no offer, no support Project status →
Operations processes
Teams think in processes: patching, releases, incidents, restores. Vaicoa expresses them as signed catalog entries and rules the core and the nodes enforce. Vaicoa is the approval layer, not a ticket system: it refers to your tickets.
Playbooks: one plan, one approval
Section titled “Playbooks: one plan, one approval”A playbook is a signed sequence of catalog actions with checks between the steps, approved once as a whole plan.
- The core expands the full plan in advance: every step, every target, every check, the stop rule. You approve exactly that expansion, by digest.
- Every step runs as its own op carrying the playbook reference; the nodes check each step against the approved plan and refuse a step that already ran.
- A red check stops the run. A rollback that was part of the plan runs at once, without a new approval.
maxSharelimits the share of a cluster a run touches (per batch for rolling playbooks); a larger plan is refused before anyone is asked.
| Playbook | What it does |
|---|---|
patch.window | per node: cordon, drain, OS update, reboot, wait until Ready with a new boot id, uncordon; one node at a time |
release.rollout | release tag, Flux reconcile, a canary share of the replicas, a check; on red the replicas go back |
incident.first_response | reads events, logs and metrics of a workload, acknowledges the alert, writes the findings into the incident |
restore.drill | checks the core’s backup, restores your newest backups into throw-away objects, checks and deletes them |
gameday.run | kills a pod, isolates a node, adds latency, each with a check; never in production |
Agents see playbooks as vaicoa-run tools named after them.
Change types and the calendar
Section titled “Change types and the calendar”| Normal change | proposed, approved by the quorum, run |
| Standard change | a template in the signed roster: which routine actions run without their own approval, within which bounds (a CEL expression), in which environment, for how long. Allowed under starter, an audited weakening under standard, never under strict. |
| Emergency change | with an incident reference, self-confirmed, reviewed by two owners within 24 hours; until then owners are woken and the change shows red. Nothing destructive; off under strict. |
| Freeze and maintenance windows | in the roster, binding for the nodes too. During a freeze normal and standard changes are refused; incidents and alerts stay possible. |
| Error budget | from SLO files and metrics.query. When a service’s budget is spent its changes are frozen (or need one more approval); changes labelled reliability pass and are counted. If the metric source is unreachable the core refuses rather than waves through. |
| Readiness review | an environment becomes production only through env.production, whose plan is a checklist the core checks: SLO files, alert routes, a backup target, a passed restore drill no older than 90 days, playbooks. |
Incidents, postmortems, continuity
Section titled “Incidents, postmortems, continuity”- Incidents are references to your ticket (for example
jira:OPS-42). Ops proposed with that reference belong to it; results can be written back as a Forgejo comment or a signed webhook. - Alerts can be silenced in Alertmanager for at most 24 hours, only with specific matchers.
- Postmortems (
report.create kind=postmortem) build the timeline from the audit and name roles, not people. - Restores of CloudNativePG databases and volumes go into new objects; replacing a cluster is destructive and needs the typed name.
- Restore drills and game days end with a report the core signs with its audit key, with measured RPO and RTO.
Security operations and reviews
Section titled “Security operations and reviews”cert.renew,secret.rotate(without anyone seeing the value),host.isolateand its release,host.packages.- Access review every 90 days: members, devices, roles, tunnel grants, session templates. What is not confirmed within the deadline lapses through a roster change the core drafts; nobody can be locked out by it.
- Reports from the audit: DORA metrics, toil, on-call; each key figure next to its counter-metric.