Backups and recovery tests

The hub’s own backup Built

• The hub’s SQLite file is replicated continuously (Litestream, a unit of its own beside the hub) to a private bucket in your project, made by vai up.
• Every secret in it (tokens, S3 keys, passwords, keys) is sealed with the hub’s data keys, and the data keys are not in the bucket. Metadata (server names, addresses, the audit, device names) is readable to whoever holds the bucket’s key.
• The recovery key, shown once by vai up, restores the data keys onto a new server.

Measured on Hetzner (20 October 2026): the hub’s server deleted, a new one on the same primary IPv4, the database restored from the bucket in 4.0 s, the hub up 66 s after the delete; the devices’ keys and pairings held, the servers resumed without enrolling again. Data written in the last second before the delete was lost (RPO ≈1 s).

Your servers’ data Built

Each production or staging server backs up its data volume into a Kopia repository of its own in the same bucket, under its own folder. Each repository has its own password, made by the hub, sealed, never shown; Kopia encrypts every blob with it. A twin with a copy and a recovery test read production’s repository; they write into none.

A Hetzner S3 key reaches every bucket of its project. A separate backup project, so that the hub’s key reaches nothing else, is offered as an option and is not the default.

Recovery tests Built

$ vai api POST /recovery-tests '{"production": "web-1"}'

A recovery test restores a production server’s data on a throwaway server in the twins’ project, on production’s release and runtime, checks it, deletes the server and writes a report: the milestones with their times (requested, server created, enrolled, reported, deleted), the checks, and what the test does not cover. It opens no port. One test per server at a time.

Measured on Hetzner (stage-2 gate, 20–21 October 2026): start to server deleted, 64 s / 75 s.