Alert rules

Alert rules are part of Vaicoa Pro and Vaicoa Team. A free account’s uptime checks show their results but send no alerts.

Every rule Vaicoa evaluates for your team. To follow an alert from the notification to the cause, see When something needs attention.

How to read the tables

Column	Meaning
Notification	on iPhone, the title of the push notification, after Critical: or Warning:
Rule	the rule_key first, then the rule’s internal name. The app shows the rule_key under the alert’s title (Warning · disk_space_low), webhooks send it as alert.rule, PagerDuty as custom_details.rule
Severity	critical or warning
Fires when	the condition, and how long it must hold before the alert fires

Rules are evaluated every 30 seconds. The thresholds are the same for every team. The only values you set yourself are a check’s latency budget and a job’s maximum age.

Servers: Hetzner Cloud metrics

For cloud servers in a connected project, from the metrics Hetzner reports for them. Dedicated servers have no Hetzner metrics, so these two rules do not cover them.

Notification	Rule	Severity	Fires when
server offline	host_offline · HostOffline	critical	Hetzner lists the server as running but has sent no metrics for it for more than 3 minutes. Keeps firing for as long as that lasts; resolves when metrics return, or when the server is powered off or deleted
CPU saturated	cpu_saturated · CPUSaturated	warning	Hetzner’s CPU value for the server is above 95 % for 15 minutes

• server offline never fires for a server that Hetzner reports as off, including one powered off between two syncs, for a deleted server, while the Hetzner API cannot be reached or refuses the project’s token (the token alert covers that), or when Hetzner goes quiet for every team at once.
• A server powered off without you doing it does not page either: Hetzner reports it as off and gives no reason. An uptime check on its address watches for that.

Servers: the agent stops reporting

For servers that run the agent. The agent sends metrics every 30 seconds; this rule fires when they stop arriving.

Notification	Rule	Severity	Fires when
stopped reporting	agent_silent · AgentSilent	critical	A dedicated server, or a cloud server that is not linked to its Hetzner resource, has sent no agent metrics for more than 5 minutes
stopped reporting	agent_silent · AgentSilent	warning	A cloud server’s agent has sent no metrics for more than 5 minutes while Hetzner still delivers the server’s own metrics

• The alert keeps firing for as long as the server is silent and resolves with the first new report.
• Critical where the agent is the only signal: a dedicated server, or a cloud server registered with its own agent token rather than a fleet token, which Vaicoa cannot match to its Hetzner resource. The alert cannot tell a server that is down from an agent that stopped; an uptime check on the server’s address can.
• Warning for a cloud server that Hetzner still reports: the server runs, but until the agent reports again there are no disk, memory or log alerts for it. systemctl status cloudviewer-agent on the server shows why.
• When Hetzner is silent too, only server offline fires, not both. A cloud server that is not linked to its Hetzner resource can get both.
• It does not fire for a server that has never reported (Waiting for agent) or only sends job reports, a retired server, a server whose tokens are revoked, a team that is not on Pro or Team, a linked cloud server that Hetzner does not report as running (powered off, being rebuilt), or a check target.
• If every team’s agents go silent at once, the cause is on our side, and the rule holds back instead of paging everyone.

Servers: the agent

For cloud and dedicated servers that run the agent, from what it measures on the host.

Notification	Rule	Severity	Fires when
disk space low	disk_space_low · DiskSpaceLow	warning	A filesystem is more than 80 % full for 10 minutes
disk almost full	disk_space_critical · DiskSpaceCritical	critical	A filesystem is more than 90 % full for 5 minutes
disk fills within a day	disk_fills_within_24h · DiskFillsWithin24h	warning	A filesystem is more than 60 % full and, at the rate of both the last six hours and the last hour, full within 24 hours, for 90 minutes
disk fills within hours	disk_fills_within_4h · DiskFillsWithin4h	critical	A filesystem is more than 70 % full and, at the rate of both the last two hours and the last half hour, full within 4 hours, for 45 minutes
memory pressure	memory_pressure · MemoryPressure	warning	Memory use is above 90 % for 10 minutes

• Disk rules apply to each mount point separately. Memory-backed and image filesystems (tmpfs, devtmpfs, ramfs, squashfs, erofs, overlay and similar) are skipped, and so are the volume mounts of single pods on a Kubernetes node (/var/lib/kubelet/pods/…).
• The two forecasts need both rates to agree, so a single burst, such as an image pull or a restore, does not fire them. Only steady growth does.
• Memory use is total minus available memory, so page cache the kernel can free does not count.

Servers: LVM thin pools

For servers that report their LVM thin pools as node_lvm_thin_pool_data_percent and node_lvm_thin_pool_metadata_percent (by vg and lv, the percentages lvs shows), as PaaSbox hosts do.

Notification	Rule	Severity	Fires when
thin pool filling	thin_pool_space_low · ThinPoolSpaceLow	warning	A thin pool’s data is more than 80 % full for 15 minutes
thin pool almost full	thin_pool_space_critical · ThinPoolSpaceCritical	critical	A thin pool’s data is more than 90 % full for 5 minutes
thin-pool metadata filling	thin_pool_metadata_low · ThinPoolMetadataLow	warning	A thin pool’s metadata is more than 80 % full for 15 minutes
thin-pool metadata almost full	thin_pool_metadata_critical · ThinPoolMetadataCritical	critical	A thin pool’s metadata is more than 90 % full for 5 minutes

• A full pool, or full metadata, fails every write to every volume in the pool, even when each volume’s own filesystem has space left.

Log lines: the agent

From the warning-level journald lines the agent sends for log context. Only lines that were stored count, after the per-server limit of 120 lines a minute.

Notification	Rule	Severity	Fires when
critical log lines	log_critical_lines · CriticalLogLine	warning	At least one line at priority 0–2 (emergency, alert, critical: for example a kernel oops or an out-of-memory kill) in the last 10 minutes. Fires at once and resolves 10 minutes after the last such line
error-log burst	log_error_burst · ErrorLogBurst	warning	More than 30 lines at priority 0–3 (error or worse) within 15 minutes, for 5 minutes

SSH and sudo lines have no rule.

Job reports

From your job reports.

Notification	Rule	Severity	Fires when
job failed	user_job_failed · UserJobFailed	warning	The job’s latest report has a non-zero exit code, for 5 minutes
job stale	user_job_stale · UserJobStale	warning	The job’s last successful run is older than the max_age_seconds it declared, for 5 minutes

• A job that declares no max_age_seconds only gets job failed.
• Stale counts from the last successful run. A job that has never succeeded never goes stale.

Uptime checks

From your uptime checks. A check is Down only when both networks agree. Mixed and No data never alert.

Notification	Rule	Severity	Fires when
unreachable from outside	check_down · CheckDown	critical	The check is Down for 1 minute
certificate expiring	check_cert_expiring · CheckCertExpiring	warning	The certificate has less than 14 days left, for 15 minutes
certificate expires imminently	check_cert_expiring_critical · CheckCertExpiringCritical	critical	The certificate has less than 3 days left, for 15 minutes
slower than its budget	check_slow · CheckSlow	warning	The response time from either network is above the check’s latency budget for 10 minutes. Only for checks with a budget, which you set in the portal

The notification says “unreachable” rather than “down”, because it can appear next to a resource that Hetzner still reports as running.

Suppressed alerts

While the critical alert fires, the matching warning is not sent.

While this fires	it suppresses	on the same
disk_space_critical	disk_space_low	server and mount point
disk_fills_within_4h	disk_fills_within_24h	server and mount point
thin_pool_space_critical	thin_pool_space_low	server and thin pool
thin_pool_metadata_critical	thin_pool_metadata_low	server and thin pool
check_cert_expiring_critical	check_cert_expiring	check

Where alerts go

Channel	Severities	Muted alerts
The app: Active alerts, History, the resource’s screen	all	shown, marked as muted
Push notifications	all; critical ones are time-sensitive on iPhone	not sent
Email digest	warning and critical, or critical only	never included
Alert targets: webhook, PagerDuty	warning and critical, or critical only, per target	never sent

The digest and alert targets never carry alerts with severity info. None of the rules on this page is info. Setting up the digest and targets: Send alerts where you work.

Mutes

• Durations: 1 hour, 4 hours or 24 hours. The app can end a mute early with Unmute and lengthen it with Extend mute.
• A muted alert keeps firing. It stays visible, marked as muted, and is recorded in the history.
• Any team member can mute an alert in the app.

History

• Alert history is kept for 90 days.
• History in the app shows the last 30 days. Fired N× this week counts the last 7 days.
• The Claude connector’s get_alert_history returns the last 7 days.

What has no rule

• Load balancers, floating IPs, volumes, networks and primary IPs have no rules of their own. To watch a load balancer or a floating IP, add an uptime check anchored to it in the portal; its alerts then appear on that resource.
• CPU on dedicated servers: the CPU rule reads Hetzner’s Cloud metrics, which dedicated servers do not have.
• Load average, disk I/O and network traffic: charted, no rule.