sentinel
Alerts & states
Meaning behind the different states a monitor can be in.
Monitor state
A monitor can be in one of the following four states:
| state | meaning |
|---|---|
| waiting | created, first run not due yet |
| ok | a good backup arrived on schedule |
| failed | the latest run reported an error |
| overdue | no good backup by deadline + grace — the alerting state |
When alerts fire
Alert fire when a previously healthy monitor is overdue or failing - a successful ping hasn't been for the current period + the grace period. Alerts are only sent once per transition.
Once the monitor recovers by receiving a good backup ping, another alert will be sent to notify successfuly recovery.
A failing backup is not a live backup
The clock measures from the last good run. A box that pings faithfully and fails every
time still goes overdue — "running and failing" and "stopped running"
both mean you have no recent backup, and only a good run clears the alert.
Take a monitor scheduled 0 3 * * * (every day at 03:00) with one
hour of grace:
| when | what happens | state | alert |
|---|---|---|---|
| Mon 03:00 | backup runs and reports success | ok | — |
| Tue 03:00 | backup runs but reports an error — not a sign of life | failed | — |
| Tue 04:00 | grace runs out with no good backup since Monday | overdue | overdue |
| Tue 05:00… | still no good backup | overdue | — (never re-alerts) |
| Wed 03:00 | backup runs and reports success | ok | recovered |
Note what the Tuesday failure did not do: it fired no alert of its own, and it did not push the deadline back. The alert came from the absence of a good backup, and only Wednesday's good run cleared it.
Version mismatch
pg_dump must be at least as new as the PostgreSQL server it reads
— an older client can write a dump that is incomplete or refuses to restore
The gem reports both versions with every run, and Sentinel compares their major versions. If the client is behind, a warning appears next to the monitor in your list and on the monitor itself.
Repeat alerts
§ One alert is not enough — a missed message means a dead backup nobody hears about.
§ While a monitor stays overdue it is alerted again: every 6 hours for the first day, then once a day.
§ Repeats stop after ~30 days of an unattended outage; a recovery resets the whole cycle.
§ Copy reads "still OVERDUE" from the second alert onward, so a repeat isn't mistaken for a new incident.
§ Webhook receivers: each repeat is a new alert `id` with an incrementing `sequence` — dedupe on `id`, use `sequence` to tell first from repeat.
§ Nothing to configure; the schedule is the same for every monitor.
Delivery guarantees
We try to guarantee at-least-once delivery for every configured integration.
Every alert is persisted before delivery and is retried independently up to 5 times before giving up. If your webhook integration fails, this won't affect the Slack notification, for example.