sentinel
Alerts & states
Meaning behind the different states a monitor can be in.
Monitor state
A monitor can be in one of the following four states:
| state | meaning |
|---|---|
| waiting | created, first run not due yet |
| ok | a good backup arrived on schedule |
| failed | the latest run reported an error |
| overdue | no good backup by deadline + grace — the alerting state |
When alerts fire
Alert fire when a previously healthy monitor is overdue or failing - a successful ping hasn't been for the current period + the grace period. Alerts are only sent once per transition.
Once the monitor recovers by receiving a good backup ping, another alert will be sent to notify successfuly recovery.
A failing backup is not a live backup
The clock measures from the last good run. A box that pings faithfully and fails every
time still goes overdue — "running and failing" and "stopped running"
both mean you have no recent backup, and only a good run clears the alert.
Take a monitor scheduled 0 3 * * * (every day at 03:00) with
one hour of grace:
| when | what happens | state | alert |
|---|---|---|---|
| Mon 03:00 | backup runs and reports success | ok | — |
| Tue 03:00 | backup runs but reports an error — not a sign of life | failed | — |
| Tue 04:00 | grace runs out with no good backup since Monday | overdue | overdue |
| Tue 05:00… | still no good backup | overdue | — (never re-alerts) |
| Wed 03:00 | backup runs and reports success | ok | recovered |
Note what the Tuesday failure did not do: it fired no alert of its own, and it did not push the deadline back. The alert came from the absence of a good backup, and only Wednesday's good run cleared it.
Delivery guarantees
We try to guarantee at-least-once delivery for every configured integration.
Every alert is persisted before delivery and is retried independently up to 5 times before giving up. If your webhook integration fails, this won't affect the Slack notification, for example.