Insights:How to Configure Availability Monitoring & Alerting Tools: From Alerts to On-Call
Segment 1: Data Sources
- External synthetic monitoring: covers homepage and key pages, 2–5 regions, every 1–5 minutes.
- Internal metrics: CPU, memory, disk, and connections collected every minute.
- Synthetic transactions: key flows such as form submissions and logins, every 5–15 minutes.
After configuring each data source, verify it independently once to confirm data reports correctly.
Segment 2: Alert Rules
| Object | Rule | Parameters |
|---|---|---|
| Homepage | Consecutive failures | 3 times / 1 min |
| Key pages | Consecutive failures + content validation | 3 times |
| Database | Connection check | 2 consecutive failures |
| Certificate | Days remaining | 14-day warning / 7-day critical |
| Disk | Usage rate | 80% warning / 90% critical |
Rules must be documented, and any changes must be logged—otherwise no one will know why they were set six months later.
Segment 3: Alert Routing
- P1 (site down, core flow failure) → IM + phone call.
- P2 (single-page issue, response timeout) → IM.
- P3 (certificate expiring soon, backup failure) → email + daily report.
- Info (inspection results) → daily report only.
The key to routing is clear severity levels; don't let all alerts flood a single channel.
Segment 4: Deduplication & Aggregation
- Merge alerts from the same source within the same time window.
- Site-level incidents trigger a single notification with a list of affected pages.
- Maintenance windows are automatically muted and restored after completion.
- A recovery notification is sent upon restoration.
Without noise reduction, the team will mute group messages within a week.
Fifth: On-Call and Escalation
- The on-call schedule should specify the first responder, backup, and time slots.
- Escalation rules: 15 minutes no response → manager; 30 minutes → executive management.
- Alert content includes location details: URL, status code, response time, and region.
- Record the start and recovery time for every incident.
Sixth: Review and Cleanup
- Review total alerts and false-positive rate monthly.
- Clean up rules that haven't triggered in a long time quarterly.
- After each P1 incident, convert "human-discovered" issues into "monitorable" ones.
Acceptance: Simulate an Incident
- Manually stop the service and receive a P1 alert within 5 minutes.
- Receive a recovery notification after restoration.
- The alert should show which page and what error occurred.
- On-call staff can locate logs using the documentation.
Boundaries
Tool configuration ensures "signal reachability," but response quality depends on processes and people. Without an on-call system, even the best tools are just notifications in the status bar.
Related reading: