How to Configure Availability Monitoring & Alerting Tools: From Alerts to On-Call

The essence of tool configuration is 'getting the right signal to the right person.' Misordered setup produces a flood of unattended alerts. Below we walk throu…

Insights:How to Configure Availability Monitoring & Alerting Tools: From Alerts to On-Call

Segment 1: Data Sources

  • External synthetic monitoring: covers homepage and key pages, 2–5 regions, every 1–5 minutes.
  • Internal metrics: CPU, memory, disk, and connections collected every minute.
  • Synthetic transactions: key flows such as form submissions and logins, every 5–15 minutes.

After configuring each data source, verify it independently once to confirm data reports correctly.

Segment 2: Alert Rules

Object Rule Parameters
Homepage Consecutive failures 3 times / 1 min
Key pages Consecutive failures + content validation 3 times
Database Connection check 2 consecutive failures
Certificate Days remaining 14-day warning / 7-day critical
Disk Usage rate 80% warning / 90% critical

Rules must be documented, and any changes must be logged—otherwise no one will know why they were set six months later.

Segment 3: Alert Routing

  • P1 (site down, core flow failure) → IM + phone call.
  • P2 (single-page issue, response timeout) → IM.
  • P3 (certificate expiring soon, backup failure) → email + daily report.
  • Info (inspection results) → daily report only.

The key to routing is clear severity levels; don't let all alerts flood a single channel.

Segment 4: Deduplication & Aggregation

  • Merge alerts from the same source within the same time window.
  • Site-level incidents trigger a single notification with a list of affected pages.
  • Maintenance windows are automatically muted and restored after completion.
  • A recovery notification is sent upon restoration.

Without noise reduction, the team will mute group messages within a week.

Fifth: On-Call and Escalation

  • The on-call schedule should specify the first responder, backup, and time slots.
  • Escalation rules: 15 minutes no response → manager; 30 minutes → executive management.
  • Alert content includes location details: URL, status code, response time, and region.
  • Record the start and recovery time for every incident.

Sixth: Review and Cleanup

  • Review total alerts and false-positive rate monthly.
  • Clean up rules that haven't triggered in a long time quarterly.
  • After each P1 incident, convert "human-discovered" issues into "monitorable" ones.

Acceptance: Simulate an Incident

  • Manually stop the service and receive a P1 alert within 5 minutes.
  • Receive a recovery notification after restoration.
  • The alert should show which page and what error occurred.
  • On-call staff can locate logs using the documentation.

Boundaries

Tool configuration ensures "signal reachability," but response quality depends on processes and people. Without an on-call system, even the best tools are just notifications in the status bar.

Related reading: