Insights:How to Configure Fault Emergency and Troubleshooting Tools: From Alerts to On-Call
Three-Layer Configuration
| Layer | Tool | Key Points |
|---|---|---|
| Data Collection | Site availability, resources, business metrics | Cover critical paths |
| Judgment | Thresholds and grading | Avoid noise |
| Notification | Group chat, SMS, phone | Graded escalation |
Data Collection Layer
Four required items:
- Site availability (HTTP status).
- Response time.
- Server resources (disk, memory).
- Business metrics (successful form submissions).
Judgment Layer
| Level | Trigger Condition | Notification Method |
|---|---|---|
| High | Site unreachable for 2 minutes | Phone/SMS |
| Medium | Error rate exceeds threshold | Group message |
| Low | Disk usage above 80% | Daily summary report |
Start with wide thresholds, tighten after two weeks of observation.
Notification Layer
- High-level alerts notify the primary responder and backup simultaneously.
- Medium and low alerts go to group chat only, to avoid disturbance.
- Alert content must include: what happened, when, scope of impact, recommended actions.
Shift Handover
- Handover log records unresolved alerts.
- Confirm that alert recipient lists are up to date during handover.
- Fixed shift change times to avoid gaps in coverage.
Minimum Viable Configuration
- Site reachability alerts.
- Backup failure notifications.
- Group notification + one on-call responder.
These three measures cover the most critical scenarios.
Common Issues
- Alerts sent only to individuals fail when they're on leave.
- Overly sensitive thresholds trigger dozens of notifications per day.
- Notifications lack actionable guidance, leaving recipients unsure what to do.
Boundaries
Tool configuration addresses "detection and notification," not mitigation or repair capabilities. Tool selection must be tailored to your existing environment; this article does not recommend specific products.
Related Reading: