Insights:What to Check First When Monitoring and Alerting Breaks: 5-Step Troubleshooting
Step 1: Verify Whether the Alert Is Real
- Manually visit the target page using a browser or from another network.
- Use command-line tools from a different machine to check status codes and response time.
- Compare against the detection timestamp to see if it overlaps with a maintenance window.
Conclusion branches: Cannot access manually → real incident, proceed to troubleshooting; Access works manually → likely false positive, go to Step 2.
Step 2: Inspect the Detection Source
- Is the detection node online? Has its quota been exhausted?
- Is the detection region on the same network path as the target site? (Detectors co-located in the same data center often produce false positives.)
- Is the detection frequency too high, causing the target firewall to throttle it?
- Have the detection rules been modified (e.g., URL changed, keywords swapped)?
The most common cause of false alerts is an issue on the detector side itself.
Step 3: Review Alert Rules
- Has the consecutive failure threshold been set to 1?
- Is content validation overly strict (e.g., markers changed after a page redesign)?
- Does certificate checking treat "warnings" as "critical" triggers?
- Are alerts of different severity levels routed to the same channel?
Step 4: Check Channels & Muting
- Test whether alert messages are delivered normally.
- Has the alert channel been restricted or blocked by group bots?
- Was the maintenance window forgotten for muting, or was it not unmuted afterward?
- Were any alerts manually dismissed into permanent muting?
Step 5: Review Monitoring Coverage
- Are newly launched pages added to monitoring?
- Do new dependencies (caches, search services) have monitoring in place?
- After old pages went offline, were their monitoring items cleaned up?
- Are there duplicate rules monitoring the same URL?
Logical Order of These Five Steps
Work inward from external symptoms: first confirm the fault's reality, then inspect the signal source, then review rules, then check channels, and finally verify coverage. Checking in reverse order often leads you in circles through rule configurations.
Common Conclusions & Responses
| Phenomenon | Likely Cause | Resolution |
|---|---|---|
| Multiple alerts at the same time | Probe-side network failure | Switch to a different probe point for verification |
| Repeated alerts for a single page | Content validation too strict | Relax keyword matching |
| Alerts are not being received | Notification channel failure | Send test messages to verify delivery |
| New pages have no monitoring | Monitoring coverage not updated | Add pages to the monitoring checklist |
| Alerts are consistently ignored | High false-positive rate | Redesign alert rules |
Scope
The five-step troubleshooting above targets issues within the monitoring system itself. If a real incident is confirmed, follow the standard incident-handling process: restore service first, then fix the monitoring.
Related Reading: