Insights:Log and Data Analysis Troubleshooting: From Symptoms to Root Cause
Four Types of Incidents
| Type | Phenomenon | Entry Point |
|---|---|---|
| Metric anomaly | Sudden spikes or drops in values | Compare with same period and check data completeness |
| Missing data | No records for a given day | Collection jobs and disk usage |
| Inconsistent definitions | Two different numbers for the same thing | Filter conditions and timezone settings |
| Slow queries | Analysis takes too long | Data volume and scan range |
Metric Anomaly
- First, check whether the time口径 (definition) is consistent.
- Then, verify whether collection was interrupted (look for gaps in log files).
- Finally, check whether a release or redesign caused behavioral changes.
The three causes should be排除ed in order; the first two account for most cases.
Missing Data
- Check whether the collection job failed (check for alerts).
- Check whether the disk is full, causing write failures.
- Check whether the rotation config accidentally deleted that day's files.
Inconsistent Definitions
- One statistic includes crawler traffic, the other does not.
- One is counted by session, the other by request.
- Different timezones cause跨-day discrepancies.
Documenting unified口径 is the only way to resolve这类 issues.
Slow Queries
- Data volume has grown, but queries still perform全量 scans.
- No time-based partitioning is in place.
- Analysis scripts repeatedly compute the same dataset.
Approach: add time filters first, then consider预-aggregation.
Troubleshooting Order
- Confirm whether the data exists.
- Confirm whether口径 are consistent.
- Confirm whether collection is complete.
- Lastly, suspect the code logic.
Most people start at step 4 and end up taking a longer route.
Scope & Boundaries
This guide covers small-to-medium website scenarios. Log consistency in distributed systems involves distributed tracing and requires a separate design.
Related Reading: