Log and Data Analysis Troubleshooting: From Symptoms to Root Cause

When analysis results look off, it's rarely because the logs are wrong — more often the metric definition has shifted or data collection has stopped.

Insights:Log and Data Analysis Troubleshooting: From Symptoms to Root Cause

Four Types of Incidents

Type Phenomenon Entry Point
Metric anomaly Sudden spikes or drops in values Compare with same period and check data completeness
Missing data No records for a given day Collection jobs and disk usage
Inconsistent definitions Two different numbers for the same thing Filter conditions and timezone settings
Slow queries Analysis takes too long Data volume and scan range

Metric Anomaly

  • First, check whether the time口径 (definition) is consistent.
  • Then, verify whether collection was interrupted (look for gaps in log files).
  • Finally, check whether a release or redesign caused behavioral changes.

The three causes should be排除ed in order; the first two account for most cases.

Missing Data

  • Check whether the collection job failed (check for alerts).
  • Check whether the disk is full, causing write failures.
  • Check whether the rotation config accidentally deleted that day's files.

Inconsistent Definitions

  • One statistic includes crawler traffic, the other does not.
  • One is counted by session, the other by request.
  • Different timezones cause跨-day discrepancies.

Documenting unified口径 is the only way to resolve这类 issues.

Slow Queries

  • Data volume has grown, but queries still perform全量 scans.
  • No time-based partitioning is in place.
  • Analysis scripts repeatedly compute the same dataset.

Approach: add time filters first, then consider预-aggregation.

Troubleshooting Order

  1. Confirm whether the data exists.
  2. Confirm whether口径 are consistent.
  3. Confirm whether collection is complete.
  4. Lastly, suspect the code logic.

Most people start at step 4 and end up taking a longer route.

Scope & Boundaries

This guide covers small-to-medium website scenarios. Log consistency in distributed systems involves distributed tracing and requires a separate design.

Related Reading: