Insights:What Are Logs and Data Analytics? Explained in One Diagram
Five-Layer Architecture
Visit / Error / Business Events
↓ Collection
Raw Logs
↓ Storage
Queryable Archives
↓ Cleaning
Deduplicated Records
↓ Aggregation
Metrics (PV/UV/Conversion/Error Rate)
↓ Decision
Actions (Modify Page/Fix Config/Add Protection)
Each layer compresses information, leaving only the numbers that can drive action.
Collection Layer
- Access logs: who requested what and when.
- Error logs: where failures occurred and their causes.
- Business events: registration, form submission, checkout, etc.
The key to collection is uniform formatting and parsable fields.
Storage Layer
- Small sites: store directly as files, split by day.
- Medium sites: ingest into databases or log services.
- Key requirement: fast queries by time range.
Cleaning Layer
- Remove crawlers and internal traffic.
- Deduplicate records (retries, replays).
- Unify timestamps and time zones.
Logs left uncleaned will produce systematically inflated metrics.
Aggregation Layer
| Metric | Meaning |
|---|---|
| PV/UV | Traffic volume |
| Bounce & Depth | Content usefulness |
| Conversion rate | Proportion of forms/submissions |
| Error rate | System stability |
| Traffic sources | Channel effectiveness |
Decision Layer
- Metric anomaly → Identify the specific page or time period.
- Pinpoint the cause → Formulate an action item.
- Execute the action → Review metric changes.
Without a judgment layer, the first four layers are all cost.
Common Breakpoints
- No business-event logging; only access records are captured.
- Logs exist but field meanings are unclear.
- Someone reads reports, but no one does comparisons.
- Focus only on traffic, not error rates.
Boundaries
This article covers principles and does not involve specific tool configuration. For selection and commands, see other articles in the same cluster.
Related reading: