Fault Emergency and Troubleshooting: Understand These 6 Concepts First

When emergency procedures don't work well, it's often not due to insufficient content but because concepts are used interchangeably. Once the six concepts are c…

Insights:Fault Emergency and Troubleshooting: Understand These 6 Concepts First

Six Concepts

Concept One-Sentence Definition
Incident Severity Level of seriousness based on scope of impact
Response SLA Maximum time from detection to ownership assignment
Containment Restore availability first; do not pursue root cause yet
Root Cause The underlying cause that triggered the incident
Postmortem Post-incident analysis and improvement
On-Call Who is responsible and when

Incident Severity

Three common tiers:

  • High: Full site outage or core functionality failure.
  • Medium: Partial functionality issue with a workaround available.
  • Low: Styling, copy, or non-critical page issues.

The purpose of grading is to determine who gets woken up.

Response SLA

  • High: Acknowledged within 15 minutes.
  • Medium: Acknowledged within 1 hour.
  • Low: Addressed the same day.

The SLA commits to acknowledgment, not fixing.

Containment

  • Roll back the most recent change.
  • Fail over to a standby environment.
  • Temporarily disable the affected feature.

Containment comes before root-cause analysis—users are waiting.

Root Cause

  • Investigated only after containment is confirmed.
  • Requires logs, change records, and a reconstructed timeline.
  • Conclusions must be translatable into one actionable improvement item.

Postmortem

  • Document the timeline clearly: detection time, containment time, restoration time.
  • Identify which control points failed.
  • Assign improvement items with clear owners.

On-Call

  • Designate the first responder.
  • Define escalation path (escalate after a set time if nobody picks up).
  • Log open items from the previous handover.

Consequences of Mixing Concepts

  • Treating a stop-loss measure as a permanent fix leads to recurring issues.
  • No severity levels — everything is labeled "urgent."
  • Post-mortems only describe symptoms, with no improvement actions.

Scope

This article covers concepts. For detailed process design, see other articles in this series. Severity classification must be defined based on your own business impact and should not be copied verbatim.

Related Reading: