Security Incident Response Playbook: Severity, Containment, and MTTD/MTTR Math
It is 02:47 on a Tuesday. Your phone lights up: failed logins from an unfamiliar range, a database connection pool exhausted, a checkout error rate climbing past anything normal. Only the person holding the on-call phone is awake, with ninety seconds to decide: false alarm, bad deploy, or a costly incident.
Incident response means detecting a security event, confirming it is real, stopping its spread, removing the attacker, restoring service, and learning from the episode. NIST SP 800-61 Revision 3 (April 2025) frames it as part of wider cybersecurity risk management under the NIST Cybersecurity Framework 2.0. SANS gives the six-step PICERL sequence.
Who this playbook is for, and who should not run their own response
This playbook is for a founder, CTO, or operations lead at a company of roughly five to two hundred people, running production systems.
It is not for everyone. A solo founder needs a backup restore and a password reset, not a twelve-page plan. A business in a regulated sector — payment card data, health records, financial services — must satisfy auditors and regulators; this playbook is a starting point, not a compliance defence.
Founder & CEO Insight — Enow A. Jovial
The two frameworks, and which one to run
NIST SP 800-61 Rev. 3, titled Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, was published in April 2025 and superseded Rev. 2 (August 2012), withdrawn on April 3, 2025. Rev. 3 maps the old four-phase lifecycle onto the six functions of the Cybersecurity Framework 2.0: Govern, Identify, Protect, Detect, Respond, and Recover.
SANS PICERL keeps the operational sequence practitioners actually execute. NIST covers governance; SANS covers what your analyst does at 03:00 and in which order. Run PICERL as the sequence and use NIST Rev. 3 as the governance wrapper.
Why slow response is a balance-sheet event
PagerDuty puts it plainly: incident response protects data, reputation, and revenue. Its page cites the Home Depot breach of 2014: almost $200 million in breach-related pre-tax losses.
Take a business earning $120,000 in monthly recurring revenue; a month has about 730 hours, so the revenue at risk per hour of full outage is:
A six-hour outage puts about $986 of gross revenue at risk. If your contract promises 99.9 percent uptime, the monthly error budget is 43.2 minutes — 43,200 minutes times 0.1 percent. A six-hour outage is 360 minutes, 8.3 times the month's allowance, the point at which credits, renewals, and churn start. Add three engineers at $90 per hour for six hours: $1,620.
Every term shrinks when detection and containment get faster — price your own outage with the Break-Even Analysis Calculator.
The two clocks: MTTD and MTTR
Mean time to detect (MTTD) is the average time between an incident starting and someone noticing. Mean time to respond and recover (MTTR) is the average time between detection and full restoration. NIST Rev. 3 puts detection under Detect and recovery under Recover.
Worked example, from stated inputs. An alert fires 22 minutes after a misconfigured credential starts being abused. The on-call engineer acknowledges it 8 minutes later, isolates the affected service 47 minutes after that, and restores verified-clean service 96 minutes after containment: MTTR is 8 plus 47 plus 96, or 151 minutes. Total time from first abuse to restoration is 173 minutes. Containment is the longest segment: cutting it from 47 to 12 minutes drops the total from 173 to 138.
Severity triage: decide the scale before the incident
Triage is the decision everything else depends on. The scale below is this playbook's editorial model: a severity is a trigger, a first response, and a named owner.
| Severity | Trigger | First response | Authority |
|---|---|---|---|
| SEV-1 | Production down, data exposed, or active attacker access | Page commander and all responders | NIST SP 800-61 §2.3 |
| SEV-2 | Service degraded or intrusion, no data exposure | Page on-call engineer and service owner | NIST §2.3; SANS PICERL |
| SEV-3 | Anomaly confirmed, no impact, or known fix, unexploited | Ticket to owning team; 24-hour acknowledgement target | SANS PICERL; PagerDuty |
| SEV-4 | Informational: a scan, a probe, a failed login burst | Log for the weekly review | PagerDuty |
Severity is set by impact, never by who reported it. Anyone may raise a severity; only the incident commander may lower one, with a written reason in the log.
The operational sequence, phase by phase
- Prepare. Write the plan, name the team, set up logging and detection, keep offline backups, and practise. SANS describes a tiered team: Tier 1 triages, Tier 2 investigates, Tier 3 leads complex cases. Line up counsel, public relations, law enforcement, and a forensic retainer before the crisis.
- Identify. Detection is monitoring noticing an anomaly; identification is the human decision that it is a real incident, with a scope and a severity. Correlate more than one source: endpoint telemetry, network flow, identity logs, threat intelligence. A single alert is a rumour; three agreeing sources are evidence. Assign two responders to a confirmed incident. The correlation runs on concrete artefacts: the alert arrives as a JSON payload over an API webhook, the vulnerability carries a CVE identifier, DNS and TCP flow records show where the host talked, and SQL queries against the AWS log store confirm whether the account touched production.
- Contain. Short-term containment stops the spread: isolate the endpoint, disable the compromised account and revoke its sessions, block the malicious traffic, and capture forensic images before changing anything. Long-term containment patches and monitors. Complete segmentation is more secure but can halt operations; targeted isolation preserves uptime but risks incomplete containment.
- Eradicate. Remove the foothold: eliminate malware, close the exploited vulnerability, remove persistence mechanisms such as backdoor accounts and rogue services, and reimage systems where cleaning cannot guarantee integrity. Without the initial access vector, re-compromise is likely — SANS calls partial eradication one of the most common causes of re-compromise. Define exit criteria first: every host named, every credential rotated.
- Recover. Restore from clean, validated backups, rebuild where restoration is insufficient, and validate restored images before reconnecting to production. CISA's guidance is the same: maintain offline, encrypted backups and test them regularly, because a backup you have never restored is a hope, not a recovery plan.
- Learn. Within two weeks of resolution, run a formal review with everyone who participated, not only the response team. Document the timeline, what worked, what failed, and produce specific action items with owners and deadlines. 'Deploy detection rules for service accounts by next quarter' is an action item; 'improve monitoring' is not.
Roles: one voice, one scribe, one clock
PagerDuty's role model maps onto the NIST roles in section 2.2 of Rev. 3. Six roles cover the work.
The incident commander runs the incident, makes the decisions, and owns communications. The deputy keeps time and routes updates to the commander. The scribe documents everything, then assembles the timeline. The internal liaison updates stakeholders; the customer liaison updates customers. The subject matter experts — usually the service owners — do the technical work.
The incident commander is the only person who speaks for the incident externally, and the only person who may lower the severity. The scribe starts writing at minute zero: a timeline rebuilt from memory is fiction. With five people, one holds two roles; with two, the commander is also the scribe.
On-call rotation and alert-volume staffing
A rotation that burns people out fails at 02:47. One on-call seat, twenty-four hours a day, seven days a week, is 168 hours per week.
Five is the practical minimum for one seat, ten for a primary-plus-secondary pair. With three engineers, you have business-hours cover — say so in your contracts.
Suppose monitoring produces 1,200 alerts per day, 8 percent actionable, each taking 6 minutes to triage:
That is 1.2 full-time equivalents on triage alone. If your actionable rate is 8 percent, the other 92 percent is noise, and the fix is tuning, de-duplication, and correlation — PagerDuty's guidance is explicit that events must be normalised, de-duplicated, and correlated in a single hub — not hiring more people to read noise. Recompute monthly: triage load climbing while incidents stay flat means detection is degrading.
Build, retainer, or managed service: the decision matrix
NIST Rev. 3, section 2.2, recognises three staffing models: on staff, on contract, or available when needed.
| Option | What you get | What it costs you | When it fits | Authority |
|---|---|---|---|---|
| In-house team | Fastest response; deepest system knowledge | Salaries; five people minimum for one 24/7 seat | You have the headcount | NIST SP 800-61 §2.2 |
| Incident response retainer | Senior help; forensic capability | Retainer fee; the firm needs your logs | Small team, low frequency, high impact | NIST SP 800-61 §2.2 |
| Managed detection and response | Monitoring, triage, first response outsourced | Monthly fee; loss of direct control | You cannot staff nights | NIST SP 800-61 §2.2 |
| Do nothing | Nothing | The full incident cost | Never | PagerDuty |
A retainer does not replace your playbook; it assumes one. The firm asks for your logs, asset inventory, and escalation contacts — and if those do not exist, the first hours are spent building them. Buy the retainer after the playbook, not instead of it.
Failure modes: how good playbooks still fail
Most failed responses are failed executions of a sound framework. Partial eradication: service restored before the full scope is confirmed; the attacker returns through a missed persistence mechanism. Evidence destroyed by the response: rebooting or reimaging before capturing forensic images destroys the evidence. Capture first, change second. Alert fatigue: when 92 percent of alerts are noise, responders stop reading them, and MTTD quietly doubles. Counter it with the monthly triage-load calculation: any rule firing fifty times a month without an action is tuned or deleted. The single-person dependency: the playbook lives in one engineer's head — on holiday when the incident lands. Communication failure: the technical response succeeds and the customer response fails. First customer update within 60 minutes of a SEV-1, hourly until resolved.
Cross-border context: where the clock is not yours to set
Breach notification clocks are a legal question, covered in the companion data breach response playbook. Do not let a notification deadline dictate your containment order, or containment delay a notification your counsel says is due. Certification expectations are covered in the ISO 27001 vs SOC 2 comparison.
CISA's #StopRansomware Guide, developed through the Joint Ransomware Task Force established by Congress in the Cyber Incident Reporting for Critical Infrastructure Act of 2022, gives the federal sequence: isolate hosts, hunt to scope the intrusion, report to the FBI, apply eviction countermeasures, then harden the network. Victims should report via IC3 or a Secret Service Field Office, and can request technical assistance from CISA.
In the European Union, data protection law imposes its own notification duties; your playbook should name the person who calls counsel. If you hold payment card data, the same applies — see the PCI DSS scope and SAQ assessment playbook.
The post-mortem: blameless, or it is useless
The post-mortem is where the incident pays for itself, and it only works if it is blameless: punishing the engineer who made the honest mistake at 03:00 teaches everyone else to hide the next one. SANS requires the review within two weeks, with everyone present. Run it with four questions: what happened, minute by minute; what went well; what went badly; and what changes, with an owner and a deadline. Track the action items like any other engineering work.
The build checklist
Done means written down, named, and rehearsed.
- [x] Severity scale written: SEV-1 to SEV-4 triggers and owners.
- [x] Incident commander named: one per slot, with a deputy.
- [x] Pre-authorised containment list: isolate, disable, revoke, block.
- [x] Roles assigned: all six roles, with backups.
- [x] Comms cadence set: first update within 60 minutes of a SEV-1.
- [x] Evidence rule written: capture images before changing anything.
- [x] Backups verified: offline, encrypted, restore-tested quarterly.
- [x] Retainer or provider chosen: contacts documented.
- [x] Clocks instrumented: MTTD and MTTR per incident.
- [x] Triage load calculated: monthly formula run.
- [x] Tabletop exercise run: a SEV-1 rehearsed with the full team.
- [x] Post-mortem template ready: four questions, owners, deadlines.
The decision in advance
An incident response playbook does not prevent incidents. It prevents the second disaster — the slow, confused, improvised response that costs more than the incident itself. NIST gives the governance, SANS the sequence, PagerDuty the roles, CISA the federal playbook. Your part is the thresholds, the names, and the rehearsal. Write it this week, while the phones are quiet. For continuity planning when the outage is not an attack, see the business continuity and disaster recovery playbook.