When something happens · 4.4
The first hour, and who decides
The first hour, and who decides. What it costs, who decides, and what usually goes wrong.
For a related human-factors concept, limbic resonance provides additional context for interpreting behaviour without jumping to conclusions.
For an independent reference point, see FTC data-breach response guide.
The tension
Contain or preserve
Disconnecting a compromised machine stops whatever is happening and destroys the volatile evidence that would explain it. Leaving it running preserves the evidence and allows whatever is happening to continue.
There is no general right answer, which is why the decision needs to belong to somebody in advance rather than being improvised by whoever noticed.
The first decisions
Four, in order
Is this real. Many alerts are not, and the entry on false positives in the reference section covers what that costs.
Who is in charge. Named, present, and empowered to disconnect production equipment, which is a decision with a cost that somebody has to be authorised to accept.
Contain or observe. Taking advice from the forensics contact at this point is what the retainer is for.
Has the reporting clock started. The entry on clocks sets out why this cannot wait until the technical picture is clear.
What not to do
Three actions that destroy the answer
Reimage. The instinct of every capable engineer and the end of the investigation.
Log in and look around. Every action changes the system and overwrites some of what was there.
Delete anything. Including the obviously malicious thing, which is evidence.
Where the machine must be taken out of service immediately for safety or production reasons, do so, and record precisely what was done and when.
Communication
One channel, possibly not the usual one
If the network or the mail system may be compromised, coordinating the response on it tells whoever is there what you know and what you plan.
Agree an out-of-band channel in advance: a phone tree, a separate messaging arrangement, anything that does not depend on the systems in question. This is another item that cannot be arranged during the event.
Writing during an incident
Assume everything will be read later
Notes, messages and emails written in the first hours may be read by a customer, a regulator, an insurer or a court. Speculation about cause and blame written at hour two has a long life.
Record facts, times and actions. Leave the conclusions for the review that has evidence behind it.
The checklist
What the first page should contain
Who to call, in order, with numbers. The contain-or-observe decision tree. The reporting clocks. What must not be done. And a place to write the timeline as it happens, because reconstructing times afterwards is unreliable and the timeline is what everybody will want.
Afterwards, immediately
Preserve before restoring
Whatever the technical resolution, the images and logs required by the clause are taken before systems are rebuilt. This is the point at which the obligation is most likely to be breached, by capable people under pressure to get production running.
Make it the responsibility of the person in charge rather than of the engineer doing the rebuilding.
Also
Elsewhere in when something happens
- What compliance actually costs, itemisedWhat compliance actually costs, itemised. What it costs, who decides, and what usually goes wrong.
- Reading a proposal that quotes a certificateReading a proposal that quotes a certificate. What it costs, who decides, and what usually goes wrong.
- What to build and what to buyWhat to build and what to buy. What it costs, who decides, and what usually goes wrong.
- Making the budget case to somebody who resents itMaking the budget case to somebody who resents it. What it costs, who decides, and what usually goes wrong.
- The costs that arrive after the purchase orderThe costs that arrive after the purchase order. What it costs, who decides, and what usually goes wrong.
- Doing this with almost no moneyDoing this with almost no money. What it costs, who decides, and what usually goes wrong.
- Who owns compliance, and why it cannot be nobodyWho owns compliance, and why it cannot be nobody. What it costs, who decides, and what usually goes wrong.
- Running it as a project rather than as a documentRunning it as a project rather than as a document. What it costs, who decides, and what usually goes wrong.
- Reporting to a board that wants one numberReporting to a board that wants one number. What it costs, who decides, and what usually goes wrong.
- When the date slips, which it willWhen the date slips, which it will. What it costs, who decides, and what usually goes wrong.
- Staying compliant after the assessmentStaying compliant after the assessment. What it costs, who decides, and what usually goes wrong.
- Working to two standards at onceWorking to two standards at once. What it costs, who decides, and what usually goes wrong.
- Training that changes what people doTraining that changes what people do. What it costs, who decides, and what usually goes wrong.
- How a control looks from the machineHow a control looks from the machine. What it costs, who decides, and what usually goes wrong.
- Contractors, temps and the visiting engineerContractors, temps and the visiting engineer. What it costs, who decides, and what usually goes wrong.
- Hiring for a role most suppliers have never filledHiring for a role most suppliers have never filled. What it costs, who decides, and what usually goes wrong.
- Giving people a way to say a control is unworkableGiving people a way to say a control is unworkable. What it costs, who decides, and what usually goes wrong.
- What staff are told about monitoringWhat staff are told about monitoring. What it costs, who decides, and what usually goes wrong.
- What counts as an incidentWhat counts as an incident. What it costs, who decides, and what usually goes wrong.
- Reporting obligations and their clocksReporting obligations and their clocks. What it costs, who decides, and what usually goes wrong.
- Being able to answer afterwardsBeing able to answer afterwards. What it costs, who decides, and what usually goes wrong.
- Telling a customer something happenedTelling a customer something happened. What it costs, who decides, and what usually goes wrong.
- What changes afterwards, and what shouldWhat changes afterwards, and what should. What it costs, who decides, and what usually goes wrong.
- The other frameworks you have also metThe other frameworks you have also met. What it costs, who decides, and what usually goes wrong.
- Export control, named and not advised onExport control, named and not advised on. What it costs, who decides, and what usually goes wrong.
- Your own suppliers, and what to ask themYour own suppliers, and what to ask them. What it costs, who decides, and what usually goes wrong.
- Suppliers outside the United StatesSuppliers outside the United States. What it costs, who decides, and what usually goes wrong.
- Where the data physically sitsWhere the data physically sits. What it costs, who decides, and what usually goes wrong.
- What is changing, and how to tellWhat is changing, and how to tell. What it costs, who decides, and what usually goes wrong.