When something happens · 4.4

The first hour, and who decides

The first hour, and who decides. What it costs, who decides, and what usually goes wrong.

For an independent reference point, see FTC data-breach response guide.

The tension

Contain or preserve

Disconnecting a compromised machine stops whatever is happening and destroys the volatile evidence that would explain it. Leaving it running preserves the evidence and allows whatever is happening to continue.

There is no general right answer, which is why the decision needs to belong to somebody in advance rather than being improvised by whoever noticed.

The first decisions

Four, in order

Is this real. Many alerts are not, and the entry on false positives in the reference section covers what that costs.

Who is in charge. Named, present, and empowered to disconnect production equipment, which is a decision with a cost that somebody has to be authorised to accept.

Contain or observe. Taking advice from the forensics contact at this point is what the retainer is for.

Has the reporting clock started. The entry on clocks sets out why this cannot wait until the technical picture is clear.

What not to do

Three actions that destroy the answer

Reimage. The instinct of every capable engineer and the end of the investigation.

Log in and look around. Every action changes the system and overwrites some of what was there.

Delete anything. Including the obviously malicious thing, which is evidence.

Where the machine must be taken out of service immediately for safety or production reasons, do so, and record precisely what was done and when.

Communication

One channel, possibly not the usual one

If the network or the mail system may be compromised, coordinating the response on it tells whoever is there what you know and what you plan.

Agree an out-of-band channel in advance: a phone tree, a separate messaging arrangement, anything that does not depend on the systems in question. This is another item that cannot be arranged during the event.

Writing during an incident

Assume everything will be read later

Notes, messages and emails written in the first hours may be read by a customer, a regulator, an insurer or a court. Speculation about cause and blame written at hour two has a long life.

Record facts, times and actions. Leave the conclusions for the review that has evidence behind it.

The checklist

What the first page should contain

Who to call, in order, with numbers. The contain-or-observe decision tree. The reporting clocks. What must not be done. And a place to write the timeline as it happens, because reconstructing times afterwards is unreliable and the timeline is what everybody will want.

Afterwards, immediately

Preserve before restoring

Whatever the technical resolution, the images and logs required by the clause are taken before systems are rebuilt. This is the point at which the obligation is most likely to be breached, by capable people under pressure to get production running.

Make it the responsibility of the person in charge rather than of the engineer doing the rebuilding.

Also

Elsewhere in when something happens