What the requirement actually says · 1.4
Self-assessment, and why the scores drifted
Self-assessment, and why the scores drifted. What the requirement says, what it means in practice, and what an assessor will ask.
For a related human-factors concept, self-reporting bias provides additional context for interpreting behaviour without jumping to conclusions.
For an independent reference point, see NIST SP 800-171 Rev. 3.
What it is
Scoring yourself against the standard
A self-assessment establishes, requirement by requirement, whether each is implemented. A published methodology converts that into a numerical score by deducting points for unmet requirements, weighted by their significance.
The score is submitted into a government system and is visible to contracting officers and, in effect, to primes deciding who to work with.
The scoring methodology published by the Department of Defense for assessing implementation of the NIST publication, and the acquisition regulation clause requiring submission of the resulting score.
Both are published; the methodology states how points are deducted and what the maximum and minimum scores are.
How it goes wrong
Three ways a score becomes optimistic
Reading a requirement as met because it is intended. The policy says it happens; nobody checked whether it does.
Reading it as met without evidence. The control operates and there is no record. For a self-assessment score this may pass; for an assessment it does not, and the gap between the two is where most surprises live.
Partial implementation counted as full. A control applied to some systems in scope and not others.
None of these is dishonesty. All of them produce a score that an assessment will not reproduce.
Doing one properly
Four rules
Assess against the boundary you documented, not against the business generally. Require evidence for every requirement marked as met, and record where it is. Have somebody other than the implementer review the conclusions. And write down, for each unmet requirement, what would be needed.
The last turns the assessment into the remediation plan at no extra cost.
The plan of action
Unmet requirements need a documented route
Where requirements are not yet met, a plan of action recording what will be done and by when is expected, and in some circumstances is what permits progress despite gaps.
A plan with no dates, or with dates that have passed, is worse than none: it demonstrates that the gap was known and not addressed.
Affirmation
A statement somebody signs
Where affirmation of continued compliance is required, an individual attests to it. That is a personal act with a name attached, and it should follow a check rather than a diary reminder.
The blog entry on staying compliant covers the check that should precede it.
The drift
Why scores and assessments disagree
Reported self-assessment scores have, on the whole, been more optimistic than what assessments subsequently found. The three causes above account for most of it.
The useful response is to assess yourself as though somebody else were doing it: require the evidence, have a second person review, and resist the reading of a requirement that happens to be convenient.
Before submitting
Two questions
Could I show a stranger the evidence for every requirement I have marked as met, today, without preparing anything? And does the score I am about to submit match what I would tell my own board?
If either answer is no, the score is not ready, and the entry on reporting in the blog explains why the second matters.
Frequency
Annually, and after significant change
A score reflects a moment. A system migration, an acquisition or a new production line changes the picture, and a score submitted before such a change describes something that no longer exists.
Keeping the working
Not just the number
Retain the requirement-by-requirement assessment, the evidence references and the reviewer's notes. The score is the output; the working is what allows next year's assessment to start from somewhere and what answers a question about how a conclusion was reached.
Who signs it off
Not the person who did the work alone
A review by somebody else before submission catches the convenient readings. In a small supplier this can be the sponsor rather than a specialist: the value is a second person asking what the evidence is, which does not require expertise.
The conversation with a prime
They can see your score
Primes look at submitted scores when deciding who to work with, and a low score with a credible plan is a better position than a high score that cannot be substantiated.
Being ready to explain the number, including what is not yet done and by when, is worth preparing before somebody asks.
Tooling
A spreadsheet is adequate
Requirement, status, evidence location, reviewer, date. Products exist and are useful at scale; below a certain size they add a licence and a dependency to something a maintained sheet does perfectly well.
What matters is that the evidence references are real and that somebody keeps them current.
Timing the submission
Submit when the evidence exists, not when the deadline arrives
A score submitted the week it is due, assembled in a hurry, tends to be the optimistic kind. One submitted after a proper assessment with evidence references is defensible when somebody asks.
Work back from the date and start early enough that the review is not the thing that gets dropped.
Also
Elsewhere in what the requirement actually says
- What the three levels are, and which appliesThe level is set by your contract, not by your size. Most suppliers handling controlled unclassified information are at the second.
- Reading the clause that binds youReading the clause that binds you. What the requirement says, what it means in practice, and what an assessor will ask.
- Which systems are in scopeWhich systems are in scope. What the requirement says, what it means in practice, and what an assessor will ask.
- Who assesses you, and how each kind worksWho assesses you, and how each kind works. What the requirement says, what it means in practice, and what an assessor will ask.
- What the standard does not requireWhat the standard does not require. What the requirement says, what it means in practice, and what an assessor will ask.
- Machines older than the requirementsA control written for a server, applied to a lathe with a shared login, a decade-old operating system and a USB port.
- Drawings, job packets and paperDrawings, job packets and paper. What the requirement says, what it means in practice, and what an assessor will ask.
- Removable media, and why bans failRemovable media, and why bans fail. What the requirement says, what it means in practice, and what an assessor will ask.
- Vendor remote accessVendor remote access. What the requirement says, what it means in practice, and what an assessor will ask.
- Technical data packages and what they containTechnical data packages and what they contain. What the requirement says, what it means in practice, and what an assessor will ask.
- Segmentation, and what it does not solveSegmentation, and what it does not solve. What the requirement says, what it means in practice, and what an assessor will ask.
- What insider risk actually coversWhat insider risk actually covers. What the requirement says, what it means in practice, and what an assessor will ask.
- What detection can and cannot establishWhat detection can and cannot establish. What the requirement says, what it means in practice, and what an assessor will ask.
- Monitoring, and where the law constrains itMonitoring, and where the law constrains it. What the requirement says, what it means in practice, and what an assessor will ask.
- The fortnight around a departureThe fortnight around a departure. What the requirement says, what it means in practice, and what an assessor will ask.
- Privileged access, and who reviews itPrivileged access, and who reviews it. What the requirement says, what it means in practice, and what an assessor will ask.
- False positives, and what they costFalse positives, and what they cost. What the requirement says, what it means in practice, and what an assessor will ask.
- The documents an assessor reads firstThe documents an assessor reads first. What the requirement says, what it means in practice, and what an assessor will ask.
- What counts as evidenceWhat counts as evidence. What the requirement says, what it means in practice, and what an assessor will ask.
- The findings that recurThe findings that recur. What the requirement says, what it means in practice, and what an assessor will ask.
- Plans of action, and their limitsPlans of action, and their limits. What the requirement says, what it means in practice, and what an assessor will ask.
- What the assessment week is likeWhat the assessment week is like. What the requirement says, what it means in practice, and what an assessor will ask.
- The certificate, and what it coversThe certificate, and what it covers. What the requirement says, what it means in practice, and what an assessor will ask.