Why Unknown Answers and High Scores Need Careful Interpretation
An unknown answer is not a zero, a band is not a ranking, and a high average can sit on top of a practice that needs review. This is what each kind of profile lets you say.
Written by CSV/CSA Maturity Editorial TeamPublished 9 min read

On this page9 sections
An unknown answer is not a zero, a maturity band is not a ranking, and a high average does not settle every practice beneath it. In the CSV/CSA Maturity Assessment, "I do not know yet" keeps a question out of the score instead of scoring it as the weakest practice. The four bands are fixed ranges on a rubric with no population behind them. Three questions raise a priority review prompt whatever the total says.
These rules belong to model version csa-maturity-2.0.0, which scores self-reported answers about computer system validation (CSV, which here does not mean comma-separated values) and computer software assurance (CSA) for one team, site or system portfolio. The examples below are synthetic samples computed with the published model. None of them is a customer result or an industry average, and we publish neither.
Missing, unknown and numeric zero are three states
A question can add nothing to a score in three ways. They look alike in a spreadsheet and mean different things.
| State | What the respondent did | How the model treats it | Sensible next step |
|---|---|---|---|
| Numeric 0 | Selected the lowest practice statement, for example "We cannot provide a dependable list of the systems in scope." | Scored as zero and counted in numeric coverage. On Q02, Q05 or Q15 it raises a priority review prompt | Confirm the answer with the owner, then plan a first practice step |
| "I do not know yet" | Declined to choose because no statement could be supported | Not scored. Counted as addressed and left out of numeric coverage. The profile becomes partial | Ask the owner for the evidence named in the question |
| Missing | Gave no response | The run stays incomplete, with no overall score or band | Choose a statement or select "I do not know yet" |
Why the model does not turn unknown into a number
Two shortcuts are tempting when an answer is unknown, and both put a claim in the respondent's mouth. Take the methodology's partial example, a synthetic sample in which every answer is 2 except Q05 to Q07, which are unknown.
| Treatment of the three unknown answers | Result | What the result would assert |
|---|---|---|
| Score them as 0 | 57.1 | That the lowest statement is true for all three risk questions |
| Drop the risk area and average the other six | 66.7 | That the risk area looks like the rest of the scope |
| What the model reports | No overall score. Possible range 57.1 to 71.5. Numeric coverage 83.3% | Only what the 15 numeric answers support |
The first two rows are numbers the model refuses to print. Scoring unknown as zero punishes a respondent for being honest and teaches the next one to guess. Dropping the area rewards leaving difficult questions open. The third row is less tidy and it is the only one that is true. You can see how the assessment explains unknown answers before you start.
An unknown answer is often the accurate one
Not all uncertainty is a warning sign. A validation lead may not know how supplier releases are assessed (Q12) because another team owns that work. That is a question of who to ask, not a failed control. The tool responds with an evidence-gathering action for the question. For Q02, Q05 and Q15 it shows a verification prompt and places the check ahead of improvement steps, because other answers lean on those three practices.
Numeric coverage describes the score, not the evidence
Numeric coverage is the share of the 18 questions with a numeric answer. Fifteen numeric answers give 83.3%. It is a different figure from progress through the questionnaire, where an explicit unknown counts as addressed.
Coverage is easy to over-read in both directions:
- It is not confidence. Coverage of 83.3% does not mean the result is 83.3% reliable. It means three questions are not scored.
- It is not verification. Coverage of 100.0% means every question has a number. It says nothing about whether anyone opened a record before choosing it.
A partial profile also shows a possible range, the interval the overall score must lie in once every unknown is replaced by a value. It is built by putting 0 and then 3 in place of each unknown answer. The range is arithmetic. It has no probability attached and it does not predict where the score will land.
An average hides variation
The overall score is a mean of seven area scores, and each area score is a mean of two or three answers. Means are good at summarizing and bad at showing what they summarize. Two synthetic profiles make the point.
Illustrative example: Profile A answers 2 to every question. Profile B answers 3 to both questions in system inventory, intended use and approvals, and gives 0, 1, 1 in risk-based assurance, 1, 1, 2 in traceability, 1, 2, 2 in change management and 1, 1, 2 in evidence and review readiness.
| Practice area | Profile A | Profile B |
|---|---|---|
| System inventory | 66.7 | 100.0 |
| Intended use | 66.7 | 100.0 |
| Risk-based assurance | 66.7 | 22.2 |
| Traceability | 66.7 | 44.4 |
| Change management | 66.7 | 55.6 |
| Approvals and record controls | 66.7 | 100.0 |
| Evidence and review readiness | 66.7 | 44.4 |
| Overall | 66.7, Defined | 66.7, Defined |
| Priority review prompts | None | Q05 |
Both profiles show 66.7, Defined. One describes even, documented practice. The other describes three areas at the top of the rubric, a risk area in the Ad hoc band and a 0 on the question about function-level risk reasoning. Anyone who reads only the headline number cannot tell them apart.
The same thing happens inside an area. In the methodology's 92.9 example, every answer is 3 except Q15 at 0. The approvals area shows 50.0, Defined, although one of its two practices sits at the lowest statement, and the overall result is 92.9, Measured. The worked example with every step shows the arithmetic.
This is why the profile lists seven area scores before anything else, and why three questions carry prompts that sit outside the arithmetic. A prompt does not lower the score. It keeps one answer visible to a reader who looks only at the total.
Rubric bands are not market percentiles
The four bands are fixed ranges on a 0 to 100 scale: Ad hoc below 25.0, Developing from 25.0, Defined from 50.0 and Measured from 75.0. They were set when the rubric was written. They do not move with anyone's data, and a score is computed from one respondent's answers alone.
A percentile is a statement about a population, such as "higher than most comparable scopes". Publishing one would need all of the following:
- a defined population and a sample that represents it
- the same model version and comparable scope definitions for every respondent
- some control over how answers were chosen, since self-report varies by respondent
- permission to use respondents' answers for that purpose
None of this work has been done for the assessment. It publishes no average, percentile or ranking, and a band cannot be read as "typical", "behind" or "ahead". Defined means that the mean of the seven area scores, displayed to one decimal place, is at least 50.0 and below 75.0 on this rubric. Measured is the name of the top band. It does not mean that evidence was inspected or that controls are effective.
Any future benchmark would be a separate piece of work, with its own published research design, a suitable sample and a privacy and permission review, before a single comparative number appeared.
Respondents, scope and versions move the number
When two runs give different results, practice may have changed. Four other things may have changed instead.
| What changed between runs | Illustrative example | How to describe the difference |
|---|---|---|
| Practice | A pre-implementation impact review was introduced and is applied | A change in practice, if evidence supports the new answer |
| Evidence | Someone read the records and an assumed 2 became a 1 | A more accurate description. Practice may be unchanged |
| Respondent | A system owner answered the first run and a quality lead the second | A difference of perspective to reconcile |
| Scope | The first run covered one team and the second the whole site | Two different profiles |
| Model version | Question wording, weights or bands changed | A change of methodology |
Only the first row is improvement or decline. A lower score after an evidence check is the assessment working as intended.
Five profiles and what each one lets you say
Synthetic samples. The first four are test profiles of the published model, so each number can be reproduced.
| Profile | Synthetic sample | What you can say | What you cannot say |
|---|---|---|---|
| Complete, low | Every answer 1: overall 33.3, Developing, coverage 100.0% | For this scope, we selected statements describing informal or inconsistent practice in every area, and we know where to start | That the organization is non-compliant, that an inspection would go badly, or that the scope is below an average |
| Complete, high | Every answer 3: overall 100.0, Measured, coverage 100.0% | For this scope, we selected statements describing reviewed and maintained practice everywhere. The next step is to sample evidence | That controls are effective, that the scope is compliant or inspection-ready, or that nothing needs checking |
| High with a priority flag | Every answer 3 except Q15 at 0: overall 92.9, Measured, with a priority review prompt on Q15 | Most selected statements are at the highest level, and one practice needs review whatever the total: how assurance records are protected | That 92.9 offsets the Q15 answer, or that the prompt is a finding |
| Partial | Every answer 2 except Q05 to Q07 unknown: 15 of 18 numeric, coverage 83.3%, possible range 57.1 to 71.5 | Fifteen answers are scored, the three risk answers are unknown, and the overall result lies between 57.1 and 71.5 | That the score is 57.1, 66.7 or the midpoint, that the band is Defined, or that the range is a confidence interval |
| Incompatible versions | A profile from csa-maturity-2.0.0 set beside one from a different model version or question bank |
These are two profiles under two methods, and the gap between them is a change of methodology | That practice improved or declined by the difference in points |
Version 2.0.0 is the only version published so far, so the last row will matter when a later version appears. The same reasoning applies today to two profiles with different scopes.
What a useful reassessment needs
A second run is worth comparing with the first only if you can say what changed and why. Keep a short record with each run. The assessment does not store your evidence, so this belongs in your own files.
| Record | Why it matters |
|---|---|
| The scope statement, word for word | A profile describes one scope. A different scope is a different profile |
| Model version and question bank version | Scores under different versions are not comparable |
| Who answered, by role | A change of respondent can move answers with no change in practice |
| For each changed answer: old value, new value and the evidence seen | Separates a change in practice from a change of opinion |
| How each former unknown was resolved | An unknown should become a number because someone checked, not because a number was wanted |
| The date of each run | Shows whether changes had time to be applied and sampled |
Two habits keep the comparison honest. Answer each question again against its four statements instead of adding a point because an action was completed. And leave an answer unknown if the evidence is still out of reach. The assessment does not prescribe how often to repeat a run. Set the interval through your own quality system, and use the change log in the methodology to confirm that the version has not changed in between.
What this article does not establish
- It contains no data about how organizations score, so it cannot tell you what is typical. The model was not built from such data, and none is published.
- The profiles are synthetic and were chosen to show the arithmetic. They are not targets.
- It makes no regulatory claim. A profile is not a determination of compliance, and a band is not a regulatory status.
- The equal weighting of areas and the equal spacing of the answer values are conventions of the model. Other reasonable conventions would give different numbers.
- A prompt or a low answer is a reason to look at evidence. Neither is a finding.
Related reading
- The 54.8 profile, calculated step by step reproduces the complete, partial and 92.9 profiles by hand.
- The assessment methodology gives the rules for unknown answers, bands, prompts and versions in full.
- Self-assessment vs quality audit compares what a self-reported profile and an evidence review can each support.
This article describes assessment model csa-maturity-2.0.0.