Guide 070 template

How to report scan coverage honestly

Report scan coverage using separate inventory, acquisition, analysis and finding ledgers, with clear denominators and a worked example.

A security scan coverage report should say which revision, components, files or requirements were in scope; what the method actually examined; what failed or was skipped; and which conclusions still need evidence. Give each percentage its own denominator and definition. “No findings” means no findings were reported under the stated method and scope, not that the application is secure. If repository enumeration was incomplete, do not invent a whole-repository percentage from the files the tool happened to receive. A report should help its reader choose the next assessment or fix, not merely celebrate a completed job.

OWASP's ASVS assessment guidance calls for clear scope, requirements checked, exclusions, method and evidence. It also warns that running an automated tool is insufficient to verify many complex requirements. The same distinction matters in a routine repository review: an indexed file, a file parsed by one ruleset and a verified security claim are different accomplishments.

Use four separate ledgers

One “coverage” number usually hides four different questions. Report them separately, even if one is unknown.

Ledger Denominator Numerator or outcomes What it can tell a reader
Inventory Repository material at a pinned revision, if enumeration completed. Listed files or components, with missing branches noted. Whether the source population is known.
Selection and acquisition Explicitly selected files or components. Readable content, blocked content and exclusions by reason. Whether the planned material was available to the method.
Analysis and review Named files, rules or requirement-component obligations selected for a method. Processed, reviewed, partial, unstarted and blocked. What work actually ran and at what depth.
Finding disposition Candidate claims produced within that scope. Supported, unresolved, or rejected with counterevidence; record review depth separately. What may be sent to remediation and what remains uncertain.

The ledgers are related but not interchangeable. A file can be inventoried without being selected. A selected file can be read only as context, without analysis for a particular weakness. A rule can run and find nothing while a business-logic requirement remains unreviewed. A candidate can have exact line references without being a confirmed vulnerability.

Filled template: one bounded repository assessment

Illustrative case, not a customer result or executed assessment: The AppSec lead at Pinewater Apps assessed the billing monorepo at commit E84B1. The decision was whether ordinary account users might obtain invoices from another account through download or export. The team used a named static ruleset on supported TypeScript source and inspected relevant call paths manually. It obtained deployment evidence for the direct-download routes; the export worker mapping remains unknown. No application code, hooks or dependencies were executed as part of this source review.

This is the report's concise top section:

  • Scope and decision: Pinewater billing monorepo, commit E84B1; invoice download and export, their shared authorization calls and relevant delivery configuration. The assessment supports a decision on these two flows, not a repository-wide security conclusion.
  • Method: complete tracked-file inventory; bounded source acquisition; static ruleset TS-Security-2026.10; manual review of selected authorization and delivery paths; direct-download mapping obtained and export worker mapping requested.
  • Key result: one legacy direct-download route finding independently verified and approved; one newer route candidate rejected with a cited ownership control; one export candidate remains unresolved pending worker deployment mapping. These are invented editorial figures.
  • Material limit: 30 selected files could not be processed; 90 retrieved files were indexed for context but not analyzed by the named ruleset. No runtime behavior was validated.
  • Next decision: service owner supplies export worker artifact and delivery policy; AppSec lead completes the partial obligations and schedules the deferred components.

Then give the counts with definitions:

Measure Pinewater value Honest interpretation
Tracked files inventoried at E84B1 800 Complete provider listing for this revision; no claim about external services or omitted repositories.
Explicitly excluded from this assessment 200 120 generated/vendor files and 80 archived samples, each recorded by path and reason.
Candidate files after exclusions 600 800 − 200; a planning denominator, not a count of reviewed files.
Selected for this pass 420 180 candidate files deferred with owners and a later-pass reason.
Selected content retrieved 390 of 420 30 blocked: 20 exceeded the declared size limit and 10 had unsupported encoding.
Analyzed under named static ruleset 300 of 420 90 retrieved files served as context only; 30 were blocked. This says nothing about rules not enabled or patterns they cannot detect.
Requirement-component review obligations 14 reviewed, 6 partial, 4 unstarted of 24 selected A reviewed obligation has recorded evidence and a conclusion for the declared method; it is not a guarantee of compliance or absence of flaws.
Candidate claim dispositions 1 verified/approved, 1 rejected, 1 unresolved The unresolved claim remains investigation, not a confirmed issue or a clean pass.

The useful file ratio is 300/420 selected files analyzed by the named ruleset, not “71% of the application is secure.” The useful obligation ratio is 14/24 selected review obligations completed to their declared method, with six partial and four unstarted. Completed review is not the same as a passing control: retain each obligation's supported, failed or non-applicable conclusion beside its completion state. It should not be called “58% ASVS compliant.” If an assessment adopts ASVS requirements, state the exact standard version, selected level or requirements and component mapping; OWASP's ASVS guidance expects the report to identify what was checked and what was excluded.

Reconcile raw receipts before writing the summary

Pinewater's 800-file inventory is a population record. The analyzer and manual reviewer produce different work receipts. A small excerpt illustrates the difference; these records are synthetic and were not produced by a tool here:

path                   selected  acquired  static outcome    manual outcome
api/download.ts        yes       yes       analyzed          reviewed: owner check
shared/auth.ts         yes       yes       analyzed          reviewed: current grant
workers/export.rb      yes       yes       context-only      partial: delivery missing
generated/client.ts    no        no        excluded          not selected
legacy/large.ts        yes       no        size blocked      blocked

api/download.ts counts once in the selected-file denominator, once in the static-analyzed numerator and once in the relevant manual work record. Do not add those two methods and call it two files covered. Conversely, workers/export.rb can have useful manual review even though the named TypeScript ruleset did not analyze it. “Not statically analyzed” is not synonymous with “nobody read it.”

The requirement ledger has another unit. One authorization obligation may span api/download.ts, shared/auth.ts and delivery configuration. The obligation remains partial if a required downstream check is missing, even though all visible files parsed successfully. Two requirements can also use the same file. That is why file coverage cannot be converted into requirement coverage by multiplication.

For the larger synthetic counts above, calculate each stage directly:

inventory: 200 excluded + 180 deferred + 420 selected = 800
acquisition: 390 retrieved + 30 blocked = 420 selected
static analysis: 300 analyzed + 90 context-only + 30 blocked = 420
review obligations: 14 reviewed + 6 partial + 4 unstarted = 24

The analysis receipt should explain the 90 context-only files and 30 blocked ones individually, not merely repeat the totals. If ten unsupported-encoding files later become readable, that changes acquisition first. They do not enter the analyzed numerator until the relevant method actually processes them.

Write different summaries from the same facts

The AppSec lead needs the actionable remainder:

At E84B1, the named ruleset analyzed 300 of 420 selected files. Thirty selected files were unavailable and 90 were context-only for that method. Fourteen of 24 selected review obligations are complete; export delivery is among six partial obligations. Recover blocked content and obtain the worker deployment mapping before expanding the export conclusion.

A customer-facing assessment summary needs the supported scope without internal progress arithmetic pretending to be assurance:

The assessment examined the specified invoice download and export source paths at E84B1. Direct-download review reached a bounded conclusion; export applicability remains limited by missing worker deployment evidence. No runtime behavior was validated. The detailed report records method, exclusions and unresolved work.

Neither summary says “71% secure.” They are different projections of the same receipts, not different versions of the truth. A stakeholder who asks for repository-wide assurance should see the deferred 180 and excluded 200, plus the external services that were never part of the file population.

If enumeration stops halfway, replace any whole-repository percentage with a bounded statement about the named subtree whose inventory is complete. A lower bound on files seen is useful; an invented denominator is not. OWASP's reporting guidance supplies the scope-and-limit discipline, while the arithmetic and sample receipt here are original illustrations.

Audit the denominators before sharing

Pin the material. Check that the commit, inventory and analysis run refer to the same repository state. If a tool analyzed an archive while the report lists a later branch tip, correct the report or rerun. If the production artifact cannot be mapped to the source, say that deployment applicability is unverified.

Name the unit. “24 of 30” is meaningless without saying whether the unit is files, routes, rules or requirement-component obligations. A requirement reviewed for one API does not automatically cover another API. Do not add unrelated percentages or average them into one “security score.”

Check the transitions. For Pinewater, 200 excluded + 180 deferred + 420 selected = 800 inventoried. Within selected material, 300 rule-analyzed + 90 context-only + 30 blocked = 420. The two ledgers reconcile, but neither proves that the rules covered all weaknesses. Include reason counts so the next owner knows what to recover.

Separate claim status from priority. A verified issue can be low business priority; an unresolved possible cross-account disclosure may require urgent investigation. Show evidence status, possible impact, practical exploitation difficulty and business priority as distinct fields. A source citation establishes where code exists; the path and counterevidence support the technical conclusion. Record whether a separate reviewer challenged that conclusion, without making review status a substitute for evidence. The Web Security Testing Guide's reporting chapter emphasizes reports usable by technical and executive readers; mixing these concepts makes both audiences less able to act.

State the check on your conclusion. The AppSec lead should have another reviewer locate the cited revision and verify the arithmetic, exclusions and reported dispositions. A service owner should confirm deployment facts and planned follow-ups. The report is finished when the supported conclusion and its limits can be reproduced from the ledger, not when every progress bar reaches 100%.

If inventory or analysis failed

Suppose the provider stopped listing files halfway through the monorepo. You cannot report “300 of 420 selected files analyzed” as a repository-level ratio if the 420 was drawn from an incomplete list. State the observed lower bound, the listing failure and the unknown remainder. Resume enumeration or narrow the report to a named subtree whose inventory is demonstrably complete. If a scanner times out after producing two findings, retain those evidence-backed results but label remaining work partial; do not replace the report with “clean” or count the unfinished obligations as reviewed.

For the next assessment, use the first-scope decision record before the run, then fill this form from its actual receipts. When a candidate finding appears, apply the finding-triage gate before it enters a developer worklist.