How to distinguish files discovered from files actually reviewed

A file is discovered when the assessment knows it exists; it is reviewed only when a named method has examined it for a named security question and recorded the result. Between those points, a scanner may acquire the content, index it for search, retrieve a snippet or supply that snippet to an analyst or model. None of those intermediate steps proves that the file's relevant behavior was assessed. Report each stage, its denominator and exceptions. The answer changes with the claim: an authorization assessment of one route may need a handful of files, while a claim about the whole repository needs a complete inventory and a defensible account of all in-scope paths.

For a security leader, this distinction prevents a dashboard from calling a repo “covered” because the tool listed its tree. OWASP's secure code review guidance treats analysis as work on design, logic and code; a file name in an inventory is only a starting point. OWASP ASVS 5.0.0 supplies requirements to evaluate, not a rule that every discovered file counts as verified.

Five stages that answer different questions

Some workflows skip a stage: a human may review a file directly without a search index. Keep actual states rather than forcing every file through a fictional pipeline. Define file identity as repository + revision + path. Identical bytes under two paths can have different callers or configuration, so do not deduplicate coverage solely by content hash. Count unique file identities for each stage and, for a material claim, include ranges and questions: reviewing ten lines of a 2,000-line authorization module does not mean its other exports were assessed. Also track components and requirement obligations separately. File counts measure process scope, not security strength.

Derive the counts from six visible file records

All rows and events below are synthetic. The selected tree is complete and all six files are eligible for this narrow authorization review. “Indexed” refers to searchable content, not metadata-only path listing.

From these rows: 6 discovered and eligible; 5 acquired; 4 content-indexed; 3 unique files supplied; 2 unique files with completed question-specific review receipts. The retry of read.ts adds a retrieval/supply event, not a fourth supplied file or a third reviewed file. The parser failure does not remove legacy.conf from the eligible denominator.

A file can be acquired without a usable index, and a reviewer can later inspect it manually. A file can also be supplied to a model without a completed reasoning record. Merely logging “sent 50 lines” does not establish that the question was resolved or that counterevidence was considered.

Now a reviewer examines read.ts for Q2, restricted response fields, and creates receipt R3. Unique reviewed files remain 2. Completed file/question receipts become 3: read.ts/Q1, policy.ts/Q1 and read.ts/Q2. Neither count is the number of completed application obligations: Q1 may still require worker and deployment evidence. This is why file counts, review receipts and control conclusions need different units.

Rewrite the dashboard claim from the records

The incoming dashboard says “6 files scanned; full authorization coverage.” Replace it with:

At revision S6, six eligible paths were inventoried. Five files were acquired and four content-indexed. Three unique files had ranges supplied; two have completed review receipts for specified questions, totaling three file/question receipts after the response-field pass. The export worker has no completed conclusion, one configuration file failed indexing, and the list route was not acquired. Application authorization coverage remains incomplete.

The most useful next request is not “scan more.” Ask for the worker receipt or explicitly resume the unfinished worker question, and recover the list route if it can alter the collection-authorization decision. The receipt needs its question, exact revision/ranges, method, counterchecks and outcome. A newly created note that only repeats the supplied ranges does not retroactively prove analysis occurred.

For larger inventories the same method can group rows, but every total should be reproducible from stable file identities and completion evidence. Do not infer the number of unknown files from a truncated tree. Here enumeration was complete; in another case even the first denominator might be unknown.

Make the numerator and denominator auditable

Before reporting a ratio, specify its purpose:

1. Inventory completeness: Did listing finish for the selected repositories and immutable revisions? A provider's first page, truncated tree or timed-out subtree must be marked incomplete. The total number of undiscovered files is then unknown.
2. Eligibility: Which paths were excluded, why, and by whom? Avoid relying only on directory names. Infrastructure definitions, migrations, tests and docs can explain a security control.
3. Acquisition and indexing: Which eligible content could not be read or searched? Give reasons such as permissions, size, encoding, symlink policy, parser failure or budget stop.
4. Review: Which exact behavior or requirement was investigated in which component? Link the evidence record and reviewer. One file may support several requirements, and one requirement may need many files plus deployment evidence.
5. Change control: What revision and time do these numbers describe? A new commit or newly connected repo requires a new scope statement or a clearly bounded incremental review.

The owner should retain both a machine-readable manifest and a short plain-language summary. The manifest lets another reviewer reproduce the counts and find skipped paths. The summary lets an engineering manager decide whether to spend time extending the assessment. OWASP's testing reporting guidance supports telling different audiences what was tested and what findings mean; a coverage note should give them the same scope clarity.

Connect reviewed files to ASVS requirements carefully

Suppose the reviewed files include the API route, canReadInvoice, the database query and gateway configuration source. That may be useful evidence for ASVS v5.0.0-8.2.2, which addresses permission to specific data items (versioned catalog). It does not prove the PDF worker enforces the same rule or that the gateway source matches production. The requirement row should therefore identify the reviewed API path, the unreviewed worker path and the missing deployment artifact. “Reviewed” describes the investigation completed within its declared scope; “ASVS requirement satisfied” needs an independently justified conclusion for the declared application boundary.

If the team stops because of time or cost, preserve a stop reason and a resumable list. Do not convert budget-exhausted or unreachable files into benign results. Rank follow-up by the decisions those files could change: a tenant-policy module that failed acquisition matters more to this authorization question than an unrelated generated icon. State the sampling method if only selected paths received deep review. A targeted sample can be valuable; it is not whole-repo coverage.

Reconcile a retry without counting the same work twice

Suppose the failed retrieval of api/list.ts succeeds in a later attempt at the same revision. The acquisition count becomes six. If its content is then indexed, the content-indexed count becomes five, because config/legacy.conf still lacks a usable index. Nothing about those events changes the number of files with completed review receipts. The list route becomes reviewed only when its named question has an outcome record.

Now suppose the analyst receives the list-route range but the analysis stops before finishing. Record supplied content and interrupted work. A retry can resume that question, yet the combined history should still yield one unique file/question result for the completed task, with earlier attempts attached. Counting each attempt as an additional reviewed file would reward instability and obscure what remains unfinished.

These synthetic changes give the leader a useful reconciliation:

If the later attempt uses a different revision, do not combine its rows into the earlier snapshot without a declared comparison method. A new path identity is a new source subject. The old receipt may be reusable after a change analysis, but that analysis needs to be visible. A dashboard that silently merges revisions can show an impressive aggregate while describing no actual release.

Ask whether the missing file can change the answer

The six-file ledger is small enough to show why volume alone is a poor scheduling rule. The list route may determine whether a caller can enumerate another tenant's records, even if the detail route is scoped. The worker may govern export output. The failed configuration parser may or may not affect the authorization question; a bounded manual inspection can establish its role. The web view may explain a user journey without enforcing the server boundary.

Prioritize the next read by that relationship to the security question. Record why a file was selected, which range mattered and what the reviewer could conclude. This gives a future reviewer a defensible sampling rationale. It also lets the leader explain why a small additional review changed the authorization conclusion more than indexing hundreds of unrelated assets would have done.

The question to ask at the next review meeting

Ask for three artifacts: the pinned inventory with completion status, the stage ledger with exclusions and failures, and a requirement-to-evidence map for the risk that matters most. Then choose one gap to close, such as obtaining the worker policy or retrying the failed list-route retrieval. Recalculate only the relevant stage counts and keep the old snapshot for comparison.

For the deeper control decision, use the ASVS evidence mapping guide. For wording that makes the remaining scope clear to a customer, use the partial assessment report template.