How to distinguish fixed from no longer detected

Fixed is a conclusion about the harmful behavior at a stated revision and scope. No longer detected is a fact about one tool run. The latter can happen because a developer removed the unsafe path, but also because the file moved, the rule changed, a scan failed, the route was excluded or the tool lost the necessary context. To close a finding, map the old actor-to-effect path to the new revision and gather positive evidence that the unsafe behavior is prevented or gone. If you cannot explain why the alert disappeared, record inconclusive, keep an owner and name the missing check.

The distinction matters even when the tool is usually accurate. A detector can be silent for a valid reason without proving the application safe. Conversely, a persistent alert may refer to stale evidence after a successful fix. Treat tool output as one observation in a revision-bound review, not as the disposition authority.

A silent result with two very different explanations

Illustrative case, not a customer finding: Harbor Metrics has a search route that accepts a report name. At OLD_SHA, an independently reviewed finding traces that value from the request into SQL text. The team refactors the reports module. A scheduled scan at NEW_SHA reports zero alerts for this rule.

There are two possible new states:

1. The route now calls a prepared statement, passing the report name as a bound value. All callers of the old query helper have been accounted for. An authorized staging check confirms the legitimate search still works and an adverse input is handled as a value. This supports fixed for the stated scope.
2. The vulnerable query was moved into a shared helper under a directory the new scan excluded. The scheduled job completed successfully but never examined that helper. The unsafe operation may still exist. This is no longer detected, with the security outcome at least inconclusive until the path is reviewed.

The same dashboard count appears in both stories. The difference is the evidence. OWASP's SQL Injection Prevention Cheat Sheet explains why bound parameters separate SQL structure from supplied values; it does not say a missing alert proves that separation occurred in Harbor's code.

Use this small evidence table before changing the ticket state:

Run a four-gate silence check

Gate 1: Compare like with like

Pin the original finding and its old revision. Record the new revision, detector or rule version, configuration, selected paths, exclusions, completion status and any budget or time limit. A run that scanned fewer files or stopped early has not repeated the earlier assessment. Inventory, indexing and actual examination are different events: knowing a file exists does not mean the rule or analyst evaluated it. If the original alert cannot be tied to an immutable revision, repair that evidence first.

The output of this gate is a run comparison, not a fix verdict. A useful line is: “Rule R-17 ran to completion at NEW_SHA over the reports service, but the shared query helper was outside selected scope.” That tells the reviewer exactly what silence can and cannot mean. A less useful line is: “No findings after patch.”

Gate 2: Trace the old behavior into the new source

Follow the original entry, input, relevant transformations, control and effect. Compare commits and inspect renames and copies. Git's diff documentation describes revision comparison and rename detection, but a detected rename only helps locate code; it does not settle security semantics. Search for callers of the old function and any alternate endpoint or job that reaches the same operation. If a file was deleted, ask where the route goes now and whether the old build still serves it.

For Harbor, find the POST /reports/search handler at NEW_SHA, the helper it calls, and the database call beneath that helper. If the new route takes a different path, write that path down. If the route mapping is not available, stop at inconclusive instead of assuming the deleted source was the whole feature.

Gate 3: Name the new blocking fact

What fact makes the old harmful effect impossible under the original prerequisites? In a SQL case, it may be that user-controlled data is passed as a bound parameter in a fixed SQL statement, with account-scoping constraints preserved. In an authorization case, it may be a resource ownership check using a server-derived account identity on every relevant path. “The developer added validation” is too broad: validation might constrain length without preventing the unsafe operation.

Ask the reviewer to state how the conclusion could be disproved. For Harbor, one counterexample would be an alternate query helper that still concatenates the input. Another would be a dynamic sort expression that accepts a user-selected column without an allow-list. Inspect the actual database API call: punctuation returning no unexpected rows is useful behavior evidence, but cannot alone prove parameter binding or exclude every injection variant.

Gate 4: Reconcile source, behavior and deployment

If the finding concerns an executable behavior, consider an authorized check with synthetic data against a known build. Compare one legitimate request and one request that would have exercised the old unsafe path; record the actual response and artifact identity. The check does not need to recreate a dangerous production exploit. It needs to test the security property within agreed boundaries. If runtime validation is unavailable or unsafe, say what source evidence supports and what deployment assumption remains. NIST's SSDF frames verification as part of remedying vulnerabilities; an alert count is not a substitute for assessing the remedy.

The result may be “fixed in source at NEW_SHA, production rollout unverified.” That is more useful than prematurely closing a production exposure. If the running artifact differs from the reviewed commit, assign the release owner the deployment mapping check.

Read the run manifests that explain the silence

Use this tool-neutral synthetic manifest pair. It is not a configuration format for a named scanner:

old: revision OLD_SHA; rule query-flow@7
     selected: services/reports/**, shared/query/**
     completed: yes; alert: reports/search -> build_query

new: revision NEW_SHA; rule query-flow@7
     selected: services/reports/**
     excluded: shared/**
     completed: yes; matching alerts: 0

The rule version did not change and the new run completed. The selected scope did change. Check the scanner's actual semantics too: some exclusions suppress displayed alerts while still allowing dependency analysis, whereas others omit code from analysis. This invented manifest explicitly means the helper was not examined. The refactor moved build_query to shared/query/search, and the current route still forwards the request name to it. The supplied new helper still assembles SQL text before the same driver operation. That is positive evidence of a surviving source path, rather than mere uncertainty caused by silence.

The completed comparison is: no longer detected by the new run; still exists in source at NEW_SHA; production artifact applicability follows the separate deployment record. Assign repair of the query path and correction of the scan scope as separate tasks. Restoring coverage may make the alert return, but it will not fix the code.

Now reverse the situation. The next patch changes the driver call to fixed SQL with a separate bound name and preserves trusted account scoping. The source review supports repair at FIX_SHA. A dashboard still shows the original alert, but its run metadata names OLD_SHA. That persistent result cannot refute FIX_SHA's source correction. Obtain a current run if needed and confirm the deployed artifact before reporting production closure.

The same principle handles a detector version change: record it, inspect whether the original path was modeled and evaluated, then judge the behavior separately. Do not infer a rule regression merely because output changed; the manifest explains scope, while a tool-specific investigation would be needed to explain detector semantics.

Keep the scanner repair separate from the application repair

Harbor has two distinct defects in its workflow after the refactor: the unsafe query remains, and the selected scan no longer examines its new location. Restoring the shared directory to analysis can make the alert visible again, but it does not make the query safe. Correcting the query can remove the harmful behavior, but it does not by itself establish that future shared-code changes will be examined.

Give each action a different completion condition. The application owner supplies the fixed driver call, account scope and relevant deployment evidence. The scanning owner supplies the intended source selection, completion record and any analysis limitations. Linking the tasks is useful because they concern the same moved helper, but closing one should not automatically close the other.

For a release summary, show both results directly: “Query corrected at FIX_SHA; scan selection restored for shared/query; production rollout pending.” If the scan correction is still open, say so without weakening sound source-fix evidence. If the source repair is still open, a new alert is useful renewed visibility, not a regression caused by restoring coverage.

This separation also makes a future quiet run interpretable. The reviewer can compare the selected scope and rule configuration with the retained record, then inspect the particular behavior being closed. The team no longer needs to infer the reason for silence from a dashboard count.

Report the difference clearly to engineering and leadership

Engineering needs the old and new paths, the exact unresolved check and the proposed next action. Leadership needs the effect and decision, not a claim of zero risk. A concise update might say:

“The scheduled scan at NEW_SHA no longer reports the report-search query issue. Its configuration excluded the new shared helper, so this run cannot establish closure. The supplied source trace shows the unsafe query remains in that helper; the reports team must repair it and supply the production artifact ID. Status: still exists in source; production applicability is recorded separately.”

That statement is not pessimism. It prevents a dashboard's absence from being mistaken for a verified change in risk. OWASP's reporting guidance calls for scope and limitations to travel with a result. The same principle applies to a remediation update.

For the next quiet scanner run, perform the four gates on one previously approved finding and save the reason for silence. Then use the new-commit verification guide to make the closure record, or the bounded reassessment workflow if the refactor changed shared code.