How to compare scanner findings when tools disagree

When two static-analysis tools disagree, compare the claim and evidence at the same revision and scope, not their alert totals. Translate each result into an actor, entry point, control and proposed harmful effect. Then inspect the code that could support or defeat that claim. A tool's silence means only that it produced no alert under that run's configuration; it does not prove the other tool wrong. Two alerts can also describe the same code with different rules or severities. Your conclusion changes when one run excluded the relevant files, a tool modeled a framework differently, or a control on the actual path settles the question.

This is a review workflow, not a contest between vendors. OWASP's ASVS assessment guidance notes that automated static and dynamic tools can find some issues but cannot, by themselves, fully verify complex access-control and business-logic requirements. A disagreement is a reason to inspect the application-specific path and the tools' methods.

First make the two runs comparable

Before reading either alert as a verdict, capture repository, commit, selected paths, language versions, tool versions, rulesets, exclusions, build or extraction mode and run completion status. If one tool analyzed main on Tuesday and another analyzed a release branch on Friday, you have a revision difference before you have a tool disagreement. If one run skipped the service because extraction failed, its empty result says nothing about that service.

Normalize the issue into one sentence. For example: “An ordinary account user can request an invoice belonging to another account through the download route because no ownership check runs before the PDF is returned.” This is more useful than trying to compare “broken access control” with “user-controlled key.” MITRE's CWE-639 describes the underlying pattern: access to a record identified by a user-controlled key without sufficient authorization for that record. The category helps name the hypothesis; it does not settle whether this repository has the weakness.

Then separate four differences that often get bundled together:

Normalize raw alerts before comparing them

Synthetic tool output; no scanner was run. Two reports arrive with apparently conflicting labels:

A-14 / HIGH / User-controlled object key
  routes/download.ts:44 → invoices.load(id) → sendPdf(row)

B-09 / MEDIUM / Missing authorization in background task
  workers/export.ts:71 → invoices.load(id) → archive.add(row)

Both cite invoices.load. They are not duplicates yet. The first claims an unauthorized direct PDF download; the second claims unauthorized archive content through a worker. The actor, entry point and required controls differ. Normalizing to a shared CWE alone would erase the difference that matters.

Now the manifests introduce another problem:

A: commit C91E4; api/ and shared/ analyzed; workers/ excluded
B: commit C91D8; api/, workers/, shared/ analyzed

Stop the same-revision comparison. B's export lead may still be worth reviewing on C91D8, but its silence or alert cannot grade A's behavior on C91E4. Request a matched run or explicitly keep the two source revisions separate. Do not average their severity ratings while this input mismatch remains.

At matched input, let the controls decide

Assume a later authorized comparison supplies both outputs for C91E4. Open route registration:

GET /invoices/:id/download
  → require_current_account
  → require_invoice_owner
  → download_handler

queue export_invoice
  → export_worker
  → invoices.load(id)
  → archive.add(row)

The direct route's owner wrapper compares the persisted invoice account before PDF delivery. Tool A's displayed path omitted that wrapper. Reject the exact direct-download claim with its registration and check cited. B's lack of a direct alert is consistent with the result, but silence is not the evidence that resolved it.

The export worker's trace still lacks the job authority and downstream archive-release behavior. Its alert remains a distinct investigation. A generic warning at invoices.load does not tell you whether the worker is exploitable, and rejecting A does not disprove B.

Suppose the worker is later found to load each invoice through the same mandatory owner policy using a current delegated actor. Then the export claim can also be rejected—with its own evidence. If instead it trusts the invoice IDs in the queue, it becomes a stronger source concern. The useful comparison is now between controls and claims, not brand names.

If two matched reports support the same harmful path but assign different severity labels, preserve each tool's rating method and record the team's own rationale separately. Reopen technical investigation only if the rating difference exposes a disputed fact, such as required privilege or affected data; changing labels alone does not create a second vulnerability.

Preserve disagreement that reveals a modeling gap

A tool may omit a route wrapper because its framework model is incomplete; another may suppress all reports on a helper named authorizedLoad. Inspect the actual behavior before tuning either. A rule change that teaches the wrapper's semantics may reduce noise. A broad suppression on invoices.load could also hide the unresolved worker path.

Keep raw result IDs, normalized claim ID, revision and counterevidence together. If multiple alerts really describe the same entry, missing control and effect, one disposition can reference them all. If fixing the direct route leaves export unchanged, they need separate closure checks even if a shared remediation ticket eventually covers both.

CodeQL's dataflow documentation explains why modeled paths depend on available code and framework knowledge. It does not make a majority vote among models an adjudication method. A reviewer must resolve the actual edge, or name the missing artifact instead of forcing agreement.

Reconcile the work without erasing either result

Keep two levels of identity in the review record. A-14 and B-09 are raw tool results; the direct-download and export questions are application claims. One claim may reference several raw results, and one broad raw result may need to be separated into more than one claim. Preserve the original text and location so a later analyst can reconstruct the mapping. Renaming an alert to match an internal taxonomy should not delete what the tool actually asserted.

For the direct-download rejection, attach the route registration, owner-wrapper implementation and the matched revision. Explain why the wrapper is mandatory before the PDF is returned. A helper with the right name would not be enough if another route could call the handler directly. For the export investigation, retain its distinct queue entry and list the missing delegation and delivery facts. The shared database helper can remain a common reference without forcing the two claims into one ticket or one status.

The engineering work can still be coordinated. If the team plans a shared helper that makes an account-bound lookup easier to use correctly, one change may improve both paths. The ticket should list which behaviors it intends to change and which acceptance evidence applies to each caller. Closing that ticket does not automatically settle the export claim if its final delivery policy is still unknown. Conversely, the team can implement a justified protective change without waiting for agreement on a disputed severity label.

Challenge a proposed tuning rule with the other path

Continue the fictional case after A's direct alert is rejected. Someone proposes suppressing every finding whose path includes invoices.load, because that helper appeared in the false alert. Use B's worker path as the immediate challenge: does the proposed suppression depend on the mandatory owner wrapper, or only on the helper's name? If it ignores the wrapper, it could remove the unresolved worker lead for a reason that has never been established for that caller.

A narrower model would represent the actual control and its conditions. The evaluator must check that it applies to the direct route's call sequence and does not credit a caller that lacks that sequence. If implementing that model is impractical, keep the suppression attached to the settled direct claim, exact scope and review conditions. The purpose is to reduce repeated work on known counterevidence while preserving materially different paths. A broad reduction in alert count is not sufficient evidence that tuning succeeded.

When the framework or wrapper changes, revisit the assumption rather than inheriting the old verdict through a similar function name. A new route may omit registration, pass a different account context, or use a helper only to display a warning before proceeding. These are hypothetical ways the earlier rationale could cease to apply. The relevant question is whether the control that defeated the original claim still governs the proposed actor's operation before content is returned.

After a tuning change, compare the retained guarded example and the unresolved worker example under the revised configuration. Record what the engine emits and what those outputs mean. Do not use a tuned example alone to claim general detection improvement; the exercise checks the local model change. A broader performance claim needs a separately defined evaluation with settled labels and a known population.

Close the disagreement with a decision, not unanimity

The final review note can say that A's direct-download claim was rejected by a mandatory control, while B's export claim remains unresolved for named missing evidence. Tool owners can then improve models or configuration without changing those application conclusions. The two products need not show identical dashboards for the team to have resolved the useful part of the disagreement.

If later evidence settles export, add the new disposition with its date and scope. Keep the earlier uncertainty visible rather than rewriting it as something the team always knew. This history is valuable when another tool reports the path again: the analyst can check whether the old deciding control still applies before repeating the investigation or dismissing a new variant.

If neither tool can settle it

Some disagreements end at an opaque dependency, an unavailable deployment mapping or a business rule neither scanner knows. Keep the claim in investigation, state the plausible impact conditionally, and assign the missing evidence to the owner who can provide it. If a serious exposure is plausible, the service owner can consider temporary containment while review continues; containment does not promote the claim to verified. OWASP's Web Security Testing Guide reporting chapter calls for a report that both technical staff and decision makers can use; a candid unresolved status serves them better than a forced consensus.

For the next disputed alert, create one row per claim and attach both run manifests before debating severity. Use the dataflow-trace guide when an edge is unclear, then apply the finding-triage gate before handing any confirmed issue to engineering.