Compare static application security testing tools on the same code, rule scope and review process, then judge supported issues found, known issues missed, reviewer effort, developer usefulness, coverage limits, operational fit and source-handling requirements. Total alert count measures output volume. It does not tell you which tool finds the problems your team can actually fix. The answer changes with language, framework, rule configuration and application architecture, so use a representative sample of your code as well as any public benchmark.
SAST examines code without needing a live attack against the running application. That makes it useful early in development, but source alone may not reveal deployed routes, runtime controls or real attacker reach. A report should distinguish a plausible source path from an executed exploit. OWASP SAMM's tool-selection guidance names inspection depth and accuracy, integration, usage and cost as relevant factors. Treat those as evaluation questions, not claims that one product wins universally.
Define the comparison unit before opening dashboards
Decide what an “issue” means. One unsafe data flow might trigger five rules and ten file-level alerts. Another tool may group it into one finding. Counting raw alerts rewards the tool that splits more. Use the underlying harmful behavior at a pinned revision as the primary unit, with a record of the affected entry path and asset. Count duplicates separately because they consume review time even if they are not distinct bugs.
Write down the configured scope: repository, commit, languages, framework, generated code treatment, enabled rule sets, excluded paths, time/credit limit and whether manual tuning or vendor assistance was allowed. A candidate with a narrower rule set may look quieter because it never attempted the same checks. A candidate that cannot parse a crucial framework needs a coverage note, not a misleading zero-finding result.
Prepare a reference set with a few previously reviewed issues and safe patterns. For each known issue, retain a source-supported path, any deployment evidence and the reason a counterexample does or does not defeat it. Avoid relying on one kind of weakness. OWASP Benchmark offers known Java and Python cases for exploring scanner behavior; it is a useful controlled input, while your production framework and business logic may be quite different. OWASP ASVS can organize the control questions, but a tool's mapping to a requirement is not a verified pass for the whole control.
Record outcomes beside evidence maturity
Compare coverage, finding validity and misses, developer handoff, workflow, source handling and total effort. For each, distinguish an assertion from a demonstration and a result your team can repeat. A repeated failure is strong evidence of a weakness, not a favorable product score. Declare hard requirements before opening outputs; a failed source-handling gate cannot be offset by a useful finding.
The worked comparison below makes object authorization the required job. If your priority is different, write it before looking at the table rather than changing the weights to favor a preferred product.
Measure the parts you can actually measure
For an adjudicated sample, track three quantities at the level of distinct underlying issues:
- Supported-result share: supported issues among all reviewed distinct claims from that tool. Report unresolved claims separately, not as true or false positives.
- Known-case recall: known, in-scope reference issues surfaced divided by all known, in-scope reference issues. State the tiny denominator and the weakness mix.
- Reviewer effort: minutes spent on intake, source tracing, counterevidence, decision and handoff for each reviewed issue; include duplicate handling.
These are local pilot observations, not a laboratory estimate of the tool's accuracy. “False positive” should mean the claim was checked and defeated by evidence under the defined scope. “Not actionable” can also mean the issue is real but low priority, duplicated, owned elsewhere or missing context. Keep those labels separate. Unreviewed claims are unresolved, even if a vendor calls them high confidence. A false-negative count is harder still: unknown issues cannot be counted. Use your prepared reference cases and sampled unflagged paths, and say that these methods can miss other flaws.
Severity badges do not repair poor evidence. FIRST's CVSS v4.0 specification separates Base, Threat and Environmental information; local exposure and business consequence can change the priority decision. Ask a tool to show the prerequisites and affected asset instead of only a score. A critical rule firing in test-only code and a lower-scored cross-account read in a deployed route require different work, provided both claims are actually supported.
Illustrative two-week evaluation: fewer alerts, more useful work?
Illustrative case; invented products, application and numbers. This is not a vendor ranking. Cedar Harbor evaluates Scanner A and Scanner B on the same pinned customer API and batch worker. A reviewed reference set includes three in-scope source-supported issues: an authorization gap, an unsafe input path and a hardcoded secret in a test fixture. The team explicitly separates the fixture secret from production risk. One known safe pattern uses a shared authorization helper that a shallow local rule might miss.
| Pilot observation | Scanner A | Scanner B | What the team checks |
|---|---|---|---|
| Raw alerts | 38 | 11 | Are they distinct paths or duplicates? |
| Distinct claims in the bounded sample, all reviewed | 12 | 9 | Same selection method and reviewer? |
| Supported after independent review | 4 | 5 | What path and counterevidence support each? |
| Rejected after review | 6 | 2 | Why was each wrong? |
| Unresolved | 2 | 2 | Which missing facts would settle them? |
| Known in-scope reference issues surfaced | 2 of 3 | 2 of 3 | Which issue did each miss? |
| Review minutes on selected claims | 250 | 170 | Are duplicate and clarification minutes included? |
Here is the identity map behind those totals. K1–K3 are the three known reference cases; N1–N3 are additional supported claims. R1–R6 are distinct rejected claims and U1–U2 remain unresolved. The same identity denotes the same underlying claim across tools. This is the entire bounded comparison sample, not a random sample of all possible application defects.
| Shared claim identity | A raw messages | B raw messages | Adjudication and relevance |
|---|---|---|---|
| K1: customer object-boundary gap | 0 | 1 | Supported; the buyer's predeclared required case |
| K2: unsafe input path | 5 | 1 | Supported; surfaced by both |
| K3: credential-like secret in the synthetic test fixture | 4 | 0 | Supported for fixture detection; no production credential or exposure implied |
| N1: export permission gap | 3 | 1 | Supported by both |
| N2: unsafe archive path handling | 2 | 1 | Supported by both |
| N3: separate job authorization gap | 0 | 1 | Supported by B only |
| R1: safe parameterized query | 3 | 2 | Rejected |
| R2: compulsory downstream account control | 3 | 1 | Rejected |
| R3–R6: four other defeated claims | 12 total, 3 each | 0 | Rejected with separate counterevidence |
| U1: effective delivery policy unavailable | 3 | 2 | Unresolved |
| U2: deployment entry not established | 3 | 1 | Unresolved |
| Raw total | 38 | 11 | A: 12 distinct; B: 9 distinct |
A's supported set is K2, K3, N1 and N2: four. B's is K1, K2, N1, N2 and N3: five. The union contains six supported claims, not nine, because three overlap. A alone surfaced K3; B alone surfaced K1 and N3. The equal 2/3 known-case count conceals that A missed the required customer-boundary case and B missed the fixture secret.
The supported shares are 4/12 and 5/9 when unresolved claims stay in the reviewed denominator and are separately disclosed. If you instead calculate among resolved claims, the denominators become 10 and 7; label that different measure. Neither choice licenses calling the two unresolved claims false positives. Neither is a general precision estimate for future code, because this is a small, selected sample.
The effort record reconciles the 250 and 170 minutes: A used 50 on duplicate grouping, 150 on source/counterevidence review and 50 on clarification; B used 15, 120 and 35. These are active minutes for the specified sample, not scan duration or full license cost. Setup and integration effort need their own lines before a purchase decision.
Write the decision before adding a weighted score
The buyer declared K1-style object authorization essential before opening results. On this packet, B provides the required case, an additional supported job claim and lower review effort. Both candidates are assumed to meet the example's separate access and processing gates; if that assumption fails, it changes the decision.
Choose B for the evaluated authorization-review workflow, subject to the demonstrated handoff and reassessment gates. Keep an independent secret-detection control because B missed K3. Retain U1/U2 as unresolved. A's extra fixture result matters, but it does not compensate for missing the declared required boundary case. The conclusion covers this stack, configuration and packet; it is not a universal product ranking.
If the real purchase were primarily a secrets gate, the same result would support a different choice or a broader evaluation. That is why the job and hard requirements belong in the protocol, rather than being invented after a preferred dashboard wins.
A useful finding note in this pilot might read:
Claim: An ordinary account member may request a report ID belonging to another account and receive its export. Evidence: the selected route passes the ID to the export service; the service lookup is keyed only by report ID at the pinned revision. Counterevidence checked: sign-in middleware is present, but no report ownership check was found on this traced path. Still unknown: whether a deployed data policy blocks the foreign report. Status: unresolved pending policy evidence. Next owner: AppSec and report-service engineer.
The note is actionable even before approval because it asks the exact missing question. Keep its verdict unresolved while the decisive deployed policy is unknown. An owner can still investigate or make a proportionate defensive correction; track that work without presenting the allegation as a confirmed vulnerability. If the policy blocks access, the claim should be rejected with evidence. If it does not and independent review supports the path, the issue can be approved and handed to an owner.
Test how the result enters engineering work
Have a developer inspect one supported finding without the vendor narrating it. Can they understand the harm and find the relevant code? Does the suggestion preserve the actual policy or merely silence the rule? Ask whether the scanner links the issue across renames and revisions, whether repeated scans create duplicate work, and whether an owner can dispute a claim with counterevidence. A clean result after a merge is insufficient if the changed path fell outside scope.
Look at the integration artifact. GitHub's SARIF documentation explains how third-party analysis can appear as code-scanning alerts; the file format and GitHub display are only part of the workflow. Verify stable issue identity, repository permissions, access to sensitive source excerpts, export fields and CI behavior on a partial scan. Decide whether the desired mode is advisory, scheduled or a bounded gate, and require the tool to represent uncertainty in that mode.
Before signing, compare the vendor's claimed locations and permissions with the actual integration and contract. Ask where source, prompts, logs and findings are processed and retained, who can read them, and how access is revoked. Add analyst and developer time to the license and run costs. The best score on a public test suite cannot compensate for a failed hard requirement in your environment.
Write the final memo as “chosen for this use case under these conditions,” with known misses and review burden visible. Then run the two-week application scanner pilot if you have only seen demos, or calculate alert-noise cost if two candidates have similar validated detection but different triage workload.