Evaluate the claims a developer would actually receive, at a fixed repository revision and scope. Deduplicate equivalent alerts, select a documented sample, and have reviewers decide each claim against exact source and relevant countercontrols. Count independently supported claims, rejected claims, and unresolved claims separately. Report precision among decided claims as supported ÷ (supported + rejected), plus the unresolved share and median review effort. Do not call that number the tool's “accuracy” or treat a source-supported claim as an executed exploit. To learn whether the scanner misses real issues, run a separate known-issue or independently reviewed scope exercise.
The answer changes with what you are choosing. A tool for pull-request gating needs a low burden at the configured gate threshold; a baseline investigation tool may be acceptable if it offers useful leads that analysts can resolve. Compare tools only under the same scope and decision rules. OWASP Benchmark's scoring model uses both intentionally true and false cases because a tool that reports everything can catch known issues while producing unusable false positives; a tool that reports nothing creates no false alarms but finds nothing.
Define the unit before counting
Use one unique vulnerability claim at one pinned revision, actor, affected path and effect as the unit. Three AI messages pointing to the same route and failed control are one candidate with multiple evidence items. A new route or a different harmful effect may be a different candidate, even if the tool gives it the same weakness label. Publish your merge rule before the review. Otherwise a scanner can look better or worse simply by changing how it groups alerts.
Freeze the environment: repository revision, deployment assumptions, selected files, exclusions, model and scanner version, settings, prompt or policy version, budget and time limit. Record what was inventoried and what the scanner actually examined. Do not compare a targeted scan of one service with a full scan of a monorepo as if their denominators were equal. OWASP's description of static code analysis notes that automated analysis can produce false positives when relevant components are unavailable; missing context has to stay visible in the evaluation.
Define disposition in advance:
| Outcome | Review rule | Count in decided precision? |
|---|---|---|
| Supported | Independent review establishes a source-supported actor → entry → control gap → effect for the bound scope, with material counterevidence addressed. | Yes, as supported. |
| Rejected | The precise claim is contradicted by an applicable control, unreachable path, wrong asset, or another documented fact. | Yes, as rejected. |
| Unresolved | Missing source, deployment state, competing supported interpretations, or review limit prevents a decision. | No; report separately. |
“Supported” here means the claim survives source review; it is not a statement that an exploit was executed. If the evaluation includes authorized runtime validation, record that as a separate evidence level. A source-only exercise cannot prove production exploitability or absence of other vulnerabilities.
Illustrative measurement: one scanner, one application
Illustrative numbers and repository, not a product result or market statistic: An AppSec lead runs one AI scanner on an immutable revision of the fictional Meridian Portal. The selected scope is the web API and its access-control middleware; a payment worker in another repository is explicitly excluded. The team receives 64 raw alerts. Fourteen are repeats of the same actor-path-effect claims, leaving 50 unique candidates. Because the team can review all 50 this week, it uses a census rather than a sample. Two reviewers apply the rules above without accepting the scanner's severity as a verdict.
| Decision on 50 unique candidates | Count | Example reason |
|---|---|---|
| Supported by independent source review | 28 | A route accepts a resource ID and all reviewed controls omit the required ownership check. |
| Rejected with counterevidence | 12 | The route always passes through middleware that enforces the specific resource rule. |
| Unresolved | 10 | A downstream policy or deployment artifact is unavailable. |
The decided-case precision is 28 / (28 + 12) = 70%. The rejected share among decided claims, sometimes called the false discovery proportion, is 12 / 40 = 30%. The unresolved share of all unique candidates is 10 / 50 = 20%. Do not call 30% the statistical false-positive rate: that rate normally needs a known set of negative cases, including ones the scanner never flagged. In an ordinary repository, the true-negative denominator is usually unknown.
The ten unresolved claims matter. If all eventually proved rejected, supported share among all 50 would be 28 / 50 = 56%; if all proved supported, it would be 38 / 50 = 76%. That 56–76% interval is only a possible-outcome bound, not a confidence interval and not a forecast. The measured result today is 28 supported, 12 rejected, 10 unresolved. Report the raw counts next to the percentages so a leader does not mistake a small or incomplete denominator for a precise product property.
The AppSec lead also records review time: for example, 16 reviewer-hours for 50 candidates, with a median of 14 minutes per decided claim in this illustrative exercise. Keep the total and the distribution, because a few costly investigations can consume the week's capacity even when median triage is fast. Do not infer that a model saves time without comparing this workload with the team's existing review process.
Repeated runs reveal a different problem from false positives
An AI scanner can change which candidates it emits under the same intended settings. Keep the original 50-candidate census intact, then conduct a separately labeled repeatability exercise on the same pinned scope. Do not replace the first run with whichever rerun looks best.
A synthetic comparison might contain:
| Claim | Run1 | Run2 | Run3 | Adjudicated source status |
|---|---|---|---|---|
| C-1, missing export owner check | Reported | Reported | Not reported | Supported |
| C-2, guarded preview | Reported | Not reported | Reported | Rejected |
| C-3, opaque worker policy | Reported | Reported | Reported | Unresolved |
C-3's consistency does not make it true. C-1's absence in Run3 does not make the source repaired. C-2's intermittent appearance creates repeated review cost unless stable claim identity preserves the rejection and its revision-bound counterevidence.
For each run, retain its own unique candidate count, supported/rejected/unresolved outcomes and reviewed scope. You may also report the union of distinct claims across runs, explicitly labeled as a different population. Do not pool three repeated versions of C-1 as three independent successful detections. Model/version, policy, retrieval budget and source availability must stay visible; if any changed, the exercise measures that change as well as output variability.
Translate the measurement into staffing
The original example used 16 reviewer-hours for 50 unique candidates. Its overall mean burden is 16 × 60 / 50 = 19.2 minutes per candidate, including unresolved work; the stated 14-minute median concerns decided claims only. Those statistics answer different questions and can coexist.
Suppose the team has eight reviewer-hours next week. At the same illustrative aggregate burden, it can plan for roughly 25 candidates, not assume it can clear another 50 because the model bill is small. It could narrow the source question, reduce repeated leads through preserved adjudications, or schedule a documented sample for evaluation. Sampling evaluates quality; it does not verify all unreviewed candidates for developer publication.
For a live rollout, reserve time for hard unresolved cases rather than reviewing only easy dismissals. If the ten unresolved claims are all cross-service authorization questions, improving retrieval may be more valuable than increasing model confidence thresholds. If high-priority supported claims appear inconsistently, a single quiet run should not be used as a reliable gate without further evaluation.
Keep sampling and missed-issue checks separate
When a census is impossible, record population and sample size for each chosen stratum, then select cases randomly within those strata using a reproducible selection record. A convenience sample of easy-to-resolve alerts describes that subset; population weights alone cannot remove its selection bias. Report strata separately or use the documented population weights; oversampling dramatic alerts without weighting does not produce an overall rate. The general precision guide develops that denominator problem.
A separate known-issue corpus checks whether the scanner detects established unsafe paths. It should include guarded lookalikes and unresolved cases with honest labels. Building a real-repository corpus covers label validity. Precision among emitted claims cannot reveal an issue the tool never emitted.
The decision is therefore multi-part: Is the developer-facing queue useful? Can the team afford its review burden? Are material known paths missed or unstable across runs? OWASP Benchmark provides controlled positive/negative scoring context, but none of the fictional numbers here measures a product or predicts a future repository's performance.
Interpret the result without overpromising
A decided-case precision figure depends on the repository, category mix, source completeness, tool settings and review rules. It may shift after a model update, prompt change or new framework. A small sample should be presented as a bounded observation, with the underlying counts and unresolved cases, not a universal rate. NIST SSDF calls for defined criteria and information gathering around software security checks; a repeatable decision record is more useful than a vendor's single percentage.
For a purchase or rollout decision, set an acceptable weekly review budget and a minimum evidence standard before seeing results. If too many claims are unresolved because the tool cannot retrieve middleware or deployment context, improving retrieval may matter more than changing a confidence threshold. If many claims are rejected by the same control, improve retrieval using those cases, then evaluate on a separate, preselected set whose adjudications were not supplied to the scanner. Rerunning the tuning cases checks that change; it does not show how well the tool handles unfamiliar cases. If precision is high but known issues are missed, widen scope or add a complementary method before calling the tool effective.
Record why cases were hard to decide
The unresolved share tells the lead how much uncertainty remains, but the reasons determine what to change. For Meridian, classify the ten unresolved cases by their decisive missing artifact. If six need the same payment-worker repository, three need deployment mapping, and one needs an unwritten business rule, those are three different constraints. A larger token allowance might help none of them.
Record one primary reason per case for this particular breakdown so the counts reconcile, while retaining secondary reasons in the evidence record. The categories are a proposed evaluation aid, not universal scanner states. They help the purchasing team distinguish a tool that cannot retrieve available code from an assessment that was deliberately denied access to necessary code.
Also record reviewer disagreement before reconciliation. A claim initially supported by one reviewer and rejected by another should receive a final disposition based on evidence, but the disagreement is useful context about label difficulty. Do not silently discard that claim from the denominator because it makes comparison inconvenient. If adjudication remains unsettled, it belongs in unresolved work.
When presenting the pilot, include one representative rejection, one supported claim, and one unresolved case alongside the counts. The examples let engineering leaders judge whether the review rules match their intended workflow. A scanner may be suitable for an analyst's investigation queue while producing too much uncertainty for an automatic merge gate. That is a deployment decision the pilot can support without claiming a universal quality score.
The next action is to take one recent scanner run, deduplicate its claims, and adjudicate a documented sample with the three-outcome table. For the reviewer procedure behind each disposition, use the AI report verification checklist. For the decision rule that keeps unsupported claims out of the developer queue, continue with why AI findings need independent verification.