How to measure scanner precision without gaming the denominator

To measure how often alerts are useful, first fix a cohort of all alerts emitted for a declared scope, revision range, scanner version and time window. Deduplicate by a written claim rule, independently adjudicate every claim you can, and leave unresolved claims visible. Among resolved claims, precision = supported true claims ÷ (supported true claims + rejected false claims). Call that reviewed precision if only part of the cohort was reviewed; do not present it as the scanner's overall precision without a defensible sample. A change in rules, exclusions or review selection can improve the number while making detection worse, so report coverage and suppressions beside it.

The phrase “false-positive rate” is often used loosely to mean “fraction of alerts we rejected.” In measurement terms, that is the false discovery fraction, FP / (TP + FP), the complement of precision for resolved alerts. The strict false-positive rate is FP / (FP + TN): it needs a count of non-vulnerable opportunities the scanner did not flag. Alert review alone does not supply those true negatives. NIST's SATE V report defines true/false positives and negatives, precision and recall separately, and discusses why ground truth is hard in production software. Name the metric you actually measured.

Write the measurement contract before seeing results

Use a one-page card so the next month uses the same rules:

This definition prevents two common denominator tricks. First, do not count one genuine bug once while counting ten duplicate false alerts as ten separate failures, or do the reverse. Decide whether the business question is raw alert burden or distinct claim quality, and show both counts. Second, do not calculate precision from only the easy alerts that a reviewer happened to close. The unresolved and unreviewed claims remain in the cohort.

NIST describes SATE as a methodology study rather than a ranking of the “best” tool. Its SATE V report notes that realistic software, enough instances and complete ground truth are hard to obtain together. A local production-code metric should therefore say how labels were reached and where they may be wrong. A source-supported path is not an executed exploit; for some claims, runtime or deployment evidence is decisive.

Work a stratified sample without hiding the weighting

Illustrative numbers, not CodeSecPilot results: A security team freezes a four-week cohort of 120 distinct SAST claims: 30 authorization claims and 90 injection claims. It randomly selects 20 authorization and 40 injection claims for independent review. The rule versions and scope are fixed. Reviewers label a claim supported only after source and applicable deployment evidence justify the path; they reject with counterevidence and leave missing facts unresolved.

The naive 26 / (26 + 28) = 48.1% describes resolved claims in this reviewed sample. It overrepresents authorization alerts, because two-thirds of that stratum were sampled while less than half of injection alerts were sampled. To estimate the cohort mix, weight each reviewed authorization claim by 30/20 = 1.5 and each injection claim by 90/40 = 2.25:

• Estimated supported: 14×1.5 + 12×2.25 = 48.
• Estimated rejected: 4×1.5 + 24×2.25 = 60.
• Estimated unresolved: 2×1.5 + 4×2.25 = 12.
• Weighted precision among resolved claims: 48 / (48 + 60) = 44.4%.

Those 48, 60 and 12 are weighted estimates, not counts of 120 individually examined cases. If all weighted unresolved claims eventually prove false, the corresponding precision estimate would be 48/120 = 40%; if all prove supported, it would be (48+12)/120 = 50%. This 40–50% range shows the effect of unresolved labels within the sample estimate. It is not a statistical confidence interval and does not account for sampling error or reviewer mistakes. Publish the sample design and raw counts alongside the estimate so a reader can see the limits.

If the team cannot sample randomly, call the result “precision among reviewed claims” and show how cases were chosen. A convenience sample of high-severity alerts or developer disputes cannot support a statement about all emitted alerts. A “verified only” sample is even less useful: it excludes the numerator's competitors by construction.

Compare periods without rewarding lost coverage

Extend the example with a second synthetic period. A noisy injection rule is disabled. The remaining mix contains 30 authorization and 30 injection claims, all reviewed: authorization has 21 supported and 9 rejected; injection has 12 supported and 18 rejected. There are no unresolved claims in period two. This is a new cohort, not follow-up adjudication of the first sample.

The first period's weighted estimate remains 44.4% among estimated resolved claims; it should not be silently replaced by 48.1% when that makes the trend look more attractive. Nor can the missing 60 second-period claims be assumed false: they were never emitted under the reduced rules.

Decision memo: “Do not claim overall precision improved from this comparison or retain the broad injection-rule disablement solely on these figures. Reevaluate the omitted rule on a comparable authorized cohort, report its supported and rejected claims separately, and disclose the scope reduction. The defensible observation today is lower emitted burden under a smaller rule set.”

This memo rejects an attractive but unsupported conclusion while preserving a useful operational fact. A separate tool-evaluation exercise can compare identical cases under both configurations; this article has not performed one.

Correct labels without rewriting history

Suppose one of period one's 14 sampled authorization claims was originally marked supported, but a later evidence audit finds a compulsory control already present at that original revision. Correct the original cohort label, with the reason retained:

• Sample totals become 25 supported, 29 rejected and 6 unresolved; reviewed precision is 25/54 = 46.3%.
• Weighted supported becomes 13×1.5 + 12×2.25 = 46.5; weighted rejected becomes 5×1.5 + 24×2.25 = 61.5.
• Weighted precision among resolved claims becomes 46.5/108 = 43.1%.

Fractional weighted counts are estimates, not half a finding. This correction is appropriate because the original judgment was wrong. If the original flaw was real and a later patch fixed it, its historical label remains supported. Risk acceptance likewise changes response, not truth. A suppression changes visibility, not the cohort's adjudicated label. Keep original and corrected versions of the metric so readers understand the revision.

Use the numbers to choose an improvement experiment

The measurement should lead to a decision the team can evaluate. In the first synthetic period, the injection stratum consumes review time and has many rejections. That observation identifies a place to inspect reasons; it does not justify disabling the whole rule group. Separate rejections caused by a repeated safe wrapper from unresolved deployment gaps or incorrect human labels. Each cause suggests a different improvement.

Suppose the review notes identify several alerts whose counterevidence is the same fixed query wrapper. The team can propose a narrow model adjustment and compare the old and proposed configurations on a fixed authorized set of cases. Include known safe uses of the wrapper and deliberately distinct unsafe callers. The purpose is to see whether the adjustment removes the targeted noise while retaining useful warnings on cases the wrapper does not protect. No such comparison has been run for this article.

Define success before looking at the resulting totals. It might require fewer repeated wrapper rejections, no loss of detection on the selected unsafe comparison cases, and lower analyst effort for the same review scope. Passing those conditions would support that bounded configuration change. It would not establish recall across all production code or predict the next repository's results.

Also record the cost of obtaining labels. A small increase in reviewed precision may be unattractive if it requires much longer investigations or leaves more high-consequence claims unresolved. Conversely, a tool that emits additional valid findings may lower the apparent average review speed while doing useful work. Reporting time alongside outcome and scope helps the team understand these tradeoffs.

For later cohorts, retain the measurement contract even when the result is disappointing. If ownership or deployment evidence is missing, show the unresolved share and the practical work needed to reduce it. Do not move those cases out of the denominator to produce a cleaner chart. If an operational queue intentionally prioritizes high-consequence findings, report its reviewed precision as a selected workflow measure, separate from any random-sample estimate.

The reader should be able to reconstruct every displayed percentage from the published counts and weighting rule. That simple check prevents arithmetic presentation from concealing a changed cohort, changed label policy or reduced scope.

Check the metric against work and coverage

Precision answers “How much of what we reported was supported?” It does not answer “How many real vulnerabilities did we miss?” That is a recall question requiring credible ground truth for missed weaknesses, which production teams rarely have for an entire codebase. Record completed and excluded scan scope, rules enabled, files or components not reviewed, and any analysis failures next to precision. A scanner that raises one correct alert while skipping most of a repository can have 100% observed precision and still be unhelpful.

Also record the median analyst minutes per resolved claim, time to an actionable developer handoff, reversals after rejection, and the number of verified findings per completed assessment scope. These are operational measures, not substitutes for precision. Review them by repository, language, rule and severity before changing a global rule; a good aggregate may hide one especially noisy or harmful stratum. Compare periods only after accounting for changed tool versions, code mix, review policy and suppression rules.

For the next monthly review, freeze the cohort before adjudication, fill the measurement card, and publish raw alert count, distinct claim count, sampled and resolved counts, weighted estimate if sampling permits it, unresolved share and scope gaps together. Use the repeatable review process to make labels credible, and the rejected-claims history guide to retain the evidence behind them.