How to build a repeatable false-positive review process

A repeatable review process makes every scanner claim answer the same practical questions: what harmful action is alleged, which revision and scope were assessed, what evidence supports the path, what could disprove it, and who has authority to decide? Give each alert a stable record and one of three outcomes: supported and approved for action, rejected with counterevidence, or still under investigation with a named missing fact. Then feed recurring errors back into the rule or configuration. The process changes for urgent suspected exposure: notify the service or incident owner immediately, while keeping the technical verdict provisional.

The goal is a trustworthy fix queue, not the smallest possible alert count. NIST's Secure Software Development Framework, PW.7 describes recording and triaging issues from code review and analysis in the development workflow, with human review of tool output. That is a useful minimum: the tool supplies leads; accountable people resolve what each lead means in this application.

Define the unit and the clock before reviewing anything

Decide what one review record represents. A good default is a distinct claim about one harmful behavior in one affected scope at one pinned revision. Five alerts from different rules may support the same claim; preserve each source, but do not open five identical developer tickets. Conversely, two routes with different controls should not be merged merely because they share a weakness label. Record the tool, rule/version, run, alert identifiers, repository, immutable revision and excluded scope at intake.

Give the process an owner and a service-level target rather than a promise of instant certainty. For example, the AppSec intake owner checks high-consequence alerts the day they arrive, assigns a reviewer within one working day, and records a next-evidence date for unresolved work. Set targets to your team's capacity and risk, and measure misses. An unresolved alert can remain urgent; “not yet proven” must not quietly become “safe.”

Use six gates with explicit outputs

The path gate is where a line citation often loses its apparent certainty. A cited line establishes that code exists, while the security conclusion may depend on framework wiring, a downstream check or a production setting. The counterevidence gate should be symmetrical: record why a defense does cover this path, not merely that a function with a reassuring name exists. An independent reviewer need not run an exploit to compare the same bounded source and deployment evidence.

Work an illustrative alert all the way through

Illustrative case, not a customer alert: Cedar Desk's secret scanner flags a 32-character string in docs/examples/login-response.json at revision c104. The rule calls it an API key. The documentation team says it is example data and asks AppSec to dismiss it immediately.

The intake reviewer writes the claim as: “A usable service credential may have been committed in a public documentation example.” The filename is weak counterevidence. A real credential can be pasted into documentation, and a scanner cannot infer issuance from the directory name. The reviewer asks the credential-issuing owner to check whether the value or its fingerprint was ever issued, checks the repository history for earlier appearances, and confirms the example's generation process. The team avoids copying the candidate credential into broad tickets, chat or logs. In this invented case, the issuer confirms that the value belongs to a reserved synthetic fixture set and was never issued; the documentation generator produces it deterministically, and no deployed secret reference points to it. A second reviewer checks the issuer evidence and generator at the pinned revision. The specific “usable credential exposed” claim is rejected.

The filled review record could read:

• Review ID: CD-FP-014; raw alert scan-88/rule-secret-4.
• Claim: committed string may be a usable service credential.
• Scope: Cedar Desk docs repository, revision c104; publication history reviewed; credential issuance checked by service owner.
• Support: secret-shaped value in an example JSON file; exact range and hash retained in a restricted evidence store.
• Counterevidence: issuer records identify the value as a never-issued synthetic fixture; generator source reproduces it; second reviewer checked both.
• Disposition: rejected for this value and revision. This says nothing about other secret-shaped values in the repository.
• Review trigger: generator changes, issuer changes fixture policy, or a new alert names a different value.
• Rule feedback: add a narrowly scoped fixture recognition rule if it can be validated without masking real credentials.

If the issuer cannot establish whether the string was ever valid, the outcome is investigation, not rejection. The security owner may rotate or revoke a possible credential as a precaution while the claim is unresolved. If the issuer confirms it was live, open a supported issue and follow the organization's incident process; deleting the line alone would not undo earlier exposure.

Run a week with more work than reviewer time

A process becomes useful when it decides what waits. This synthetic board has four distinct claims from five raw alerts; the duplicate is linked to A2. The available independent-review capacity is two material decisions this week, an illustrative planning constraint rather than a benchmark.

Capacity is spent on F1 once its decisive evidence arrives and on A2's material path. D3 does not consume a final-review slot while its required input is absent, but it still has an owner and a scheduled check. The precaution on A2 is recorded separately from approval so the service owner can act before the technical decision is complete.

Next Monday, the fixture generator changes. F1's old rejection does not automatically cover the new string. The analyst creates a new investigation linked to F1 and asks the issuer about the new value. If the issuer cannot answer, the status remains unresolved; “same docs directory” is not reusable counterevidence.

The week's improvement proposal is correspondingly narrow: recognize only the reserved fixture set whose issuance policy and generator were checked, and invalidate that recognition when either changes. Do not exempt all JSON examples or all strings emitted by an unreviewed generator. The proposal needs tool-specific evaluation before enabling it.

Keep a disposition reason and reversal path. GitHub's alert workflow supplies one example of recorded dismissal and reopening behavior; teams using another tool need to check its semantics. The board's value is that the next owner can see a real decision or an exact missing fact, rather than four variations of “security reviewing.”

Make the review process survive an absent reviewer

A process is repeatable only if someone else can pick up its unfinished work. At Friday's handoff, D3 should name the exact route map requested, the platform owner and the decision that map will settle. “Waiting on platform” is not enough: it does not reveal whether the request was sent, what artifact is needed or what can happen while it is missing.

Use an existing tracker rather than building a separate bureaucracy. A small team can keep the claim, evidence status and next action in one issue with restricted attachments. The six gates describe questions to answer; they need not become six meetings or six people. Reserve independent challenge for consequential, disputed or uncertain conclusions, and let routine evidence reuse receive proportionate review.

For example, R4's recurrence can link to the earlier scoped-adapter decision after a reviewer checks that the relevant source and binding are unchanged. If the rule's meaning changes, that comparison is no longer sufficient. F1's new fixture value needs issuer evidence because the original never-issued conclusion concerned a different value. These two recurrence cases should not receive the same mechanical treatment.

At the weekly retrospective, choose one avoidable delay. If every deployment-dependent claim lacks an artifact identity, add that field at intake. If issuer requests go to the wrong team, improve ownership. If a reviewer repeatedly reconstructs the same wrapper, preserve its bounded evidence for reuse. Measure whether that specific change reduces rework without increasing unsupported dismissals. A lower queue count alone cannot tell you whether the process became more trustworthy.

Measure whether the process reduces wasted work

Track at least four separate counts per cohort: alerts received, distinct claims after deduplication, claims independently adjudicated, and claims still unresolved. Then track review time, reasons for rejection, reversals after reopening and developer tickets withdrawn because the evidence failed. Do not report “90% true positives” from only the easy alerts that reviewers finished. NIST's SATE V report explicitly discusses ground truth and the limits of metrics when only some warnings can be reviewed. Keep the cohort and unresolved denominator visible.

This process also has a cost: independent review takes time. Aim that effort at claims likely to reach developers or cause material harm. A low-consequence duplicate can receive a lighter review; a potentially exposed customer asset needs stronger evidence and faster escalation. Do not weaken the disposition standard to make the dashboard look cleaner. OWASP's reporting guidance emphasizes scope, limits and enough technical detail for the receiving team to act; a review record should provide the same clarity before a claim becomes a finding.

Start with one week of alerts and run every candidate through the six gates. Record where the process stalled and change one intake field or rule at a time. For the decision after review, use the suppression guide. For reporting alert quality, use the scanner precision measurement guide.