Ask the vendor to define exactly what it calls a verified vulnerability, then make that definition survive a blinded review of your own authorized code. A convincing result should identify the pinned revision, affected path, attacker prerequisites, relevant controls, plausible harmful effect, counterevidence considered, and what remains untested. Inspect rejected and unresolved claims as closely as published ones. A model's confidence, a source citation, a severity score, or a successful search for a risky pattern is not independent verification. If the product only supplies candidates for your team to verify, that can still be useful; evaluate it as a candidate finder.
The decision changes with the claim. Source review may support a code-path conclusion without proving that the behavior is reachable in production. A vendor that claims runtime exploit validation needs permissioned runtime evidence and rules of engagement. OWASP's Web Security Testing Guide reporting section expects scope, limitations and technical detail that lets a reader understand and resolve a finding. Use that level of traceability for the pilot, while making clear which facts came from source and which came from a controlled test.
Translate “verified” into an observable state
Before the demo, ask for the vendor's written state model. At minimum, distinguish:
- Candidate: a tool suspects a problem and has evidence worth reviewing.
- Supported source finding: a reviewer has traced a plausible actor-to-effect path through the relevant source and checked available controls, with remaining deployment assumptions stated.
- Runtime confirmed finding: an authorized test in a specified environment demonstrated the described behavior under recorded conditions.
- Rejected or unresolved: counterevidence defeats the claim, or required context was unavailable.
These are evaluation labels, not a universal certification scheme. Ask who or what moves a result between states, whether a second independent check is performed, whether a human can override it, and whether overrides preserve an audit trail. A model checking its own first answer with a differently worded prompt is a process step; ask what independent evidence the second check used. If a vendor offers “verified” as a single unqualified badge, request examples of both accepted and rejected cases.
NIST's Secure Software Development Framework provides a common vocabulary for software producers and purchasers to discuss vulnerability identification and response. It does not define a magic automated proof threshold. OWASP ASVS can identify a relevant requirement to investigate, but attaching an ASVS ID to one issue does not verify the whole requirement or establish application compliance.
Prepare a blinded, bounded review set
Choose a repository and immutable revision that your organization is authorized to share or scan. Write down the deployment route, reachable interfaces, protected resources and exclusions. Remove secrets and customer data from a test fixture where possible. Prepare four classes of case before the vendor sees the answer key:
- An independently reviewed issue with a clear code path, such as missing object ownership enforcement on one endpoint.
- A lookalike path where middleware or a data-layer predicate blocks the same attack.
- A path whose outcome depends on unknown deployment configuration or external policy.
- A fixed issue at a later revision, to check whether the tool can explain a changed outcome rather than merely drop an alert.
If you have no historical reviewed issue, use a controlled fixture and label it as such. A planted example tests the vendor's reasoning on that pattern, not its detection rate in your production estate. Ensure the vendor receives the same revision, scope, allowed tools and time budget as any other candidate. Record whether a vendor analyst has coached the tool; assisted and self-service operation are different evaluation modes.
Use the same evidence rubric on every result
For each claim, capture claim → artifact → observed pilot result → limit → decision. The AppSec lead owns the technical verdict; a service owner supplies architecture context. Mark each criterion pass, fail, unknown, or outside declared scope. Keep artifacts with the verdict so another reviewer can reproduce it.
| Verification question | Evidence to request | Passing observation | Unknown or failure signal |
|---|---|---|---|
| Was the right code assessed? | Revision hash, selected files, skipped paths and retrieval limits | Source references resolve to the selected revision and relevant controls were in scope | Line exists, but middleware or configuration was not read |
| Is there an actor-to-effect path? | Entry point, controllable input, authorization boundary, data flow and effect | Reviewer can follow the path without inventing an attacker capability | Pattern match with no route, actor or harmful effect |
| Were defenses and counterexamples checked? | Middleware, policy, validators, queries, call sites and rejected-case record | Vendor explains why a specific control does or does not block the path | “No sanitizer found” after a truncated search |
| What does “verified” mean here? | Actual assessor, evidence and decision record; independent review or authorized runtime evidence when claimed | Status accurately names source-supported or tested behavior | “Exploit confirmed” with only a model narrative |
| Can someone fix and reassess it? | Plain impact, precise conditions, owner, remediation idea, new-revision comparison | Engineer can change the right control and reviewer can assess the new path | Alert disappearance is treated as proof of closure |
Ask the vendor to export one complete result, including the evidence that did not support its initial theory. OWASP WSTG reporting guidance describes technical findings alongside scope and limitations; a polished summary without those fields is difficult to adjudicate. The rubric is intentionally stricter than “the cited line is real” because a real line can appear in a blocked, unreachable or test-only path.
Open a four-case packet and its answer key
This packet, vendor response and adjudication are synthetic. No scanner was run and no request was executed. Meridian Ledger gives the evaluator four labeled source variants while withholding the answer key. The fragments are pseudocode: they expose the relevant control and omit normal application machinery. current_account comes from the trusted authenticated context, not a client-supplied account field.
A / R1: direct path
invoice = invoices.by_id(request.invoice_id)
return invoice.pdf
B / R1: mandatory scoped helper
invoice = invoices.by_id_and_account(request.invoice_id, current_account.id)
if invoice is absent: return not_found
return invoice.pdf
C / R1: external policy dependency
return gateway.deliver_invoice(request.invoice_id, current_account)
# gateway implementation and effective binding are absent from packet
D / R2: corrected version of A
invoice = invoices.by_id(request.invoice_id)
if invoice is absent: return not_found
if invoice.account_id != current_account.id: return not_found
return invoice.pdf
The packet also supplies the relevant contracts. In A, by_id is a global lookup, the PDF return has no later account check, and ordinary authenticated users can enter the source route. In B, by_id_and_account applies both predicates and the route must use it. In D, ownership is immutable during this read and the compared invoice is the one returned; those assumptions keep the exercise from silently depending on a race or a second unchecked lookup. Production deployment mapping is absent for all four cases.
The sealed answer key is:
| Case | Supported conclusion | A conclusion the packet cannot support |
|---|---|---|
| A | Source path supports foreign-invoice disclosure given a valid foreign ID | A live production exploit or an established method of discovering foreign IDs |
| B | The alleged foreign-invoice path is defeated by the compulsory scoped query | Every invoice path in the application is safe |
| C | Unresolved: the gateway could enforce or omit the decisive boundary | Confirmed vulnerable or confirmed safe |
| D | The original A path is corrected in this source variant under the supplied assumptions | The fix is deployed or every alternate route is corrected |
The vendor's fictional response calls A and B “verified,” is silent on C, and marks A “fixed” after seeing D because the alert disappeared. That produces four different adjudications:
- A: useful source-supported result if its references and reasoning match the packet. Relabel any claim of runtime exploitation.
- B: verification error. The helper is mandatory and constrains both invoice and account; citing only the caller omitted decisive counterevidence.
- C: no conclusion. Silence does not reveal whether it was reviewed. Request the scope/evidence state; keep it unassessed or unresolved accordingly.
- D: the answer key supports source closure, but the vendor has not yet demonstrated a verification method. Require it to point to the new comparison, account provenance and return path. A correct label reached only from silence is insufficient evidence for the claimed capability.
This distinguishes correct answer from demonstrated verification process. A product could guess D's outcome correctly while failing to explain why. That matters when the next patch merely renames the unsafe function.
Move the control and see whether the explanation follows it
Supply a second B variant at R3. The account predicate moves from the helper into a compulsory delivery method. The caller again uses by_id, but delivery receives trusted account context and rejects a mismatched invoice before reading its PDF. Include the method and the binding that makes it mandatory.
The correct result remains rejection of the same foreign-invoice claim, for a different control location. If the vendor now reports a vulnerability because the familiar helper disappeared, its rule may recognize syntax without following enforcement. Conversely, if the binding is withheld, the correct result becomes unresolved. This small variation separates reasoning about the path from memorizing the initial fixture.
Ask for a corrected B record that preserves the original allegation, the applicable control evidence and the reason it left the fix queue. Then ask for the same-case rerun to retain that decision unless source, control or scope changes. A global suppression of the rule would be too broad: it could also hide genuinely unsafe A-like paths.
The buyer's conclusion can be narrow:
The candidate found A, but its first verification result missed B's mandatory control and its D closure initially relied on alert disappearance. Continue only as a candidate-discovery tool unless a repeatable follow-up demonstrates counterevidence handling and positive new-revision reasoning. C remains outside any safety claim. The packet establishes no production detection rate or runtime exploit result.
Measure the pilot without inventing certainty
Review all distinct claims in the selected scope if feasible, plus a deliberately chosen sample of unflagged high-risk paths and the prepared reference cases. Record both the numerator and denominator: “two of three prepared cases surfaced,” “one of four distinct alerts rejected after review,” or “six of eight sampled routes fully inspected.” Do not turn a tiny fixture into a universal precision or recall percentage. If the tool groups duplicates, keep the underlying issue count separate from raw alert count.
Ask for a failed scan or partial review example. Did the tool expose an excluded directory, parser failure, exhausted budget or missing context? A “clean” result only describes its assessed scope. Also separate severity from verification. FIRST's CVSS v4 specification separates base, threat and environmental considerations; a high score does not prove that the claimed path exists in this application or set your organization's fix priority.
Make the buying decision
Proceed with a wider pilot when the vendor's states are honest, its evidence is reviewable, counterexamples can overturn claims, and your team can repeat the check. Narrow the use case if it is useful for candidate discovery but cannot substantiate “verified.” Extend only for a named missing test, such as deployment reachability or fix reassessment. Reject a verification promise if the vendor will not distinguish unresolved or rejected claims from confirmed issues. Save one accepted, one rejected and one unknown evidence card as the review record.
Next, use the language and framework coverage checklist to learn where that verification method actually applies, and the pilot decision memo to present the result to a buyer.