How to verify an AI-generated vulnerability report

Treat the report as a testable claim, not as a verdict. Pin the exact repository revision, reproduce every cited source line, and write the alleged path as actor → controlled input → entry point → security control → harmful effect. Search for controls and facts that would defeat that path. Check whether the route is actually deployed and whether the affected asset has the claimed business meaning. Record one of three outcomes: supported for this scope, rejected with counterevidence, or unresolved with a named missing fact. Source review can support an attack-path conclusion; it does not show that an exploit was run.

This checklist is for a security engineer deciding whether an AI-written report deserves a developer ticket. It changes if the suspected consequence demands immediate containment: notify the incident or service owner of a suspected exposure while verification continues. Do not promote the claim merely because its severity label is high. OWASP's LLM misinformation guidance warns that plausible output can be false or unsupported and calls for cross-checking and human oversight.

Use this checklist in order

The checklist follows the work described in OWASP's secure code review guidance: understand the architecture and business rule, trace data flow, inspect authorization, and review configuration. It is intentionally narrower than a general code-review plan. The output is a decision on one report.

Split one confident paragraph into three reviewable claims

Synthetic report and source; no model run or runtime request occurred. At 41d0b6e, an AI writes:

Any signed-in member can call the contacts export, the query returns every customer's contacts, and the resulting file is publicly downloadable.

The report cites a real-looking handler. That paragraph contains three claims with different decisive evidence. Open the small source packet:

router.mount("/admin", require_current_account_admin, admin_routes)
admin_routes.get("/contacts/export", export_contacts)

export_contacts(context):
    rows = contact_store.for_account(context.account).all()
    artifact = write_private_csv(rows)
    return delivery.issue_link(artifact)

Next inspect the router guard: it reads current server-side membership and requires account-admin capability before the handler. That rejects the first claim for the bound HTTP route. It does not prove the second claim false, because an administrator might still reach a global query.

Open for_account: in this synthetic continuation its all() query includes a bound account predicate derived from verified context. That rejects the “every customer's contacts” claim for this store path. The name all() was misleading without its receiver; the actual query settles it.

The third claim remains unanswered. write_private_csv and issue_link are separate implementations. A private storage object could be delivered through a bearer link; a link could also lead to an endpoint that rechecks current access. A bearer link permits access by its holder, which is not by itself proof of public discoverability or a policy violation. Check who may obtain or share it, its lifetime and intended audience, and whether those properties match the export policy. Request the storage policy and delivery code as well.

Produce a corrected report, not a binary grade

The useful disposition is:

Two claims in candidate AI-7 are rejected at 41d0b6e: ordinary-member reachability and global contact selection. Exact router and query evidence are retained. The artifact-delivery claim remains unresolved because delivery.issue_link and its policy were not acquired. Delivery owner must supply those artifacts before any exposure conclusion. No runtime behavior was validated.

Do not label the entire paragraph “false positive” and delete the delivery question. Do not retain “critical global export” because one branch remains uncertain. The approved work is a bounded delivery investigation, not a speculative rewrite of a correctly scoped query.

Now add a complication: a scheduled worker calls export_contacts directly. The HTTP guard no longer proves the worker's authority. Trace that caller as a new path with its own delegation, account provenance and recipient policy. Reuse the store evidence where applicable, but do not reuse a router check that the worker never traverses.

This approach makes review faster by identifying the exact next artifact. The source packet's missing delivery implementation is a real boundary; asking the model to rephrase the paragraph is not progress. OWASP's authorization guidance supports server-side resource decisions, but the reviewer still has to demonstrate where each decision occurs in this application.

Find the hidden assumption in each sentence

AI reports often compress several premises into one sentence. Break them apart before judging severity:

• “An attacker can call this function.” From where? An HTTP route, a queue message, an authenticated internal service, or only an unused helper? Show the entry and the actor's current permission.
• “The input is attacker-controlled.” Follow its origin and validation. A path variable may be supplied by a user, while an internal ID might be derived from a trusted server-side selection. Do not assume either without tracing it.
• “There is no authorization.” Search the route registration, middleware, service and data access layer. Verify that a control is on the alleged path and checks the specific resource, not merely login. A group check may permit use of a feature while still failing object ownership.
• “Sensitive data is returned.” Trace the result to a response, file, message, log or other recipient. A query result that is discarded does not establish the claimed disclosure; a filtered projection may change the impact.
• “This affects production.” Check the build, configuration and feature state. A test-only route or disabled feature changes applicability; an unknown deployment state remains unknown.

For web applications, OWASP ASVS is a source of versioned control questions. Cite a requirement only if you have checked the relevant control and scope. An ASVS tag attached by an AI tool is neither proof that the issue exists nor proof that the rest of the standard was covered.

Decide what the report becomes

Supported: An independent reviewer can trace the actor, entry, failed or missing control, and harmful effect at a pinned revision; relevant countercontrols and deployment assumptions have been addressed. Record any remaining prerequisites. In a process with approval, publish only after that gate. “Source-supported” is the precise phrase if no authorized runtime validation occurred.

Rejected: A cited, applicable control defeats the precise claim, the purported path is unreachable in the selected build, or the report targets the wrong asset. Preserve the AI report and counterevidence so a rerun does not recreate the same work. A rejection for one route is not a blanket assurance about every caller.

Unresolved: Source or deployment evidence is missing, reviewers disagree on a material premise, or the available time cannot settle the claim. Record the missing artifact, owner, deadline, and what decision it would change. If potential impact is serious, escalate the suspected risk for containment without changing the evidence status.

If the claim is supported, write a concise handoff with plain problem, affected asset, actor, conditions, exact source evidence, practical difficulty, proposed control, and fix acceptance evidence. OWASP's testing-guide reporting chapter emphasizes scope, clear impact, technical detail, and limitations. Keep raw model prose out of the ticket unless it has survived factual review. When a patch lands, validate the behavior at a new immutable revision; the disappearance of the AI alert is not enough.

Reconcile the corrected report with the original evidence

A review can be technically sound and still confuse the next person if the corrected conclusion loses its relationship to the original candidate. Keep the original report identifier, the exact sentences under review, and a short decision for each. This matters particularly when an alert was already copied into a ticket, a release discussion, or an incident channel. A correction that exists only in the verifier's notebook will not necessarily correct the team's understanding.

For the contacts export, attach the router evidence to the ordinary-member allegation and the account predicate to the global-query allegation. Attach the delivery evidence request to the third sentence. Give each artifact a stable location in the team's restricted review record. The developer-facing summary can be brief, but another authorized reviewer should be able to reconstruct why the original severity and proposed fix changed.

Suppose the original recommendation was “add an administrator check and account filter.” The completed source review shows both already exist. Explicitly withdraw that proposed patch for the HTTP path. Otherwise an engineer may implement redundant checks while the unresolved delivery behavior receives no attention. The new task should ask the delivery owner to establish who can obtain and use the export link, with the relevant policy and implementation named.

A useful correction note for this synthetic case reads:

supersedes: candidate AI-7 paragraph 1 and its proposed route patch
retained evidence: original report; pinned router and store ranges
withdrawn work: duplicate route-admin and account-query checks
remaining question: intended audience and enforcement of delivery link
next artifact: delivery implementation plus export-sharing policy
review boundary: HTTP export only; scheduled caller remains separate

If the owner supplies today's implementation rather than the report's revision, record that difference before continuing. It may settle the current repair decision while leaving the historical allegation unresolved. Do not silently use the new code to declare the old report false. Conversely, an old source defect does not establish that the currently deployed export still has it.

Finally, check that the revised report preserves the valid observation that led to the investigation. The handler really did delegate authorization and query construction to other layers. That is useful context for later reviewers. Rejection should teach them where the control lives, rather than merely telling them the AI was wrong.

For the next AI report, use the seven checks to identify the decisive evidence and strongest counterargument. Give a developer a precise investigation or repair task rather than making them reconstruct the raw report. If this one-report process is new to the team, start with the broader AI code-review workflow. If the hard part is checking exact lines across files, continue with how to check an AI attack path against exact source lines.