What to ask in an AI code scanning vendor demo

Use the demo to see whether one finding can survive a challenge. Ask the vendor to connect the reported harm to the source it examined, show the control that could defeat the claim, and expose what happens when evidence is missing. Then inspect a changed-revision result and the permissions and data flow needed to produce it.

A forty-five-minute meeting cannot establish general accuracy or annual cost. It can produce a short evidence log and a clear decision about whether a pilot is worthwhile. NIST's Generative AI Profile includes confabulation and information-security risks; a fluent explanation is a reason to inspect the supporting artifacts, not a verdict about the product.

Send the fixture and the agenda before the meeting

Use a non-sensitive, authorized example until private-source access has been approved. The following meeting, vendor replies and artifacts are fictional. No product was evaluated or attack executed.

Northstar Ledger supplies a pinned sample API with account-specific statements. The sample includes a route handler, storage wrapper and separately versioned gateway policy. The buyer asks for a normal result, an incomplete result and a new-revision reassessment. It does not require the vendor to implement missing features during the call; unavailable behavior should be recorded as unavailable.

Send this brief:

Please show one statement-access finding from source identity to disposition, including examined scope and counterevidence. Bring the effective policy binding if your claim depends on it. Show how a developer challenge changes the state, what a partial run displays, and how a changed revision is reassessed. Reserve time for the actual repository permissions, processing/logging/retention diagram and itemized run cost. Mark prerecorded, live, unavailable and planned behavior separately.

The agenda is deliberately constrained:

Prompt 1: “Show how the selected input reaches the harmful effect”

Ask the vendor to name the actor, input, entry, control and effect for one finding. Open the actual source at the immutable revision. “It found an IDOR” is not the answer. Can the developer see where the account relationship should be enforced and why the reviewed path fails it?

Vendor: “The handler accepts a statement ID and the storage call retrieves the file, so another account's statement is exposed.”

Buyer follow-up: “Which delivery and authorization code did this run examine? Please show that scope and the response path.”

In the fictional demo, the handler and storage references are present, but the gateway policy repository is excluded. The finding has not yet accounted for a control that could defeat it. The buyer records evidence incomplete, even if the explanation has a confidence badge. A source citation establishes where text came from; it does not establish the entire claim.

Prompt 2: “Here is counterevidence; show what changes”

The buyer supplies a versioned policy that appears to bind the statement to the active account. A screenshot of a policy editor would be insufficient by itself; the important next artifact is whether this deployed route must use that policy.

Vendor: “Our second AI reviewer agrees with the first, so the finding is verified.”

Buyer follow-up: “Did either reviewer receive this policy and its route binding? Agreement on the same incomplete packet does not settle the missing control.”

The appropriate interim state is unresolved. In the exercise's completed branch, the supplied binding establishes that every assessed statement request passes the compulsory account check before delivery. The claim is rejected for that route and revision. Ask the vendor to show the rejected record, the evidence that changed it, and where it appears on the next run.

The useful feature is not willingness to accept every developer objection. If the policy belongs to another route, rejection would be wrong. The workflow must let a reviewer compare the actual applicability and retain disagreement until it is settled. Ask who approves that decision: a model, vendor analyst, customer reviewer or a combination. “AI verified” leaves that responsibility unclear.

Prompt 3: “Show an incomplete run and a changed revision”

Use a safe missing-file or unsupported-parser scenario. Ask what the developer, CI integration and final report each display. A result can preserve supported findings while stating that a selected component was not assessed. A green status needs an explicit meaning; “no findings” on an excluded component is not a completed review.

Then open a supplied new revision that changes a separate known unsafe fixture path. Ask for the original claim, changed evidence and a fixed/still-present/inconclusive outcome.

Vendor: “The alert disappeared on the rerun, so the fix worked.”

Buyer follow-up: “Did the rerun include the changed path and its controls? Show the new source identity and the reason the original behavior is prevented.”

In Northstar's exercise, the changed service was excluded by a configuration error. The buyer records reassessment inconclusive. A planned future capability cannot satisfy this demonstration gate. The follow-up is a complete result on the specified revision, not another screenshot of an empty dashboard.

Prompt 4: “Open the exact access and processing record”

Inspect repository selection and each requested permission. GitHub's app-permission guidance recommends minimum necessary permissions; installation settings allow repository access to be reviewed. “Read-only source access” can coexist with separate write permissions for checks or issues, so request the full grant and its purpose.

Follow source into analysis, model input, logs, findings, support tools and backups. Ask what untrusted repository text can induce the agent to do: execute a command, fetch a URL, change tools or send data elsewhere. Request the enforced action boundary and a bounded demonstration. An assurance of being “prompt-injection proof” is not an artifact.

If a detailed retention answer needs a specialist, record that fact and keep private-source approval pending. The meeting can still continue using the non-sensitive fixture. Use the regional pilot dossier to examine the returned evidence.

Prompt 5: “What work and cost produced this result?”

Separate scanner runtime from human help. Ask whether a vendor analyst selected the example, tuned rules, wrote the explanation or resolved the policy question. Record setup time, active reviewer minutes, retries, model/run charges and what happens at the spend limit. A scripted five-minute demo cannot establish your team's future triage cost.

For Northstar, the final log might be:

Each row has an observed outcome and an evidence status. A repeatedly demonstrated failure is strong evidence of a weakness, not a high score. Keep both dimensions separate.

Keep the demonstration reproducible enough to challenge

Before the call, Northstar freezes the sample revision and writes the expected account boundary in a separate note. The note says which synthetic user may receive which statement; it does not supply a preferred scanner verdict. This prevents the meeting from drifting into a debate about an unstated business rule. If the vendor asks for an additional policy file, record when it entered the packet and which conclusion depends on it.

A prepared example is acceptable for exploring a product's workflow, provided preparation is disclosed. It answers a narrower question than an unfamiliar example processed without assistance. Northstar therefore records three facts beside each demonstration: what input was available, what work occurred before the call, and who made the final disposition. A vendor analyst's useful investigation may be part of the service being purchased. The buyer needs to know whether that work is included, separately charged or unavailable during ordinary use.

Do not demand an improvised attack against a live system to make the meeting feel convincing. For the statement example, the immediate question is whether the product can represent a missing control and revise a conclusion when the required evidence arrives. The synthetic packet is sufficient to inspect that behavior. Runtime validation, if needed later, belongs in a separately authorized scope with explicit identities, targets and handling rules.

Northstar also preserves the ordinary developer view. If the presenter switches to an internal analyst console to explain an exclusion, ask whether a customer reviewer receives the same scope information. An internal capability can be useful, but it does not establish that the developer's CI result or exported report communicates the limit. Record the interface actually demonstrated and the access level required to see it.

Turn unanswered demo questions into a finite follow-up

The meeting's evidence log should distinguish a demonstrated defect from an unanswered request. Northstar observed the top summary omit an exclusion; that is a specific reporting weakness in the shown version. Log retention was not explained; that is an unknown with a named evidence request. Combining both under “needs clarification” would obscure which behavior must change and which fact must be supplied.

Give each follow-up an acceptance condition. For the summary problem, the returned artifact must show the partial state in the view the intended reviewer uses, tied to the same kind of excluded component. For reassessment, the vendor must supply the specified new revision, include the changed path, and explain the original claim's disposition. For retention, the platform owner needs a written record covering the source-bearing log category and selected configuration. A slide saying the concerns are resolved does not satisfy these different requests.

Northstar sets one follow-up meeting and keeps the same sample. If the vendor offers a different polished finding instead, preserve it as additional information without closing the original gaps. At the follow-up, choose whether the evidence supports a pilot, a narrower evaluation or no further work. This avoids repeatedly spending buyer and vendor time on an expanding demo that never reaches a decision. The resulting record is small enough to compare with another candidate without pretending that either meeting measured general detection accuracy.

Use role-specific follow-ups only when they change the decision

A developer should ask whether the proposed correction preserves the intended business rule. Platform should ask how revocation and partial CI runs behave. Procurement should ask for the exact data-flow and commercial commitments. AppSec should ask which required framework paths remain unsupported; ASVS can help organize control questions but is not a scanner certification.

Close by reading back what was demonstrated, what remains asserted, and which unanswered item blocks the next step. Northstar can schedule a bounded follow-up on partial reporting and reassessment while retaining its private-source gate. It has learned something concrete even if the decision is not to pilot.

For a real comparison after the demo, use the two-week pilot protocol. Keep this meeting focused on obtaining the evidence that makes that larger commitment worthwhile.