How to describe AI scan limits to developers

Tell developers which revision and paths were examined, what kind of review was done, which claims passed verification, what remains unknown, and what they should do next. Put that answer beside the findings, not in a distant methodology appendix. If the run was targeted or interrupted, say so before any “no issues found” sentence. A source-only AI review can support a code-path claim, but it cannot by itself show that an exploit was executed or that the deployed application is safe. An inventoried file, an indexed file and a file actually examined for a specific issue are different states.

The useful level of detail changes with the handoff. A developer fixing one issue needs the exact path and missing check. An engineering manager planning a sprint also needs skipped areas and owners. Neither needs an unqualified percentage that treats downloaded files as fully reviewed. OWASP's Web Security Testing Guide reporting chapter makes scope, method and limitations part of a useful report, and those facts matter just as much in an internal AI-assisted assessment.

Use five plain-language fields

Use a stable source reference for any cited line. GitHub explains how to link a file and line range at a specific commit. The report should also say whether its assessment covered that whole call path or only an excerpt. A line that exists may still be unreachable or protected elsewhere.

Filled example: a targeted billing review

Illustrative example only: Alder Billing requests an AI-assisted review of invoice download and refund approval at invented revision 789beef. Its monorepo contains nine services. The run inventories all nine, indexes four selected services, and inspects parts of both named flows across three services. It ends on budget after checking the invoice route and part of the refund authorization chain. One AI candidate about invoice access is rejected after a reviewer finds a required account-bound storage query. A second candidate about refund approval remains unresolved because the policy service was outside the acquired scope. There is no verified finding from this run. Nothing was executed against a running service.

Here is the message the security engineer sends to the relevant developers:

Assessment of revision 789beef — targeted source review, partial result

Question: Can an ordinary billing user download another account's invoice or approve a refund?

Examined: The invoice route, its authentication and authorization middleware, account-bound storage query, refund entry route and visible caller checks. These paths span three services.

Inventory only: Six other services were listed but not examined for these claims; one of the four selected services was indexed but not substantively reviewed.

Decision: The invoice lead was rejected because the storage query binds the requested invoice to the current account on the traced route. The refund lead remains an investigation. No issue was approved as a verified finding in this assessment.

Missing fact: The refund policy service implementation and deployed route configuration were not available in this run. We cannot say whether its downstream permission check allows or blocks the proposed path.

Next action: The platform owner will provide the policy service at a pinned revision and the deployment mapping. Security will then finish the refund path review. No refund defect has been established for a fix ticket; the developer’s current task is to supply the missing policy evidence. A separately justified precautionary change can proceed under the normal release process.

Method boundary: This was source inspection. No test account made a refund request, no exploit was performed, and no claim is made about other services or all runtime controls.

This note is deliberately more specific than “scan passed with zero findings.” It avoids sending a developer a speculative fix, while still naming the artifact that would let the team decide. A rejected candidate is not deleted from the team's review record; the account-bound query and revision explain why it was rejected.

Say the same truth at three useful lengths

The detailed note above is useful for the assessment record. A pull request needs a smaller version that preserves the decision boundary:

AI-assisted review: partial at 789beef. Invoice-download claim rejected by the required account-scoped query. Refund approval remains unresolved because the policy service and deployment map were unavailable. No verified issue was approved; this is not a security pass for the refund change. Platform owns the missing evidence; the release owner applies the existing merge policy.

That last sentence prevents the scanner from inventing release authority. Whether an unresolved material question blocks this particular merge depends on the team's defined policy and risk owner. “No approved findings” is an evidence state, not an automatic merge instruction.

A finding-level note should be even narrower:

Refund lead RF-2 — investigation. Visible caller checks do not settle the downstream approval rule. Needed: pinned policy-service implementation plus deployed route binding. No repair requested yet. If the current release depends on this permission boundary, the release owner needs an explicit decision before relying on the incomplete review.

The restricted evidence record preserves the full accounting:

revision: 789beef
inventory: 9 services
indexed: 4 selected services
examined: named invoice and partial refund paths across 3 services
other services: 6 not examined for these claims
candidates: 2 = 1 rejected + 1 unresolved + 0 approved
stop reason: budget
runtime validation: none

The indexed fourth service is among the six not substantively examined. It must not be added to them as a seventh untouched service. Both flows received attention, but only the invoice claim reached a disposition. That distinction matters more than a percentage of folders.

Update the note when the missing evidence arrives

Suppose the policy service later shows a mandatory current approver grant, and the deployment map establishes that this service handles the route. The new source decision can reject RF-2 with that evidence; record a new dated entry linked to the original partial run. Do not rewrite the old run as though it examined the service before stopping.

If instead the service confirms the missing approval check, create the supported finding through the review gate and send developers the exact rule and fix acceptance condition. The PR summary changes because new evidence changed the state, not because the model became more confident.

If the supplied service is from a different revision, establish which service version the assessed route actually uses. Independently deployed services need not share a commit ID, but their version relationship must be evidenced. Without that mapping, the annex does not complete the original assessment; review a newer deployment as a separately dated scope. GitHub's permanent-link guidance helps preserve source identity; WSTG reporting provides the wider scope-and-limit discipline.

The point of three lengths is usability. The developer should understand the next action from the short note, and a reviewer should be able to reconstruct its basis from the record. Neither audience should have to infer that “passed” really meant “stopped before the decisive service.” All counts and events here remain illustrative, with no scan or request performed.

Write the limit as a dependency the developer can resolve

Alder's refund note becomes more useful when it identifies the smallest missing artifact, the person who can supply it, and the decision it affects. “Runtime not tested” is true but broad. “We do not know which policy-service version handles refund approval in the assessed deployment” tells the platform owner exactly what to retrieve. The distinction reduces unnecessary requests for a complete environment when a deployment mapping may settle the next source question.

Use the same precision for source gaps. If a function was truncated after the initial permission check, name the unread continuation rather than saying the whole service is unavailable. If a generated route map was excluded, distinguish that from the hand-written handler that was examined. If a budget stopped the review before the worker, record that as unreviewed work, not a parser limitation. Different causes lead to different remedies.

The developer should be able to answer three questions after reading the short note: What can I rely on? What cannot I rely on? What is the next useful action? For the invoice path, the answer is a rejection of one cross-account claim under the traced account-bound query. It is not a general assurance about invoice exports, cached downloads, or other callers. For the refund path, the current action is an evidence handoff, possibly accompanied by a release-owner decision under existing policy.

Handle an interrupted rerun without losing the first result

Suppose a continuation acquires the policy service but stops before completing its call chain. The report should now say that the service source was obtained and partially examined; the decisive authorization question remains unresolved. Do not leave the old “service unavailable” sentence unchanged, because it would misdirect the next owner. Do not change it to “policy reviewed” merely because the files arrived.

A small dated addendum can express the difference:

RF-2 continuation: policy service acquired at its own pinned revision
new evidence: entry function and first permission lookup inspected
still missing: downstream decision branch and deployed version mapping
current action: finish that branch, then reconcile deployment identity
effect on prior invoice conclusion: none identified

This synthetic addendum does not invent a new finding. It advances the evidence state while preserving the limits of the assessment. If a later continuation establishes the policy, link the final decision to both partial records so reviewers can see how the conclusion was reached.

Keep the next action visible when the note is copied

Assessment notes often travel into tickets and release summaries. A copied sentence such as “zero verified findings” can lose the adjoining statement that refund approval was incomplete. Put the important qualifier in the same sentence: “No verified finding was approved in this partial invoice/refund review; refund authorization remains unresolved.” This is an editorial design choice that makes the summary harder to misunderstand.

Avoid burying all unknowns in one generic disclaimer. Attach each unknown to the claim it limits. The missing policy service affects refund approval; it does not erase the invoice query evidence. Likewise, the invoice rejection does not settle the refund path. Developers can act on a bounded conclusion when its dependency is clear, even when the wider assessment remains unfinished.

For the next developer handoff, copy the five fields and the stopped-run sentence, then ask a developer to explain back what was reviewed and what remains unknown. If the assessment needs a fuller coverage ledger, use the scan coverage report guide. For source-supported AI claims, use the exact-line attack-path check.