Use source review to establish what the selected code and configuration appear to permit; use runtime validation to observe what a particular authorized environment actually does for a defined request or event. Record both against the same revision and deployment. Neither alone answers every security question. A source trace does not show that an attack was executed, while one successful denial in staging does not prove all users, objects and delivery paths are safe. The split matters most when a claim depends on route mapping, identity context, feature flags, external policy, database permissions or a control outside the reviewed source.
OWASP's ASVS 5.0 assessment guidance describes verification as a mix of methods that may require source, documentation, configuration, people and penetration testing. The right question is not which method “wins.” It is which evidence would decide a specific claim, what that evidence can show, and what remains unknown afterward.
Put the claim into two evidence columns
Before selecting a method, write the proposed harmful action in plain language. “An ordinary account user may download another account's invoice” is testable. “Broken access control” is a category and may hide several routes or actors.
| Question | Source review can show | Runtime validation can show |
|---|---|---|
| Which entry and code path exist? | Route registration and calls at a pinned revision, subject to included source and configuration. | A mapped endpoint responds in a specific environment; it may not reveal all calls or hidden paths. |
| Which control is implemented? | The check's conditions, call sites and data dependencies. | The observed result when that control is exercised with chosen roles and objects. |
| Is this deployed? | Build and deployment configuration may support an inference if complete. | The tested artifact and environment can be matched to an actual deployment record. |
| Could a user achieve the harmful effect? | A source-supported path and prerequisites, not an executed result. | An authorized attempt with synthetic data can demonstrate or fail to demonstrate one behavior. |
| Are other paths safe? | Additional review can search for alternate entries. | Each tested path contributes evidence; no finite sample proves absence everywhere. |
Keep those columns separate in the report. If a static analyzer supplies a source-to-sink path, remember that it is a model of possible value movement; CodeQL's dataflow documentation explains that model and its limits. If a tester obtains a response, keep the request, identity, object ownership, build and environment with it. A bare status code can be misleading when a gateway denies a request before the application runs.
Worked case: a 403 does not close every route
Illustrative case, with synthetic accounts and no real customer data: At commit D52F0, a security engineer reviews the GET /invoices/{id}/download handler. The code appears to load the requested invoice ID and return a PDF. Authentication runs, but the engineer cannot find an invoice ownership check in the handler. A static alert proposes cross-account disclosure. The service owner says a gateway rule denies foreign invoices and offers a staging environment for an authorized check.
The engineer's source review establishes a candidate: ordinary users supply the invoice ID; the handler calls the document service; no ownership comparison has yet been found in the examined source path. It does not establish that the gateway is absent or that the production route maps to this handler. The engineer asks for the deployed artifact hash, gateway policy and a route map. The staging owner confirms that the staging artifact corresponds to D52F0 and provisions two synthetic accounts, each with a separate invoice.
In the authorized staging check, account A requests its own invoice and receives it; account A requests account B's invoice and receives 403. This observation defeats a claim that this tested staging route disclosed B's PDF on that attempt. It does not yet explain which layer denied it. A gateway trace and policy show that the gateway checks invoice ownership before forwarding the direct download request. The engineer can reject the direct-download claim for that staging configuration, while noting that the handler itself relies on that external control.
Now the service owner identifies a background export endpoint that bypasses the same gateway route. The export path has not been source reviewed or runtime checked. The correct outcome is not “invoice access is safe.” It is “direct download denied in the tested configuration; export remains an investigation.” OWASP's authorization testing guidance focuses on whether roles and privileges can access restricted functions and resources, a useful frame for choosing the next controlled request.
The evidence ledger makes the separation visible:
| Claim | Source evidence | Runtime/deployment evidence | Decision |
|---|---|---|---|
| A reads B's invoice by direct download. | Handler lacks a local owner check; direct route uses gateway. | Matching staging artifact; A's own invoice succeeds; B's returns 403; gateway policy and trace show owner check. |
Reject disclosure for this tested direct route and configuration; document reliance on gateway. |
| A obtains B's invoice by export. | Export route appears to call the document service; downstream path not fully read. | No matching authorized test or job mapping yet. | Investigation; assign export owner to supply job and delivery evidence. |
Neither row claims a production exploit. The evidence belongs to the stated revision, configuration and synthetic identities. A later policy change or a different environment requires fresh validation.
When source and runtime appear to contradict each other
Now consider a failed first attempt before the matching packet above was obtained. This alternate initial packet shows why the reviewer needed to check build identity and the permitted request:
source under review: D52F0
staging image: image-917, built from D52E2
own-object result: 403
foreign result: 403
gateway policy: not included
This packet does not corroborate the source review. It uses the wrong source revision and its positive control failed. The foreign denial could be a universal authorization failure, an absent fixture or a gateway block. The useful next request is specific: provide the image-to-commit mapping for D52F0, repair the permitted account fixture, and identify which layer handled the request.
The corrected synthetic packet is:
staging image: image-922, built from D52F0
gateway policy: invoice-owner-v6
A reads A invoice: 200, A marker only
A reads B invoice: 403, no invoice fields
gateway trace: foreign request denied before handler
Now the apparent contradiction disappears. The source showed no local owner check; runtime plus policy evidence shows a required external check for this route. Reject the tested direct-disclosure claim at this configuration. Record the dependency on gateway policy v6 so a future route migration does not silently remove the control.
There is a different conflict if the matched runtime returns B's marker while the source appears scoped. Do not call the source check imaginary or the observation invalid by preference. Look for the narrow discrepancy: a legacy route, fallback query, unscoped cache hit, feature flag or different worker artifact. The source guard and the runtime response can both be real if the request did not traverse the assumed path.
Match the conclusion to the strongest complete chain
| Available evidence | Supported conclusion |
|---|---|
| Source path only, external policy unknown | Conditional source-supported concern |
| Foreign denial, own-object request also denied | Object-boundary validation inconclusive |
| Matched build, own succeeds, foreign denied, enforcing policy identified | Tested path defeated under that configuration |
| Matched build returns synthetic foreign marker, source route mapped | Unauthorized behavior demonstrated in the authorized fixture |
| Staging validates, production mapping absent | Staging conclusion only; production applicability unresolved |
The fourth row is not proof of historical customer access. That requires a separate investigation. The third row is not proof that export or alternate delivery is safe. Both require keeping the scope in the sentence.
A runtime request should therefore answer a question that source cannot settle, not serve as a ritual before every conclusion. If mandatory source controls already defeat the exact mechanism, record that counterevidence. If deployment is decisive, obtain deployment evidence. If the policy itself is unknown, ask its owner before calling intended sharing a vulnerability.
For the operational handoff—what to send the service owner, how to reject an incomplete returned packet, and how to write the final bounded finding—use the source-only finding validation request. This guide establishes how the two evidence types relate; that guide supplies the completed exchange.
Decide which changes invalidate the evidence
The synthetic image-922 packet is useful because it binds the observed behavior to a specific route, build, policy and fixture. A later result should identify which of those inputs changed. If the gateway owner replaces invoice-owner-v6, the old denial no longer establishes the new policy's behavior. If only an unrelated display string changes, the reviewer can inspect that difference and decide whether the relevant control and route mapping remain unchanged. Neither “any commit invalidates everything” nor “the old check passed once” is an adequate reuse policy.
Keep the reason for reusing evidence alongside the evidence itself. For example: the handler and ownership policy are unchanged, the new image is mapped to the reviewed source, and the selected route still uses the same enforcement layer. That is a bounded argument about the relevant change, not a guarantee about the entire deployment. If a required mapping cannot be established, state that uncertainty and obtain the missing artifact before extending a production claim.
A fresh response also needs interpretation. Suppose a new run again returns 403 for B's invoice, but this time A's session has expired. The result cannot replace the earlier ownership observation simply because the status code matches. Restore the intended actor and permitted operation before checking that boundary. Similarly, if the fixture no longer contains B's invoice, a not-found result does not demonstrate that access to an existing foreign invoice was denied. The setup is part of the evidence, not disposable preparation.
Be careful when transferring a staging result into a customer-facing statement. Record the verified similarities and the material differences between staging and production: deployed artifacts, route registration, policy version, relevant feature flags and service identities. The example's staging observation can inform the review without claiming a production request was attempted. If production mapping remains unresolved, preserve the staging conclusion and assign the mapping question separately.
This approach makes validation economical. Revisit the fact that changed and the behavior that depends on it. Reuse still-applicable source evidence with an explicit rationale, and reserve a new authorized observation for a question that requires it. The goal is a coherent evidence chain for the selected decision, not a ritual sequence of scanner and runtime reports.
When validation cannot run
Sometimes there is no authorized test environment, the necessary role cannot be provisioned, or a third-party dependency cannot be safely exercised. Preserve any conclusion already established from source and list the unmet runtime prerequisites. Qualify deployment or observed-behavior claims that depend on those facts; lack of a test environment does not make every source-level conclusion inconclusive. A high-impact plausible exposure may justify a temporary owner-led mitigation while evidence is gathered, but mitigation does not prove the original path exploitable. Conversely, a failed test because the service was unavailable is inconclusive, not a security pass. OWASP's reporting guidance asks for results that technical and executive readers can understand; the observed boundary belongs beside the result.
For the next source finding, write one runtime question and its required artifact mapping before asking anyone to test. Use the dataflow-trace guide to sharpen the source claim and the coverage-report template to disclose which validation paths were actually completed.