Add an authorized runtime assessment when the decision depends on behavior that code scanning cannot establish: deployed identity and access flows, service interactions, configuration, business workflows or attacker reach in a running environment. Choose an external pentest when you need independent assurance, specialist testing capability or capacity your team cannot supply. A bounded internal check may settle a narrower technical question. A source review can reveal a plausible unsafe path and guide a fix, but it does not prove the production route, permissions or gateway behave as expected. A pentest is also appropriate when a customer, contract, regulator or internal policy explicitly requires independent testing; confirm the exact requirement rather than assuming any scanner report satisfies it.
The scope changes with the question. A targeted manual review may resolve one suspected path faster than a broad external engagement. A full pentest may still leave untested areas. NIST SP 800-115 describes technical assessment methods and the need to plan and conduct tests against defined objectives. OWASP's Web Security Testing Guide frames web testing as a set of methods across the application life cycle. Use the right method for the remaining uncertainty and document its limit.
Separate the question each method can answer
Code scanning is valuable when the team needs repeatable inspection of selected source revisions: risky data flows, missing checks, known dependency issues or suspicious configuration. Depending on the tool, it may offer rule matches, source traces or assisted review. Ask what files and controls were actually examined. A cited code path may support a claim about how source is written; deployment flags, service identity and live authorizer behavior might still change its impact.
A penetration test is an authorized, time-bounded attempt to assess a running target under agreed rules. Testers can inspect reachable interfaces, use granted test accounts, challenge authentication and authorization workflows, and compare expected with observed behavior. They cannot make every possible path appear within a limited scope and time. OWASP's reporting guidance explicitly calls for scope and limitations; its example disclaimer describes a point-in-time assessment rather than a warranty that all possible issues were found.
Neither method should be reduced to an alert count. A pentest report with a clear reproduction, affected role, business effect and fix guidance may be more useful for one release decision than hundreds of scanner alerts. Conversely, a source issue found early and fixed before deployment may not need an external test for that exact defect, although other runtime questions may remain.
Use a trigger table, then choose a scope
| Situation | What remains uncertain after code scanning? | Best next action |
|---|---|---|
| New internet-facing application or major release | Real reachable attack surface and cross-service behavior | Scope an external or independent runtime test around critical workflows |
| Suspected broken object authorization | Whether two real roles or tenants can cross the boundary in the deployed route | Controlled test with synthetic identities; widen if the pattern is systemic |
| Identity-provider, gateway or cloud-policy change | Effective configuration and interactions absent from source | Configuration review plus targeted runtime testing |
| High-value multi-step business action | Abuse sequence, race conditions, workflow state and compensation | Human-led business-logic testing with service-owner context |
| Known dependency or code pattern | Whether the vulnerable component is reachable and exposed | Confirm deployment and attack preconditions; targeted test if decision depends on runtime |
| Contractual “independent pentest” requirement | Required independence, frequency, target and reporting format | Read the actual contract or policy and commission matching work |
| Routine code change in a well-tested path | Whether the fix addresses the source issue | Review new revision and focused validation; schedule broader tests by risk |
This is a decision aid, not a fixed frequency mandate. NIST's Secure Software Development Framework describes a set of practices for finding and responding to vulnerabilities through development. Use code review, dynamic tests and independent assessment together where the risk warrants them. OWASP ASVS can help turn an application control into a test objective, but naming an ASVS requirement does not prove it passed.
Illustrative decision: the gateway may change the result
Illustrative pilot; invented company, application and finding. Elm Records has a B2B document service. A source scanner flags a route that appears to return a document by ID without checking its tenant owner. The development team says a gateway injects a tenant claim and a downstream query helper restricts documents by that claim. The scanner's source snapshot contains the route and helper, but not the deployed gateway policy or identity-provider mapping. No authorized runtime test has been performed.
The CTO should not call the scanner result a confirmed cross-tenant breach, and should not dismiss it because someone mentioned a gateway. The next decision is small and concrete: ask an independent reviewer to trace the helper at the pinned revision, inspect the deployed gateway policy, and, in a permitted test environment, attempt the same request with two synthetic tenant accounts. If the route still exposes a document, treat it as a confirmed test-environment result and follow incident and fix procedures appropriate to the real exposure. If the control holds, document the counterevidence and investigate whether other routes bypass it. If the policy cannot be inspected or the test environment differs materially from production, mark the result unresolved.
Elm also plans a new SSO integration and an export workflow involving several services. Those changes create broader questions than this one code path. The CTO commissions an external pentest focused on account switching, authorization on document and export APIs, session transitions and cross-service role propagation. The engagement uses synthetic accounts and records the environment version. It does not authorize testing arbitrary customer accounts or production disruption.
The filled decision card reads:
Claim: source indicates a possible document ownership bypass. Available evidence: pinned source trace. Required evidence: selected gateway policy, identity-provider mapping and deployment artifacts; request records from future authorized synthetic checks. Observed: route lacks a local ownership check; downstream helper appears to apply tenant scope; deployed gateway mapping unverified. Limit: no runtime behavior observed. Decision: targeted independent test now; external pentest for the SSO/export release, scoped to identity and multi-service authorization. Owner: CTO authorizes rules of engagement; AppSec lead supplies code context; external tester reports observed behavior and limits.
This is an invented planning case. It is not evidence that the hypothetical application has an exploitable flaw.
Turn the SSO/export concern into a filled scope brief
The next artifact is an engagement specification, not permission for anyone reading this article to test a target. Elm's actual authorization would have to identify real assets, owners and agreed methods separately. The example below remains fictional and uses synthetic accounts and data.
| Scope field | Elm's proposed engagement |
|---|---|
| Release decision | Whether the new SSO and multi-service export workflow meets the selected account and role boundaries before release |
| Target | Elm-controlled staging portal, document API and export worker at release candidate RC-8; host inventory and artifact identifiers in the private annex |
| Identities | A-only viewer, A administrator, B-only viewer, and one user who is administrator in A but viewer in B |
| Data | Synthetic documents D-A and D-B, plus a synthetic export generated from each; no customer documents |
| Main workflows | Sign-in, account switching, role changes, export request, queued execution, export download, logout and subsequent session use |
| Required control questions | Is current account derived from authorized membership? Does switching from A to B retain A's administrative ability? Do queued work and download apply the intended account/resource policy? What happens after role removal? |
| Evidence to be supplied | Source hypotheses, expected authorization matrix, gateway policy, IdP mapping, worker launch/configuration record and known environment differences |
| Exclusions | Customer production accounts; unrelated billing features; attacks against the identity provider's own infrastructure; availability stress beyond the agreed safe limits |
| Stop/escalation | Stop on unexpected access to non-synthetic data or a material service impact; contact Elm's named incident coordinator through the engagement channel |
| Deliverables | Workflow-by-role coverage, observed outcomes, unresolved controls, version-bound findings, remediation suggestions and a defined retest allowance |
The dual-member identity is essential. Testing only A-only and B-only users can check a basic tenant boundary but cannot exercise privilege carried across an account switch. A successful login also says little about a queued export that executes after the user's role changes.
Write expected outcomes before execution. For this proposed policy, the dual-member user may administer A, may view authorized B documents after switching to B, and may not perform B administrator actions. An export request records its authorized account and resource scope; later execution and download must meet the specific current-permission rules Elm declares. If the business intentionally allows already-authorized queued work to finish after a role change, that exception must be explicit so the tester does not invent the policy.
Compare proposals against the unanswered question
One assessor offers a targeted review of the original document lookup and its gateway binding. That can settle the immediate scanner dispute. Its report should state the specific route, identities, configuration and limits. It does not cover the new SSO transitions or queued exports merely because they belong to the same application.
A second assessor proposes a broader “SSO and API pentest,” but its draft scope lists login and direct document requests only. It omits account switching, the dual-member role transition and export download after queued execution. The attractive title does not satisfy Elm's release question.
Elm's response is concrete:
Add the dual-member A-admin/B-viewer identity, account switching in both directions, role removal during the export lifecycle, and authorization at export download. Show the time and access assumptions for those workflows and which remain excluded. Preserve a targeted retest for any supported finding after the corrected release candidate is supplied. If the offered time cannot cover the additions, separate the gateway dispute from the release engagement and name the deferred release questions.
Now the buyer can compare actual scope and capacity instead of page counts or a generic number of testing days. The targeted assessment may still proceed first because it resolves a current blocker; it does not cancel the broader release work.
Make the retest answer the original question
Suppose the external assessment later reports a supported account-switch finding in the authorized staging scope. This hypothetical does not assert such a finding exists. Its repair check should preserve the original roles, transition and expected denial, use the corrected artifact, and include legitimate B-viewer behavior so a blanket outage is not mistaken for a correct boundary.
The retest record should say whether the original behavior is prevented, persists or remains inconclusive, with the changed revision and relevant configuration. If production uses a different gateway or IdP mapping, a staging result leaves that difference unresolved. The release owner must reconcile it or narrow the conclusion; no amount of confidence in the source scanner supplies the missing deployment fact.
Protect the evidence under the engagement's agreed handling rules. The public executive summary can state the harmful action, affected workflow and scope without exposing source, credentials or reusable attack details. The technical annex needs enough authorized detail for the owner to reproduce and repair the issue.
Apply the evidence rubric and make the decision
Capture claim → artifact → observed result → limit → decision for the combined assessment. The code scanner's “possible bypass” and the pentester's “observed cross-tenant response in staging” are different claims. Mark each supported, rejected or unresolved within its own scope. Keep severity, practical difficulty and business priority separate. If a retest shows the issue no longer occurs, tie that outcome to the new deployment revision and test conditions, not simply the disappearance of an alert.
Commission the external test when the remaining runtime or independent-assurance question is material to a release or obligation. Use targeted validation when one bounded claim is the immediate blocker. Continue source review for early, repeatable feedback between independent assessments. Take the scope and residual uncertainty into the security-tool pilot decision memo; for the source finding itself, use the verification-claim checklist.