Ask a vendor to show what the system was authorized to touch, what it actually did, what evidence supports each result, and which human decisions remain. “Autonomous pentesting” may mean source analysis, generated attack plans, requests against a running test environment, or a mixture. These are different activities and support different conclusions. A complete source trace can establish a code defect and justify repair; it does not prove that an exploit was executed. An observed result in a controlled environment can prove more for that environment, but it does not make untouched routes or production versions safe.
The right purchase decision depends on the job. If you need developer-ready evidence about code, assess the quality of source review and false-positive control. If you need assurance about a deployed application's behavior, examine the authorized runtime method and its boundaries. If you need an external test for a customer or risk program, check whether the proposed scope, supervision and reporting meet that requirement; a vendor label alone cannot answer it. OWASP's Web Security Testing Guide introduction distinguishes source code review from penetration testing and describes different strengths and limits for each.
Start with the work performed, not the product name
Use this evidence ladder when reading a sample finding:
| Evidence level | What happened | Honest conclusion |
|---|---|---|
| Candidate | A model or rule suggested a weakness | A lead needs review |
| Source-supported path | Reviewer traced actor, entry, controls and potential effect in a pinned revision | The code may permit the effect under stated assumptions |
| Independently verified source finding | A separate review checked original and counterevidence and approved the path | A source-supported issue exists for the assessed scope and revision |
| Authorized runtime observation | A defined action was safely performed in a named environment and its result recorded | That observed behavior occurred under those conditions |
| Production conclusion | Deployment equivalence and applicable runtime controls were separately established | A bounded statement about the production system may be justified |
Do not let a vendor jump from row one to row four because a model wrote a convincing request sequence. A report should say whether the request was merely proposed, sent to a sandbox, or observed against a specific application build. Conversely, source review may reveal code flaws a black-box exercise misses. The OWASP WSTG treats these as complementary techniques with distinct limits.
A realistic claim to examine
Illustrative vendor evaluation only: The fictional vendor Northstar Audit says its agent “autonomously pentested” HarborPay and found a payment approval bypass in one hour. The sample report cites a payment route at invented commit c0ffee1 and describes how a read-only finance user could call it. It contains no environment name, request/response evidence, test account role, route deployment map or rules of engagement. A separate note says the agent never sent a request; it inspected source and generated a proposed proof-of-concept.
The CTO can still value the report. Its source citations may point to a real missing authorization check. But the accurate description is AI-assisted source review with an unverified candidate, pending independent inspection of the route, policy and downstream service. Calling it an executed pentest would overstate the evidence. If HarborPay later authorizes a controlled staging check with synthetic accounts, the observed result can be attached as new evidence, labeled with the staging build and limits. No such action occurred in this illustrative case.
The CTO's decision record might read:
Vendor claim: “Autonomous pentest confirmed payment bypass.”
Observed method: Source inspection at
c0ffee1; generated request plan; no runtime request evidence.Current evidence: Route citation supplied; actor permission, downstream policy and deployed mapping not yet independently checked.
Decision: Treat as an investigation, not a confirmed exploit. Ask security to verify the source path; require separate authorization before any staging validation.
Purchase implication: Evaluate Northstar as a source-review aid on this sample. Request a separate demonstration for the runtime testing claim.
That record avoids both dismissing a useful lead and accepting a stronger claim than the vendor demonstrated.
Grade a mixed packet claim by claim
A stronger vendor packet may mix several evidence types. Do not accept or reject it as one unit. Continue the fictional evaluation with three attachments:
| Attachment | What it contains | What it supports |
|---|---|---|
| A: generated request | Model-written payment-approval request, never sent | A proposed validation step only |
| B: staging observation | Authorized read-only-role request received denial on build H-18; own permitted case succeeded | That staged route denied the attempted action under those conditions |
| C: source trace | Missing approval check in worker at c0ffee1; worker deployment not mapped |
Conditional source concern, not production confirmation |
The marketing cover says “payment approval bypass confirmed.” The attachments do not support that combined statement. B records a denial response on its staged route. It becomes stronger counterevidence once the reviewer checks that the intended control denied the action before any approval side effect. C may describe a different worker; A is a plan. The CTO's first request is the mapping between the three, not a more dramatic proof of concept.
A useful returned clarification might say: “B exercised the direct API in H-18. C concerns the scheduled worker in H-19, which was outside the active-test scope. A was generated for that worker but never executed.” The corrected report now has two bounded results: direct API denial in H-18, and an unresolved H-19 worker concern. That is valuable evidence, but it does not demonstrate the advertised autonomous confirmation.
Define the pilot's acceptance conditions
Before granting access, agree on one nonproduction objective and the evidence it must produce. For HarborPay, the objective is whether a read-only finance role can approve a synthetic payment through the named API and worker paths. The pilot contract needs actual environment/build identity, controlled roles and objects, allowed actions, stop conditions, and an owner for unexpected effects.
The evidence acceptance rules are concrete:
- Every runtime result names the request/event, actor, fixture, build, relevant response and side effect; generated requests are labeled separately.
- Every source claim cites the pinned path and decisive control or missing boundary; unavailable worker code remains unavailable.
- Pair the denied attempt with a comparable permitted case on the same intended workflow. Establish that the denial came from the relevant control, and inspect the resulting state: a denial response after a payment was already approved would not demonstrate effective prevention.
- Any scope expansion or risky action follows the authorized engagement process; repository read access does not authorize payment activity.
- The report distinguishes source-supported, runtime-observed, unresolved and approved conclusions, with no automatic production promotion.
These are proposed pilot conditions, not claims about Northstar or any real vendor. NIST SP800-115 supplies assessment-planning context. A customer requirement for an independent external engagement may impose additional qualifications and deliverables; evaluate those explicitly rather than treating “autonomous” as a substitute.
Make a bounded purchase decision
For this sample, HarborPay can approve a limited source-review evaluation if the citations and workflow are useful, while withholding credit for the unproven runtime capability. It can request a separately scoped demonstration with synthetic data to evaluate active testing. It need not grant production access or demand destructive exploitation to settle the marketing language.
Also price the human work left behind. If engineers must reconstruct every route and correct every evidence label, that effort belongs in the comparison. Inspect source/prompt/log retention and permissions before connecting a real repository; encryption claims alone do not settle who can read the data.
The best next vendor question is specific: “For attachment C, show which independent evidence supports the actor and worker deployment, and label the result unresolved if those facts are unavailable.” A vendor that answers that honestly provides a more usable assessment than one that repeats a certainty badge.
Run a bounded comparison before buying
Give each shortlisted vendor the same authorized, nonproduction question and the same known scope. Include one plausible alert defeated by a real control, one issue whose outcome depends on a missing deployment fact, and one path the team can independently check. Do not reveal the expected answers to the vendor during the run. Review reports with an AppSec engineer and a developer, then compare: correct dispositions, false positives, unresolved-status honesty, source and runtime provenance, time to a usable ticket, safety controls and total analyst work. This is an evaluation design, not a claim that any vendor achieves a particular rate.
If a vendor offers only source access, score it as source review. If it also performs authorized staging actions, score that work separately. Do not demand a dangerous production demonstration to prove a marketing term. Keep the original scope and expected evidence in writing so a polished demo cannot replace the agreed question.
Ask the vendor to explain a failure, not just a successful demo
A useful procurement conversation includes a case where the agent stopped, misread a control, or needed human help. Ask for the original scoped objective, the incomplete evidence, the stopping reason, and the eventual corrected report. The purpose is to understand how the service behaves when the happy path breaks. A polished successful example cannot show whether the system preserves uncertainty under pressure to produce a result.
For HarborPay, ask what happens if the worker's deployment identity cannot be established. The vendor should be able to describe the resulting report and the next owner without inventing a production conclusion. Ask what happens if a permitted staging request fails for an unrelated fixture error. That failure should not be presented as proof that authorization blocked the attempted action. The vendor's answer can remain descriptive until the pilot produces observable evidence.
Also examine the boundary between the service and the buyer's team. Who reviews a model's proposed action before it reaches the runtime environment? Who can stop an engagement? Who decides that an unexpected effect requires incident handling? These responsibilities should be explicit in the agreed scope and operating process. “Autonomous” does not explain them.
The same applies to completion. A report that ends because its time allowance expired should say what was attempted, completed, and left open. The buyer needs to distinguish an exhausted search budget from a completed assessment of the named objective. Ask for the stop reason in the sample deliverable before evaluating headline finding counts.
Finally, make the purchase memo match the demonstrated capability. A bounded decision could say: “The pilot produced useful source evidence and honest unresolved states. Runtime validation capability remains unevaluated because the agreed staging phase was not conducted.” That statement allows a sensible limited adoption while preserving the unanswered procurement question. It neither rewards the stronger marketing claim nor discards useful work that actually occurred.
For your next vendor call, send the pilot acceptance conditions and ask for one redacted end-to-end report with clearly labeled source and runtime evidence. If your immediate choice is between static analysis, dynamic testing and a human pentest, use the methods comparison. If code scanning is already in place, use when an external pentest remains useful.