Give each candidate the same authorized code snapshot, scope, time budget, known questions and review process. Measure whether it finds useful, defensible issues, what it misses, how much human work each result creates, and whether it can fit your source-handling and delivery rules. A scanner with the most alerts has not necessarily helped your team. A scanner with no alerts has not established that the application is safe. The decision changes when your main problem is dependency inventory, a CI gate, a complex access-control path or a strict source-location requirement; define that job before inviting vendors.
Two weeks can establish whether a tool deserves a larger evaluation. It cannot establish a universal detection rate, future operating cost or compliance. The useful output is a signed decision memo with observed evidence and open questions. NIST's Secure Software Development Framework treats vulnerability identification and response as practices within software development. OWASP SAMM's security testing practice explicitly considers depth, accuracy, integration, usage and cost when choosing tools. Those are better pilot inputs than a vendor's category label.
Decide what the pilot must answer
Start with the business bottleneck. A CTO of a lean B2B software company may ask: “Can we identify a small set of customer-impacting code risks and get them fixed without consuming the next sprint in alert review?” The AppSec lead's corresponding question is: “Does the evidence support each claim in our application?” The platform lead must ask whether a partial scan, failed integration or revoked credential is visible. Procurement and privacy reviewers need the source and finding data flow before anyone grants repository access.
Write three to five hypotheses that could be falsified. For example:
- The scanner can identify at least one previously reviewed access-control path in our chosen service, with a reproducible explanation of actor, entry point, control and effect.
- The reviewer can reject a deliberately non-issue using visible counterevidence, rather than hiding it with a broad rule suppression.
- A merged fix produces a new result tied to the new revision, with “fixed,” “still present” or “inconclusive” supported by evidence.
- A scope exclusion or timeout is shown as incomplete coverage, not as a clean bill of health.
- The repository grant, processing locations and retention terms satisfy our policy or have named exceptions.
Declare thresholds before seeing results. If a hypothesis falls outside a tool's scope, record that boundary; decide whether it disqualifies the tool for your required job.
Keep quality separate from strength of evidence
For each result, record the observed outcome and whether it is asserted, demonstrated by the vendor or repeatable by your team. A repeatable failure is strong evidence of a weakness. Review coverage, claim validity and known misses, handoff, failure behavior, source handling and total effort. A failed hard requirement cannot be averaged away by good results elsewhere.
OWASP ASVS can organize control questions, and OWASP Benchmark supplies controlled cases. Neither turns a few selected results into general application coverage. The common packet and case ledger below make the comparison inspectable.
Set up a fair, bounded sample
Select two representative services if possible: one customer-facing path with authorization decisions and one service with the language or framework that consumes much of your engineering time. Pin immutable revisions. Record what is deployed, which branch is scanned and which repositories or folders are excluded. Make the code sample big enough to include middleware, data access and configuration, but small enough to review results within the pilot. For a monorepo, name selected components and refuse any “full monorepo coverage” label unless the evidence supports it.
Prepare a small reference set before vendor results are visible. It might include a previously fixed issue, one independently confirmed open issue, a known false alarm caused by a real control, and a path requiring deployment context. Keep the evidence and answer key with your reviewers. If no confirmed cases exist, use a deliberately vulnerable authorized test fixture and say what it cannot represent. Do not plant live faults in production. Have a reviewer inspect all purported new positives and some sampled negatives. This helps catch both noisy alerts and silent misses, though it cannot prove absence of other misses.
Give every candidate the same written instructions: allowed repository and revision, scan mode, time and spend ceiling, whether a human vendor analyst may assist, whether any data may leave the chosen boundary, and how questions will be answered. If one vendor receives three days of expert tuning and another receives default setup, compare those modes explicitly. Record whether the scanner executes build scripts, dependencies or repository instructions; policy may prohibit that on untrusted customer code. For a hosted scanner, get a data-flow description before connecting source.
Repository access deserves an exact review. GitHub's installed-app documentation explains how to inspect permissions and limit access to selected repositories. Capture the requested permissions, actual installation selection and revocation procedure. For temporary tokens, record scope, expiry and redacted issuance metadata; never retain live token values in pilot evidence. A claim of “read only” should be matched to the actual grant and every delivery path, including comments, checks or issue creation.
A two-week pilot with a result you can audit
All companies, candidates, outputs and timings in this exercise are fictional. No evaluation was run. Harbor Metrics has one AppSec engineer and wants actionable authorization findings in a Node service, with a Go worker as a secondary coverage question. Before results arrive, its protocol permits a Node-only next step but requires separate evidence before any Go adoption. It does not permit silent source execution, unapproved processing or automatic ticket closure from a quiet rerun.
The two candidates receive the same pinned Node revision, its shared authorization code, the Go scope, a ten-working-day window and the same authorized analysis conditions. Vendor assistance is logged separately. The reference packet includes an unsafe synthetic authorization variant, a safe query variant and a case whose deployment control is intentionally withheld. Reviewers retain the answer key until initial results are frozen.
| Day | Artifact due | Failure that changes the plan |
|---|---|---|
| 1–2 | Written required/optional scope, permission record, immutable source packet, reference-case key | Unapproved access or processing: do not transfer private source |
| 3–4 | Run configuration, complete/partial status and initial immutable outputs | Unsupported component: record the exclusion; do not call the estate covered |
| 5 | Scope reconciliation and first adjudication | Missing source or parser failure: stop the affected comparison and decide whether a permitted narrower scope remains useful |
| 6–7 | Distinct-claim ledger, counterevidence, active-review minutes | Reviewer disagreement: unresolved until evidence settles it |
| 8–9 | One developer handoff and reassessment at a changed revision | Wrong revision or excluded changed path: inconclusive, never fixed |
| 10 | Decision memo with hard gates and the remaining question | Extend only for a bounded question that could change the decision |
On day five, B cannot parse the Go worker and exposes the exclusion. Harbor uses the protocol's predeclared Node-only fallback. A's Go results are retained as separate capability evidence, but the direct A/B comparison below covers only the common Node packet. If Go coverage had been mandatory, B would have failed that requirement; a cleaner Node result could not erase it.
Reconcile twelve alerts and four alerts
The common Node packet produces these synthetic results. C1 is the reference unsafe variant; C2 is the reference safe variant; C3 requires the withheld effective policy. C4–C6 are other distinct claims that reviewers adjudicate from the supplied source and controls.
| Underlying claim | A raw alerts | B raw alerts | Independent exercise adjudication |
|---|---|---|---|
| C1: account check absent on the reference path | 3 | 1 | Supported |
| C2: query injection despite parameter binding | 2 | 0 | Rejected; binding defeats the allegation |
| C3: Node export delivery may bypass account policy | 1 | 1 | Unresolved; effective control missing |
| C4: separate export permission gap | 2 | 1 | Supported |
| C5: HTML issue defeated by contextual encoding | 2 | 1 | Rejected |
| C6: unreachable demo-only handler | 2 | 0 | Rejected for the claimed deployed path |
| Total | 12 | 4 | A: 6 distinct claims; B: 4 |
Both identify two supported claims. A has three rejected claims and one unresolved; B has one rejected and one unresolved. A's six extra raw messages beyond its six claims are duplicates under the exercise's issue-grouping rule. They still cost review time; they do not create six more vulnerabilities. B's silence on C2 is a useful result for this known safe case, not proof of safety on every unflagged path.
The active-effort log is separate from elapsed waiting:
| Work in common Node scope | A | B |
|---|---|---|
| Setup and configuration | 60 min | 45 min |
| Initial deduplication | 45 min | 10 min |
| Claim adjudication, including counterevidence | 180 min | 110 min |
| Developer handoff and clarification | 40 min | 30 min |
| Reassessment review | 30 min | 25 min |
| Observed exercise total | 355 min | 220 min |
Do not turn the 135-minute difference into an annual saving yet. It comes from one deliberately selected packet, and the candidate setup costs may recur differently. Record per-role time if you intend to price it; a minute of platform setup and a minute of developer interruption need not have the same cost.
Make the last gate determine the recommendation
On day nine, A's reassessment includes the changed Node path and provides a usable new-revision explanation. B's run excludes that changed path after a configuration error, yet its summary says “no open alerts.” The buyer records B's fix result as inconclusive. A failure to demonstrate this workflow matters even though B required less review time.
The day-ten memo is therefore:
Conditional Node-only extension for B; no rollout yet. On the common Node packet, B produced the same two supported claims with fewer rejected claims and 220 versus 355 active minutes. Its Go scope failed and remains excluded. Its changed-revision reassessment did not examine the corrected path, so it has not met the required closure-workflow gate. Allow three additional working days to demonstrate explicit partial-scope reporting and a complete rerun of that path at the new revision. If either remains missing, choose A for the demonstrated Node workflow or make no purchase. Go adoption requires a separate decision. Private-source handling must remain within the approved configuration throughout.
An extension here buys one answer. It does not turn an unfulfilled requirement into a soft preference or conceal the stronger evidence A supplied. If the deadline passes without the required artifacts, finish the decision rather than starting another open-ended pilot.
Challenge both the alerts and the silence
A reviewer should inspect the pinned source and controls independently of the vendor's explanation. Record what would defeat each proposed path. Check the prepared reference cases and selected unflagged high-risk paths; silence is not a negative result unless the relevant scope was examined. “Two of three known cases surfaced” describes that packet, not general detection accuracy.
Put a supported issue through a developer handoff and positive new-revision reassessment. The buyer should be able to identify the intended business rule, owning service and changed control without the vendor narrating the screen. Retain unsupported and inconclusive outcomes so the final memo reflects what actually happened.
Keep the access gate concrete
Before a private-source run, retain the actual repository grant, selected processing route, source-bearing logs, support-access terms and retention/deletion evidence. A polished finding cannot compensate for a failed hard requirement. Use the regional data-handling walkthrough for that dossier instead of repeating a generic trust questionnaire in every pilot.
For delivery, inspect the actual issue or code-review integration: stable finding identity, revision, partial-scope state and access control must survive export. GitHub's SARIF documentation explains an interchange route for third-party results; accepting the format does not establish that those workflow details are correct in your setup. Keep annual fees, per-run charges and human effort separate, and model only the frequency and scope you intend to buy.
Finish with a decision that can be revisited
Use four outcomes. Proceed when the hard requirements are met and the observed result is repeatably useful. Extend when a narrow unanswered question could change the answer; name it and set a deadline. Narrow when a tool serves only one selected repo or workflow well. Reject when a hard trust, coverage or delivery requirement fails. Capture dissent from the developer, AppSec and platform reviewers so a business sponsor sees the real tradeoffs.
At the end of day ten, the CTO should have a one-page decision memo, a scope ledger, an adjudicated finding sample, an effort log and a list of unresolved vendor answers. The next practical step is to use questions for an AI code-scanning demo to close evidence gaps, then compare SAST tools on common criteria if static analysis is the central purchase.