How to evaluate a SAST tool with a real repository

Run each candidate against the same pinned, representative repository scope; judge what it actually analyzed, which known paths it found, how many reported paths survive review, and whether developers can use the output. A demo project or a raw finding count is too weak for a buying decision. The answer changes when your languages, frameworks, deployment model, source-handling rules or available review time change. Define those constraints first, then conduct a bounded pilot with independently labeled examples and a clear “unknown” category. The result should be a decision memo that explains both the tool's value and the work it leaves to your team.

OWASP's SAST selection criteria include language and framework support, accuracy, build requirements, integration and output interoperability. Its Benchmark project provides controlled cases and scoring concepts, while warning that benchmark cases can be simpler than real applications. Use a benchmark to inspect a tool's general behavior, then use your own authorized repository to discover local fit. Do not present a vendor's benchmark score as a measured false-positive rate in your codebase.

Prepare a fair pilot before connecting a vendor

Choose a revision that will not change during the comparison. Inventory the relevant services, languages, frameworks, generated code, shared libraries, build conditions and application boundaries. Select one representative high-value service, one awkward service that stresses framework handling, and one known high-risk workflow. If a monorepo is too large, say which directories are excluded and why. “Connected the repository” is not evidence that each file was parsed or reviewed.

Agree on authority and source handling before uploading or granting access. The repository owner should know exactly which read permission, branches, files and metadata the candidate receives; where source or derived data is processed and retained; whether a model provider or subprocess is involved; who can see findings; and how access will be revoked. A vendor answer is a claim to verify against its documentation and contract. If the tool requires running repository code, hooks, package installs or a build, treat that as a separate security and environment decision; do not silently execute untrusted source to make the scan succeed.

Create a small labeled reference set with a reviewer who knows the application. Include confirmed unsafe paths, similar-looking paths with effective controls, and important business rules that may require manual review. Pin each label to source and any necessary deployment evidence. The labels are not sacred: if the pilot reveals a missed control or an incorrect old finding, record the correction and re-review it independently. Do not mark an unresolved path as “safe” merely because an incumbent tool did not report it.

Build an answer key that can survive being challenged

Synthetic corpus, not measured scanner performance: Meridian Desk pins revision MD-301. Instead of starting with two vendors' totals, the reviewer records four application-specific cases:

Each row needs actor, entry, effect, revision, supporting source, defeating controls considered and label owner. “Unsafe” here is a corpus label for the bounded source claim, not a claim that a customer incident occurred. X-1 remains in the corpus as an uncertainty case but is excluded from accuracy scoring until its intended rule is known.

For U-1, write the match criterion before seeing output:

A matching detection identifies the queue consumer's caller-controlled
report/account selection, the missing required ownership decision,
and the resulting possibility of foreign report content processing.
A warning about weak logging in the same file is not a match.

This prevents a tool from receiving credit for merely pointing near the known defect. It also prevents an evaluator from rejecting a correct finding because its category name differs from the incumbent's.

Correct the reference, then correct every score

During review, someone finds that U-1's consumer always calls a previously omitted data-service guard before fetching content. The initial answer key was wrong. At the pinned revision, that guard checks the report's stored account and the current delegation. Rename the case G-3, preserve the old label and reason in history, and have another qualified reviewer adjudicate the correction. Do not edit the label invisibly after seeing which candidate would win.

Suppose a fictional run alerted on U-1 and G-1, and correctly stayed quiet on G-2. Before correction, the local tally was one matching unsafe detection and one false alert. After correction, both alerts concern guarded paths. The scan output did not change; the ground truth did. Any published fraction based on the old label must be recalculated or withdrawn. This four-case corpus now has three guarded cases and one unresolved case, with no established unsafe case. It can expose false alerts, but has no positive-case denominator for measuring detection or recall. Add a separately reviewed unsafe case before making that comparison; do not preserve the erroneous label just to keep a score available.

If the newly found guard's mandatory call order is itself uncertain, the correct label is unresolved, not guarded. Remove that cell from the scored denominator until resolved. Keep the unresolved count visible so the evaluation cannot improve by quietly discarding its hardest examples.

Decide whether the comparison is valid at all

A useful run manifest is small enough to inspect:

revision: MD-301
selected: api/, worker/, shared-auth/
rules: security-pack R7
expected languages: TypeScript, Python
actual analysis: TypeScript complete; Python extraction failed

The worker case was not analyzed. That is an operational coverage failure for the selected scope; it must not be reported as a same-scope detection comparison with a run that analyzed Python successfully. Record both facts: the pilot failed its intended coverage condition, and there is no valid detection observation for the worker cell. A buyer may still consider the failure material, but should not disguise it as a false-negative rate measured on analyzed code.

Use three comparison states:

A valid comparison does not require every tool to use identical internals. It requires the same evaluated question and enough receipts to establish what each actually assessed.

Keep configuration work separate from the comparison

A failed first run can be informative without being the final measurement. In Meridian's pilot, ask the candidate's owner to diagnose the Python extraction failure and record the configuration change needed to address it. Preserve the failed receipt. If a corrected run completes, compare its output against the same pinned cases using the same label version. The successful retry does not erase the setup effort, and the failed attempt does not supply an accuracy score for code that was never analyzed.

Use a short run record for each attempt: selected revision, configuration, tool and rule versions, admitted files, excluded files, completion evidence, raw result location and evaluator notes. Add whether a vendor engineer changed a model or rule specifically for one example. Assistance may be appropriate during a pilot, but a tailored demonstration and the default product are different operating conditions. A team deciding whether it can maintain that customization needs the distinction even if the resulting finding is correct.

Define a reasonable configuration allowance before the comparison. For example, Meridian might allow each candidate one supported setup session and one repeat after correcting an agreed configuration error. That is a proposed pilot rule, not a universal evaluation standard. Its purpose is to avoid giving the preferred tool unlimited tuning while treating another candidate's first attempt as final. If a tool needs materially different preparation, document that requirement and judge whether it fits the team's normal release process.

Adjudicate ambiguous matches before calculating a score

Suppose a candidate reports “missing authorization in worker” but cites only the queue entry point. The words resemble the original U-1 label. They do not establish a match after U-1 has become guarded case G-3. Open the candidate's actual path and determine whether it identifies a different bypass, misses the mandatory guard, or merely warns that a review is needed. Those outcomes should not receive the same disposition simply because they share an authorization category.

Have the evaluator write the strongest supported interpretation of the alert in one sentence. Then compare that sentence with the settled case definition and decisive evidence. If the alert describes an alternate caller absent from the original case, create a separate review item rather than stretching G-3 until it includes unrelated behavior. If the relevant caller implementation is unavailable, leave the new item unresolved. This protects both the tool and the answer key from being judged against a moving target.

Preserve the raw output alongside the match decision. A later reviewer should be able to distinguish what the tool actually said from what a helpful evaluator reconstructed. If reaching a useful finding required discovering an omitted sink, correcting an incorrect resource identity and supplying the entire missing control analysis, note that contribution. The output may still have saved search time, but crediting the tool with the evaluator's complete conclusion would hide the work a future operator must perform.

Before expanding the corpus, review why the existing cases were selected. Meridian's deliberately difficult ownership and query cases are useful for testing fit with its architecture. They are not a random sample of all future alerts. A ratio from that set describes those cases under the recorded conditions; it does not establish the proportion of every production alert that will be useful. Keep the actual case decisions available to the buyer, who may care more about one important workflow than an aggregate percentage.

Preserve realism without leaking the answer

Include a guarded lookalike for each important unsafe pattern, plus ordinary paths and unresolved business cases. Avoid naming a function deliberate_vulnerability or placing the expected answer in a comment: that measures recognition of the fixture's label. When removing private material from a repository sample, check that the transformation did not also remove the middleware, model or configuration needed to decide the case.

OWASP Benchmark provides controlled-case and scoring context; local application corpora add realistic dependencies but usually have small, deliberately selected samples. Neither supports a universal detection claim from a handful of examples. Keep sample composition, label version and matching decisions with the result.

The output of this article is a defensible corpus and valid comparison cells. Interpreting competing tools' value, cost and workflow tradeoffs belongs in the buying evaluation. Before that discussion, a reviewer should be able to reproduce why every scored alert matched—or failed to match—the exact application claim.

For your first pilot, choose one service with an existing, independently reviewed unsafe path and one convincing safe lookalike. Compare both on a pinned revision, then expand only if the result changes your choice. Use the SAST primer for the method's limits and the clean-report guide to interpret a zero-finding run.