How to compare scanner coverage across languages and frameworks

Compare coverage on the code paths and controls your services actually use, at a pinned revision. A “supported languages” list is a starting inventory, not a measure of how much code was acquired, parsed, analyzed or meaningfully reviewed. For each important service, record its language, framework version, entry points, authorization and data boundaries, generated or excluded code, analysis mode, and what the vendor can show after a partial run. Then test one real risk path and one nearby safe path with the same scope and budget.

The answer changes when your main exposure sits outside application source: dependency packages, cloud policy, runtime configuration and browser or API behavior each need a matching method. A scanner may be useful for some of those jobs and outside scope for others. NIST's Secure Software Development Framework treats vulnerability identification as part of a wider development process; no single product label settles whether every control was assessed.

Build a service matrix before reading a vendor grid

Start with an engineering inventory, not a feature comparison. List the production services and the shared code on which their security decisions depend. For each one, capture the deployed revision, language and version, framework and version, build or generated artifacts, routing method, authentication and authorization locations, data store, infrastructure definitions, and first-party versus vendored code. Mark test fixtures separately from production paths. The platform engineer owns the inventory; service owners validate what actually deploys.

Prioritize by exposure and consequence. A language used only for a migration script needs a different evaluation than the TypeScript API that serves every tenant. A shared authorization library may be more important than a large front-end directory. If one repository holds many services, the repository boundary is too coarse; name components and shared controls.

Use this worksheet:

This table is a template, not a claim that any vendor covers these stacks. Fill actual versions. A scanner that reads generic text in Go may still be useful, but do not equate “we can ingest the file” with a demonstrated dataflow or framework-aware check.

Ask what “covered” means at each stage

Make the vendor report these stages separately:

1. Discovered: the path appeared in repository inventory.
2. Acquired: the relevant contents were available to the assessment.
3. Parsed or indexed: the tool could locate symbols, routes or relationships, with parser failures visible.
4. Submitted for analysis: a rule or reviewer received the relevant context.
5. Actually reviewed: a named check or investigation evaluated the path and its controls.
6. Concluded: a finding, no finding, unknown or out-of-scope outcome was recorded with a reason.

This stage model is a buyer's audit method, not an assertion about how every scanner is built. Ask for per-service counts and exclusions: unsupported encodings, enormous files, generated code, symlinks, submodules, unavailable dependencies, budget caps and timed-out workers. An inventory count by itself cannot show whether the authorization helper was inspected. A “zero issues” report on a skipped service should not be read as a passed review. OWASP's reporting guidance places scope and limitations beside findings precisely because results have boundaries.

For framework claims, ask the vendor to name the versions and features it supports. Does it understand middleware order, decorators or annotations, asynchronous jobs, object mappers and query builders in your framework? Can it follow a call into a shared library? Does it account for a gateway or service mesh control that is absent from the repository? If a security decision lives in deployment configuration, the tool must either inspect that configuration or say the answer is unknown. Do not let “language support” stand in for these questions.

Inspect a small framework-specific packet

North Quay and the results below are fictional. The snippets are unexecuted teaching fragments, not a complete application or a scanner benchmark. Pin the exercise to Express 5.1.0 with JavaScript route handlers. This is a chosen version for a repeatable example, not a claim that it is the latest release. Retain the lockfile, shared middleware and query helpers with the source revision.

Both variants below use requireSession, whose supplied contract authenticates the caller and supplies a server-derived current account. It does not authorize individual reports. reports.byId is a global lookup. reports.forAccount constrains both report ID and account ID. Reports do not change account during the read in this simplified fixture. No database row policy or later response filter adds an unstated control.

// Variant U: source-supported missing object boundary under these contracts.
app.get('/reports/:id', requireSession, async (req, res) => {
  const report = await reports.byId(req.params.id);
  if (!report) return res.sendStatus(404);
  return res.json({ title: report.title });
});

Now put the control in a custom middleware function in another file:

// Variant S: registered instead of U, not alongside it.
async function loadOwnedReport(req, res, next) {
  const report = await reports.forAccount(req.auth.accountId, req.params.id);
  if (!report) return res.sendStatus(404);
  req.report = report;
  return next();
}

app.get('/reports/:id', requireSession, loadOwnedReport, (req, res) => {
  return res.json({ title: req.report.title });
});

A useful analysis must follow the registered middleware sequence, the shared helper and the exact object returned. Express documents middleware ordering and progression; the security conclusion still depends on the application's helper contract. A parser recognizing app.get has not demonstrated that reasoning.

The answer key for the supplied source is precise: U allows a caller with a valid foreign ID to select its title under the declared contracts; S's mandatory scoped query defeats that path. Neither establishes ID discovery, actual deployment or an executed disclosure. A scanner that flags both equally has not shown the discrimination needed for this buyer's object-authorization job.

Make the artifact show which stage failed

Suppose the candidate returns these fictional receipts:

“Supports JavaScript, Go and Python” cannot summarize this result honestly. The useful answer is where the candidate produced a defensible conclusion and where work stopped. The platform owner asks for the missing middleware to be included on the same revision, then checks whether the reason changes appropriately. A better label with no new evidence is not a successful rerun.

Use a version change to challenge route coverage

Framework versions can change what a route declaration means. The Express 5 migration guide documents named wildcards: /*splat excludes the root path, while /{*splat} includes it. It also documents forwarding rejected promises from async handlers to error handling. Neither behavior should be guessed from an older framework model.

For a separate route-inventory exercise, replace an Express 4-style /* declaration with the selected Express 5 pattern. Ask the scanner to identify the registered route, its handler and whether / belongs to the intended match set. Keep the full registration order. This is an entrypoint interpretation question, not a claim that changing a wildcard automatically creates an authorization flaw.

If the selected version rejects a declaration, the application may fail to start; “no reachable finding” is not evidence of successful coverage. If /*splat intentionally excludes /, the scanner must not invent a root handler. If the app uses /{*splat}, a route extractor that omits the root has a scope gap. Record the actual migration variant and evidence instead of claiming that “Express support” persists unchanged across versions.

A minimal capability record after the follow-up would say:

Express 5.1.0 fixture: U and S distinguished after the shared middleware was included; route-pattern interpretation checked against the selected declarations. Evidence is limited to these unexecuted source examples. Go worker remains incomplete and Python's gateway-dependent outcome unresolved. Repeat the relevant checks when framework version, custom wrapper, route registration or query helper changes.

For an actual procurement exercise, replace the fictional observations with your authorized run artifacts and independent review. Do not copy the sample result into a vendor scorecard as if it had occurred.

Keep the final matrix tied to jobs

For every required service, list the source/version, selected sensitive paths, decisive controls, observed analysis stage, outcome and remaining evidence. Decide which gaps can be covered by another method and which prevent adoption. A supported safe case is useful counterevidence; an unexamined path is not a safe case.

OWASP ASVS can organize the relevant control questions. OWASP Benchmark can supply additional controlled cases. Neither replaces this application-specific account of wrappers, jobs and external policies. Require the same packet and assistance conditions from every candidate so a framework comparison does not become a comparison of different inputs.

Maintain the capability record after adoption

Keep the successful Express pair as a local reference, including the framework version, middleware arrangement, helper contract and scope configuration. If the team introduces a new authorization wrapper, the previous result still describes the old case; it does not establish that the scanner follows the new abstraction. Select a small paired example around the changed boundary and repeat the relevant evaluation before extending the claim.

This maintenance can be proportional. A documentation-only change need not trigger the entire language matrix. A changed request path, parser, generated client or review configuration deserves an explicit impact decision from the capability owner. Record what was repeated and what remained applicable.

Partial adoption is a valid outcome when the operating workflow makes the boundary visible. The team may use the demonstrated TypeScript capability while retaining another method for the incomplete Go worker and the Python gateway question. Describe that arrangement in onboarding and reporting so a new service owner does not mistake a purchased language label for evidence about their own framework path.

Handle missing evidence as an outcome

If one run truncates inventory, fails to acquire a submodule or exhausts analysis budget, do not average that service into a clean coverage percentage. Preserve the exact failed stage, reason and recovery path. Ask whether a rerun on the same revision changes the result and whether the prior partial state remains visible. For a new framework version, repeat a narrow paired test instead of assuming a prior version's result carries forward.

Coverage is a denominator problem. State the denominator as selected first-party production paths or named services, not raw lines of code alone. Lines are useful for scale, but 50 lines of shared authentication can matter more than 50,000 lines of generated schemas. If you report “three of four services reviewed,” also name the excluded fourth and which control it owns. Neither an alert count nor a file count measures quality by itself.

The platform engineer's final artifact should be a versioned matrix showing where the tool adds value now, what failed, and which services need another method or manual review. Hand the AppSec lead the paired-case evidence for validating verification claims, then use the regional data-handling checklist before granting broader repository access.