How to design a security-tool pilot without sharing all repositories

Pick a small set of repositories that represents the decision you need to make, grant access only to that set, and write down what the result can and cannot establish. A pilot should show whether the tool gives your team useful, checkable work at acceptable access and review cost. It does not need your entire organization on day one. It also cannot prove full product coverage from a convenient sample. If a vendor requires organization-wide source access for a two-repository evaluation, make that dependency explicit before granting it.

The selection changes with the buying question. A CTO deciding whether developers can use the findings needs a repository with real ownership and ordinary review practices. A security team testing monorepo scale needs a representative module, its build and shared-control context, and a way to label unreviewed components. A privacy reviewer may require a synthetic or lower-sensitivity repository first. Do not let a high-volume but irrelevant demo repository determine the purchase for a system it does not resemble.

State the pilot decision in one sentence

Before connecting anything, write: “We will expand access if the tool helps named owners reach checkable security decisions within the declared scope, stays within the approved data flow, and stops access when we remove the grant.” Add your own business constraints: review time, false-positive burden, budget, model route, source retention and CI disruption. Name the decision owner, the platform operator who controls Git permissions, the AppSec reviewer who will judge claims, and the engineering owner who will judge fixes.

Choose a fixed observation period and a pinned commit or revision for the initial assessment. This makes it possible to ask whether a later run reviewed the same code. If the vendor can only scan the default branch as it changes, record the actual revision and timing of each run. Preserve a list of excluded repositories and modules, but do not hand their contents to the vendor just to make the list. Do not require the tool to discover a new vulnerability to earn a purchase: correctly rejecting a known false claim, identifying an evidence gap and explaining a repaired boundary may be valuable outcomes. A short pilot with no alert is not evidence of a safe codebase; it may mean the tool saw little, had no relevant rules, ran out of budget or found no issue in that sample.

Illustrative case: two of 24 repositories

Illustrative case; the company, repositories, numbers and findings are invented. AsterPay has 24 private repositories. Its CTO wants to know whether a proposed scanner will help a three-person AppSec group send developers fewer, better tickets. The team selects customer-api, where account-bound authorization is central and the owner can review results, plus reporting-service, where asynchronous work and generated exports make the code path less obvious. It withholds identity-core because its source sensitivity needs a separate approval and legacy-mono because the first pilot cannot establish a truthful full-repository denominator there.

Both selected repositories include ordinary application code, manifests and realistic dependency use. The team notes where their behavior depends on a withheld shared service; a finding about cross-service authorization may remain inconclusive without that context. AsterPay does not seed a live vulnerability or run hostile code. It uses a previously reviewed issue, if one exists, as a reference case and asks the vendor to explain both the claimed path and the counterevidence. A model-suggested issue remains a candidate until independent review supports it. The CTO wants a ticket a developer can act on, not an impressive count of unverified alerts.

This is a selection record, not a scoring trick. A vendor can still be useful if a narrow pilot is inconclusive. It cannot honestly market an incomplete assessment as a clean full scan. Record what was discovered, fetched, indexed and actually reviewed as separate quantities if the vendor exposes them; do not turn “repository connected” into “all code examined.”

Agree on the access and data-flow envelope

Ask the vendor to list every Git provider permission, API route, webhook, runtime copy and model destination needed for the selected features. If using a GitHub App, GitHub allows the installer to select Only select repositories rather than all repositories for apps with repository permissions (third-party installation guide). GitHub recommends selecting the minimum App permissions required (App best practices). AsterPay would install for just customer-api and reporting-service, then review the actual permission screen; a vendor's read-only label is not enough to explain whether it also requests write access for PR comments or checks.

Document what happens after GitHub returns source. Name the processing region, any inference endpoint and model, source-copy lifetime, retained finding form, logs, backups, and support route. If an AWS Bedrock model is in the path, its current retention controls are model- and configuration-specific; ask for the exact approved route rather than accepting “Bedrock is private” as a complete answer. Do not authorize a sensitive repository until the terms fit its classification. The pilot may start with synthetic source while those questions are open.

Run one bounded evaluation and review the evidence

Start with a baseline sheet: repository, commit, owner, intended files or components, excluded paths, selected feature set, model route, time and budget. Ask the tool for its scope ledger after the run: what it discovered, acquired and examined; which work stopped early; and which findings are candidates, approved issues or unresolved investigations. If the product cannot supply an examined-scope denominator, mark coverage unknown rather than dividing alerts by all repository lines.

Review a manageable sample of outputs with the named AppSec and engineering owners. For each candidate, ask for the exact affected path, the actor who could reach it, the control that should stop it, counterevidence considered, business effect and a safe check for the proposed fix. A source citation proves a line exists, not that an attack works. Record why each claim was accepted, rejected or left unresolved. OWASP's authorization guidance is a useful independent reference when the sample concerns object access, but the vendor still must show the application's actual authorization path.

A scorecard that explains why the CTO does not expand yet

AsterPay receives eight candidate outputs in the fictional two-repository pilot. All eight are examined; this is not a selected “best findings” sample. Review time includes the AppSec discussion and developer clarification. These invented results illustrate a decision, not a product benchmark:

The total is 180 minutes for eight emitted candidates: three distinct supported issues, two rejected claims, two unresolved claims and one duplicate. Three of eight outputs became distinct actionable issues. If the team also reports three supported among seven deduplicated claims, it must show that denominator explicitly. Neither ratio estimates detection recall: the pilot does not know every vulnerability in the repositories.

Engineering accepts C1's proposed change directly. C3 needs the AppSec reviewer to explain the difference between permission to create a job and permission to receive its artifact. C7's ticket initially says “fix cache isolation” without the cache key or delivery path; it becomes usable only after that evidence is added. Record that extra work within review cost rather than attributing a polished handoff to the original output.

The scope ledger adds another constraint. Four selected workflow questions were planned across the two repositories. Two reached a reasoned outcome at matched versions; identity delegation and the retry worker remain blocked by the two missing artifacts above. Indexing both repositories does not turn those four obligations into four completed reviews. A no-finding component outside the planned questions contributes no assurance claim.

Choose the smallest next experiment that changes the purchase decision

The CTO declines organization-wide expansion. The work contains useful issues, but two questions still prevent a fair judgment of cross-service reasoning. AsterPay offers a second bounded phase: obtain the running worker mapping without any new repository access, and separately consider a narrowly defined identity-control review under its source-classification approval.

If Git access must be added for identity-core, repository-level permission may expose more than the selected functions. A vendor path filter is a separate acquisition control, not a GitHub branch/file grant. The platform and privacy owners must approve that actual access envelope before the second phase. The remaining 21 repositories would still remain outside the pilot if that third repository were added; adding one withheld dependency is not an organization-wide rollout.

The second phase asks a specific question: Does the tool revise C5 and C6 correctly when the missing governing evidence arrives, and can it explain why the conclusion changed? A tool that closes a contradicted candidate with clear counterevidence may earn more trust than one that preserves its original alert count. AsterPay also reruns the fixed C1 path at a new pin, requiring positive evidence of the original boundary repair rather than disappearance from a changed scope.

Do not compare these eight outputs with another vendor's raw count unless scope, versions, budget and review method are aligned. The useful purchasing evidence is the work needed to turn each output into a defensible engineering decision.

Close the pilot as carefully as you opened it

At the agreed endpoint, decide expand, repeat with a different scope, or stop. Require an explicit sign-off for any move from two repositories to an organization-wide grant. If you stop, remove repository access or suspend or uninstall the GitHub App and verify a new source request is denied; GitHub documents those controls in its installed-app review guide. Ask the vendor for the separate cleanup and retention record for source copies, model requests, logs, reports and backups. Revoking Git access does not delete previously delivered copies, and deleting a task-local file is not proof of physical erasure.

If the pilot was inconclusive because a critical shared service was withheld, say so. The next step may be an approved cross-repository assessment, not a favorable or unfavorable verdict on the tool. If it failed because the vendor could not limit source access, explain that as an access-fit problem. Keep the final decision memo short enough for the CTO to read: approved scope, observed work product, review cost, trust exceptions, unknowns and the exact next access request.

Next action: choose one repository with a named owner and one with a different architecture, fill the selection table, and ask for a selected-repositories install. Read the Git permissions checklist before connecting them and the tenant-isolation checklist before expanding the pilot.