The useful cost question is: which review work can we reduce while preserving the checks that find real issues? Measure active staff time, separate it by reviewed outcome, and multiply each person's minutes by an agreed loaded hourly rate. Keep duplicates, disproved claims, unresolved investigations and supported findings separate. Then price one specific improvement, including its setup and maintenance.
The worked example below starts with $1,460 of review labor. Of that, $760 relates to rejected claims, duplicates and out-of-scope work. Yet the proposed improvement recovers only $95 of recurring labor value per comparable period, and costs $200 to set up. Following that calculation prevents a common mistake: calling every dollar associated with a false alarm a dollar you can save.
All companies, alerts, rates and results in the example are fictional teaching inputs. The method uses your own activity records. OWASP SAMM connects defect measures with verification activity and improvement decisions; NIST's measurement guide provides a framework for selecting and evaluating measures. Neither establishes a universal price for an alert.
Define “noise” so the denominator means something
Use the distinct claim at a pinned revision as the main unit. Preserve raw alert count too, because duplicate notifications can still cost time. Assign a disposition after review:
| Label | Decision rule | Cost treatment |
|---|---|---|
| Supported issue | The claimed harmful path survives source and counterevidence review within declared scope | Include verification and handoff effort as necessary security work, not “waste.” |
| Rejected false positive | Evidence shows the claimed harmful path does not hold under the stated conditions | Count time spent checking and recording why. |
| Duplicate | Another claim already tracks the same behavior and affected path | Count incremental identification and merging effort once. |
| Unresolved | A material fact, scope or reviewer decision is still missing | Count work done so far; do not relabel as false positive. |
| Valid but deferred | The issue is supported, but business priority or remediation plan differs | Keep separate from false positives; count triage and coordination if relevant. |
| Out of scope | Tool checked or reported a path your protocol excluded | Count avoidable handling, then fix configuration; keep it visible. |
These categories prevent a seductive but false calculation: “dismissed alerts × average developer hourly rate.” GitHub's code-scanning guidance asks users to explore an alert before choosing a dismissal reason. A “won't fix” choice and a disproved claim tell different stories. A tool can also be quiet because it skipped a repository. Keep the coverage ledger beside the dispositions.
Log the work at the point it happens
Use a two-week sample long enough to see ordinary scans and at least one fix or rerun. Record every alert source, run, pinned revision, issue identity, owning service, initial category, final disposition, reviewer and minutes. Add a field for why an analyst stopped: claim supported, control disproved it, missing deployment fact, duplicate, or scope failure. Keep times in small blocks rather than asking people to reconstruct an entire sprint later.
Separate intake/deduplication, technical review, developer clarification and workflow maintenance. Record direct tool charges separately from labor. Each event needs a reviewer, active minutes, claim identity and a single activity category; a row can link to the resulting evidence or missing fact.
Start and stop a clock for active work; do not count passive waiting as staff hours. If a meeting has two people for 30 minutes, it uses one person-hour, but only count it once under the appropriate activity. If the same minute is “triage” and “developer clarification,” choose one category. Note context switching separately as an unquantified effect unless the team has a defensible way to measure it. Do not convert a hypothetical future breach into “noise cost”; missed detection is a separate risk, not a credit or debit you can derive from the alert log.
For a representative sample, record the selection method before you see outcomes: all alerts in two selected repositories, for example, or a stratified sample by rule and severity. If you review only the most alarming ten, do not scale their average to every alert. Keep the full alert population, reviewed subset and unreviewed subset visible. Duplicates may be many raw alerts for one distinct claim; report both denominators.
Calculate the labor before estimating avoidability
The basic calculation is:
Observed review labor cost = sum of each person's active minutes on sampled alerts ÷ 60 × that person's agreed loaded hourly rate.
Isolate rejected false positives, duplicate handling and out-of-scope work from unresolved investigations and supported-issue review. Add only recorded charges. A later improvement estimate must identify which of those activities changes; a disposition label alone does not establish avoidable cost.
Useful supporting rates include rejected claims / adjudicated distinct claims, duplicate raw alerts / all raw alerts, review minutes / adjudicated claim, and supported claims delivered / reviewer-hour. Each answers a different question. Show unresolved and unreviewed populations explicitly. Use the precision guide for sampling and classification; a review-cost ledger cannot establish detection quality by itself.
Illustrative two-week pilot with a filled ledger
Illustrative case; invented team, prices and results. No real vendor performance is implied. Willow Health Software evaluates one scanner on two pinned repositories over ten working days. The AppSec lead and one developer review all output under a fixed rule set. Their agreed loaded internal rates for the exercise are $100/hour for AppSec and $80/hour for engineering. These are accounting assumptions, not salary or market benchmarks. The pilot yields 30 raw alerts, grouped into 20 claim tickets: 6 supported, 7 rejected, 3 duplicates of other tickets, 2 unresolved, 1 valid but deferred and 1 out of scope. The duplicate tickets are merged with existing underlying issues; additional raw repeats are recorded separately. The disposition counts describe tickets, while a later rate at the distinct-issue level would exclude merged duplicates.
| Activity tied to disposition | AppSec hours | Developer hours | Cost at assumed rates |
|---|---|---|---|
| Rejected claims: verify controls and document counterevidence | 4.0 | 1.0 | $480 |
| Duplicate identification and ticket cleanup | 1.5 | 0.5 | $190 |
| Out-of-scope alerts and rule tuning | 0.5 | 0.5 | $90 |
| Unresolved claims: evidence still missing | 1.0 | 0.5 | $140 |
| Supported and deferred issues: necessary verification and handoff | 4.0 | 2.0 | $560 |
| Total observed review labor | 11.0 | 4.5 | $1,460 |
The first three rows total $760 of observed noise-related labor. The unresolved $140 stays in its own category. The $560 for supported and deferred issues is necessary verification and handoff work. The table excludes tool fees and unknown missed issues. These are costs observed in the fictional ledger; avoidability requires a specific alternative workflow.
Here is part of the activity log behind the rejected-claim row:
| Event | AppSec minutes | Developer minutes | Cost | Accounting decision |
|---|---|---|---|---|
| Source and control review across seven tickets | 180 | 0 | $300 | Active review, including rejected paths |
| One joint clarification meeting | 30 | 30 | $90 | Thirty minutes for each attendee; no second meeting charge |
| Counterevidence notes and helper explanation | 30 | 30 | $90 | Separate work after the meeting |
| Rejected-claim total | 240 | 60 | $480 | Same total as the aggregate ledger |
The two-day wait for the service owner contributes zero active minutes. It can still matter to delivery, so record elapsed delay separately. The meeting's $90 already includes both people: adding a further $90 “coordination cost” would count the same work twice.
Suppose a configuration change suppresses six recurring raw duplicates but also excludes the service containing one of the supported claims. The dashboard gets quieter and the apparent labor cost falls, yet the decision quality worsens. Record both the saved minutes and the lost scope. A better adjustment might deduplicate the repeated result while keeping the service in review. After the change, rerun the same selected code, then inspect both alert workload and prepared known cases. A missing alert alone cannot prove noise was removed safely.
The calculation sheet for a real pilot needs these columns:
run_id | revision | repository | raw_alert_id | underlying_claim_id
scope_state | reviewer | activity | active_minutes | disposition
decision_evidence | owner | hourly_rate_assumption | direct_charge
Keep technical evidence in an access-controlled system; the sheet can refer to record IDs. A security lead can publish aggregate labor and uncertainty without circulating sensitive source excerpts.
Decide how much work an intervention can recover
Willow's illustrative two-week ledger contains $760 of labor associated with rejected claims, duplicates and out-of-scope work. That does not mean a configuration change can recover $760. Some rejected claims still require careful investigation. Some repeated alerts take only a moment to recognize. A change can also create maintenance work or remove a useful check.
The team chooses one bounded intervention: stop repeatedly reopening already-linked duplicate tickets while preserving the underlying issue and assessed scope. The existing duplicate-handling row contains 1.5 AppSec hours and 0.5 developer hours. At the example's assumed rates of $100 and $80 per hour, that is:
(1.5 × $100) + (0.5 × $80) = $190 per observed two-week period
For planning only, suppose the intervention reduces those hours to 0.5 and 0.25 while adding 0.25 AppSec hours of maintenance each period. It also takes two AppSec hours to configure and review once. These are assumptions to check in a repeat period, not measured savings.
| Labor component | Original period | Proposed recurring period |
|---|---|---|
| AppSec duplicate handling | 1.50 hours | 0.50 hours |
| Developer duplicate handling | 0.50 hours | 0.25 hours |
| Additional AppSec maintenance | 0 | 0.25 hours |
| Recurring labor value at assumed rates | $190 | $95 |
The proposed recurring reduction is $95 per comparable two-week period. The one-time setup labor is $200. At those assumptions, the labor-value break-even point is a little over two periods; it is reached during the third comparable period. In the first period, setup plus the new recurring work totals $295, which is $105 more than the original $190. Reporting only the eventual saving would hide that initial cost.
The conclusion changes when recurrence changes. Keep the same staff mix and assume handling savings scale with duplicate volume, while the $25 maintenance cost stays fixed:
| Duplicate volume compared with the observed period | Handling labor avoided before maintenance | Net recurring labor value recovered | Setup break-even |
|---|---|---|---|
| Half | $60 | $35 per period | During period 6 |
| Same | $120 | $95 per period | During period 3 |
| Double | $240 | $215 per period | During period 1 |
This simple sensitivity table is a planning model. Real handling time may not scale linearly; record that assumption rather than calling the range a forecast. Lower labor value usually means staff capacity available for other work, not an automatic reduction in payroll expense.
The AppSec lead can now make a bounded decision: spend two hours on the intervention only if repeat duplicates are likely to continue and the team can check that the same services and known cases remain assessed. Re-measure the activity log in the next period. If a supposed improvement came from excluding the noisy service, report the lost coverage and do not attribute the whole reduction to better deduplication.
Keep the other $570 of the original noise-related subtotal separate. This intervention has not shown that any of it can be removed. Likewise, the unresolved $140 remains unresolved; a later verdict may move its accounting category but cannot erase time already spent.
Check the next period before expanding the change
Keep the same repository set, scope rules and activity definitions for the follow-up. Compare duplicate handling, unresolved work and known-case results alongside total labor. If the code or rule set changes, record it; the before/after difference may have more than one cause.
The decision record should say: “Approve two AppSec hours to change duplicate handling for these two repositories. Re-measure after two comparable periods. Continue only if the handling reduction exceeds maintenance and the assessed scope and known-case outcomes remain acceptable.” Give the original issue records stable links so deduplication does not erase evidence.
Start with one recurring source of work that the ledger can identify. For a purchase, carry the measured labor into the SAST comparison and the pilot decision memo. Keep recovered staff capacity separate from an actual budget reduction.