How to verify a secret is fully rotated after exposure

A rotation is complete when the exposed credential can no longer authorize new access, legitimate consumers use a replacement or another approved identity, surviving derived sessions have been addressed, and the evidence covers the systems in scope. A new secret existing in a vault is only one step. The issuer's status, consumer behavior and available audit trail must agree. If one part cannot be checked, record the rotation as partially verified and say exactly which access might remain.

The acceptance rule depends on credential type. A long-lived AWS IAM access key can be marked inactive and later deleted; an OAuth refresh token, signing key or database password has different revocation and session behavior. Use the issuer's own lifecycle controls and record the exact credential ID. The method below tests rotation acceptance after containment. If the exposure is still unfolding, contain it first; the Git secret response guide gives one first-response path.

Illustrative case: a key exposed through a support log

Illustrative case, not an observed incident: Harbor Analytics discovers that an AWS IAM access key ID and secret value were printed into a troubleshooting log bundle shared with an external support provider. The key, OLD_KEY, belongs to an export service. Operations created NEW_KEY and changed the nightly report export. The same IAM user may also authenticate a monthly archive job and a quarterly restore drill. One export run with NEW_KEY does not show that the other consumers migrated or that sessions issued using OLD_KEY have expired.

The team uses this verification record. It contains credential IDs, never secret values:

This is intentionally stricter than a “rotation completed” checkbox. It separates the access-control fact from service continuity and historical impact.

1. Bind the credential and the exposure window

Input: incident ID, exact issuer account, old credential identifier, known job inventory and the time the key was disabled. Owner: security engineer with credential owner. Output: a rotation scope that cannot be confused with another environment's key. Check: an independent reviewer can identify the issuer record without seeing the secret value.

For an AWS key, record the IAM user, account, key ID and current status. AWS's credential report includes key status and last-used fields. Last-used information is a clue, not complete proof of every API request: the report has aggregation behavior and can show N/A for several reasons. If the exposure predates the available logs, the impact review must state that gap. For other providers, find their equivalent token ID, revocation status and audit event, if offered.

2. Confirm the old credential is disabled at the issuer

Do not infer revocation from a vault update, log redaction or successful export job. Check the provider's own control plane for Inactive, revoked, expired or deleted state, and save the time. AWS's access-key update procedure describes creating a replacement, changing applications, making the old key inactive, checking for unexpected breakage and deleting it. In Harbor's case, security asks the IAM owner to make OLD_KEY inactive while the other jobs are mapped, even if an archive schedule needs repair.

If the old key must temporarily remain active for a critical dependency, label the state replacement in progress, not “rotated.” Record the business approver, compensating restrictions, the reason immediate revocation is not possible and a deadline. Limiting a key's policy can reduce exposure while migration proceeds, but that is containment, not rotation acceptance.

Do not test the leaked value by sending a live authenticated request from a workstation just to see whether it fails. Issuer status is the first check, while a controlled negative request can help verify enforcement after propagation. During an incident, coordinate any such request so its known source and timestamp can be distinguished from suspicious activity. Never distribute the leaked value merely to obtain a demonstration.

3. Prove the replacement reaches each real consumer

Inventory current references across scheduled jobs, applications, local profiles, recovery automation and third-party integrations. Search names and credential IDs where safe; do not search by broadcasting the secret value into more logs. Compare the inventory with observed authentication from each legitimate workload. A single nightly export does not exercise a monthly archive or quarterly restore script. For rarely used jobs, a controlled rehearsal or explicit deferred verification may be needed.

Harbor's nightly report export succeeds with NEW_KEY. The archive job's next normal run is three weeks away, while the restore drill will not run until next quarter. The operations owner may perform authorized rehearsals in a nonproduction account if those rehearsals exercise the same identity-delivery path. If they do not, the production consumers remain unverified until their actual route is checked. A longer-term change may replace stored keys with temporary workload credentials, consistent with AWS IAM best practices; that design work does not itself verify today's rotation.

Decision: if any known consumer still requires OLD_KEY, plan its migration but keep the old key disabled during an active exposure. Do not reactivate a compromised key for convenience without incident approval. Repairing the job with a new, scoped identity is safer than restoring known exposed authority.

4. Check derived sessions and activity separately

An old long-lived key can create temporary credentials before it is disabled. AWS explains that temporary credentials can remain valid until expiry, with separate ways to restrict or revoke their permissions; it also documents revoking role sessions. Harbor's IAM owner should determine whether OLD_KEY assumed the export role or issued sessions during the exposure window and whether those sessions can still act. CloudTrail's userIdentity reference helps connect access key IDs and role session details where the relevant events were recorded.

Review role-assumption events to identify sessions created with OLD_KEY before deactivation, then determine whether each is expired, revoked or denied the old scope. Look for requests after deactivation that reveal a stale consumer or still-effective temporary session. A denied call may identify a legitimate forgotten job; it does not by itself identify an attacker. The absence of a denied call is weak evidence if the job has not run. Historical misuse investigation remains a separate incident task after rotation is verified.

Read a completed acceptance packet

These synthetic acceptance records assume the evidence described was supplied; no AWS operations or job rehearsals were performed for this article.

The quarterly restore drill initially looked like a forgotten consumer because its runbook named the export service. The owner supplies its current configuration and authenticated run identity, showing that it uses an independent recovery role and never reads OLD_KEY. Remove it from the migration list with that evidence, not because its schedule is inconvenient. Its separate role still belongs in the organization's identity inventory.

The archive rehearsal must prove more than “a script ran.” If an engineer manually injects NEW_KEY while the scheduler still points to OLD_KEY, the rehearsal bypasses the very path being verified. In the completed packet, the scheduler's identity-delivery configuration and resulting principal are part of the evidence, and the synthetic archive operation succeeds through that route.

A credential-report last_used=N/A does not remove a consumer. AWS's credential-report documentation describes several meanings and reporting limits. In the initial packet, N/A coexists with a job that has not reached its monthly schedule. It supplies no proof that the job migrated or never depended on the key.

The closeout reads:

OLD_KEY is inactive in the named issuer account. Known derived sessions are expired or denied their former authority under the recorded containment evidence. Nightly export and monthly archive use approved replacement identities through their actual delivery paths. The restore drill was excluded from this credential's consumer scope using configuration and principal evidence. Active distribution references were updated. Historical misuse and external copies remain separate incident questions.

The owner should record evidence times, policy propagation observations and log gaps. A known uncontained session means failure; an unknown session route means incomplete evidence. Those are different operational states, even if both prevent a blanket “fully rotated” claim.

The record should include a timestamp and how long the available logs were retained, so a later reviewer does not mistake an expired audit window for a clean result. “No longer detected” is not a fourth state. A pass means the rotation met its stated scope; it does not prove that no unauthorized use happened during the exposure window or that every external copy of the value was erased.

Check that rollback cannot restore the exposed value

Consumer verification should include the configuration that a rollback or recovery action would select. Harbor's nightly job may use NEW_KEY today while an older scheduler definition, recovery archive or deployment variable still refers to OLD_KEY. The old key remaining inactive prevents that reference from restoring authority, but the stale reference can still break recovery and encourage someone to reactivate it during an outage. Record and correct those references without copying the value into the review.

Ask the release owner which configuration would be restored if the current export release failed. The answer should identify an approved identity-delivery path, not simply “the previous version.” If the previous application artifact is otherwise acceptable, its credential configuration can still need a separate update. Keep application rollback and credential rollback as different decisions.

When a later job fails authentication, preserve the failed principal identifier and time, then compare them with the rotation ledger. A legitimate stale job needs repair; a request that cannot be attributed to an expected consumer needs incident investigation. Neither case justifies silently relaxing the acceptance criteria. The already established containment result remains valid if the old authority is still disabled, while the newly discovered consumer reopens the migration scope.

This makes “verified within named scope” a maintainable statement. It identifies what was checked, retains a way to notice omitted consumers, and prevents operational pressure from turning a completed replacement into renewed exposure.

Take one recently closed rotation and reconcile its issuer state, known derived sessions and consumer inventory. Record containment and migration separately: an unexercised archive job can leave service recovery unverified even after the exposed authority is disabled. If the exposure began in Git, the earlier guide on responding to a Git leak covers containment and history cleanup. If a CI job is one of the consumers, review its token scopes before granting the replacement the old permissions.