Version: v2

Session Cookie Encryption & Key Rotation (Internal)

Overview

auth-service (wallet-auth-service) issues a browser session credential — the __Host-wallet_session cookie — after a successful HSID (OIDC) authentication or guest checkout handshake. The cookie's value carries the checkout session ID, merchant ID, customer ID, and issue/expiry timestamps, so it is encrypted (not just signed) before it ever reaches the browser. The same encryption is reused for the short-lived OIDC state parameter.

Encryption is AES-256-GCM, wrapped as a compact JWE (alg=dir, enc=A256GCM) via nimbus-jose-jwt. The key lives in Azure Key Vault (never in source code or static app config) as the secret HSID-JWE-SECRET. Every JWE stamps the exact Key Vault secret version used to encrypt it into its kid header, so decryption always resolves the precise key version from the token itself — this is what makes zero-downtime key rotation possible: auth-service can start encrypting new cookies with a new key version while old, still-valid cookies encrypted with the previous version keep decrypting correctly until that version is retired.

This replaced an earlier design that used two static, config-injected secrets (jwe-secret / previous-jwe-secret) and guessed which of the two applied on decrypt. Implemented under ticket CC-27017.

Attribute	Value
Name	`__Host-wallet_session` (`Constants.WALLET_SESSION_COOKIE_NAME`)
`HttpOnly`	`true`
`Secure`	`true`
`SameSite`	`Lax`
`Path`	`/`
`Max-Age`	Time remaining until the checkout session's `expiresAt`
Value	Compact JWE (`alg=dir`, `enc=A256GCM`) of the JSON payload below

The __Host- cookie name prefix is itself a browser-enforced security control: it requires Secure, forbids a Domain attribute, and requires Path=/, so the cookie can never be set or overridden by a subdomain or a non-HTTPS response.

Encrypted payload (`SessionCookiePayload`)

{
  "sessionId": "<checkout session UUID>",
  "merchantId": "<merchant UUID>",
  "customerId": "<customer UUID, nullable for guest>",
  "issuedAt": "<ISO-8601 instant>",
  "expiresAt": "<ISO-8601 instant>"
}

On every protected request, SessionCookieService.validateSessionCookie decrypts the cookie and checks:

expiresAt has not passed,
customerId, sessionId, and merchantId claims match the resolved checkout session/request context.

A decrypt failure (bad/disabled key version, tampered/malformed JWE) and a claim mismatch both surface as AuthorizationException → HTTP 401, using the same legacy error messages as before this rework.

Architecture

+----------------------------------------------------------------------------------+
| auth-service                                                                      |
|                                                                                    |
|  Issue / validate path (per-request)                                              |
|    SessionCookieService.buildEncryptedSessionCookie / validateSessionCookie        |
|      -> CredentialService.encrypt / decrypt (Mono, offloaded to boundedElastic)    |
|        -> SessionCookieKeyCache                                                   |
|          - currentVersion(): TTL-cached "latest version" pointer (1m default)      |
|          - getKeyForVersion(kid): per-version key material, cached forever         |
|            -> KeyVaultSecretClient (Azure or Local, by Spring profile)             |
|                                                                                    |
|  Rotation path                                                                     |
|    SessionCookieKeyRotationScheduler                                              |
|      - checkAndRotate() [twice-daily cron]                                        |
|          -> SessionCookieKeyRotationService.rotate("SCHEDULER")                   |
|             -> AdvisoryLockService (Postgres advisory lock, cluster-wide singleton)|
|             -> KeyVaultSecretClient.pushNewSecretVersion                          |
|      - checkAndDisableExpiredVersions() [every-minute cron, no lock needed]        |
|          -> SessionCookieKeyRotationService.disableExpiredPreviousVersions        |
|             -> KeyVaultSecretClient.listVersions / disableSecretVersion           |
|             -> SessionCookieKeyCache.invalidateVersion                           |
|                                                                                    |
|  External rotation path (Security-triggered emergency rotation)                    |
|    Security pushes a new HSID-JWE-SECRET version directly in Key Vault            |
|      -> Azure Event Grid: Microsoft.KeyVault.SecretNewVersionCreated              |
|        -> KeyVaultSecretEventConsumer.handle                                      |
|          -> SessionCookieKeyRotationService.onExternalVersionDetected             |
|             -> SessionCookieKeyCache.refresh()                                    |
+----------------------------------------------------------------------------------+

There is no in-app "force rotation" endpoint. An early version of this feature added POST /v1/session/encryption/key-rotation, but it was removed in review because this service's own public Ingress routes every path (config/helm/templates/ingress.yaml), so the endpoint could not actually be restricted to cluster-internal callers as intended. Emergency rotation is Security pushing a new secret version straight to Key Vault — auth-service already owns the RBAC/network wiring to react to that via Event Grid, so no new application-level attack surface was needed.

Pushing a new version is not the same as revoking the old one — see Incident Response: Revoking a Compromised Key below, which is the part of the emergency path that actually matters under compromise pressure.

Libraries & Dependencies

Resolved versions as declared/managed in wallet-auth-service's pom.xml (Spring Boot 4.1.0 parent BOM + azure-sdk-bom 1.2.27):

Library	Version	Used for
`com.nimbusds:nimbus-jose-jwt`	`9.37.2`	JWE construction/parsing (`JWEObject`, `JWEHeader`) and the `DIR`/`A256GCM` encrypt/decrypt primitives (`DirectEncrypter`/`DirectDecrypter`) in `CredentialService`.
`com.azure:azure-security-keyvault-secrets`	`4.8.6`	`SecretClient` — the actual Key Vault REST calls in `AzureKeyVaultSecretClient` (get/set/list/disable secret versions).
`com.azure:azure-identity`	`1.12.2`	`DefaultAzureCredentialBuilder` with managed-identity client ID — how the pod authenticates to Key Vault (no client secret).
`com.azure:azure-messaging-eventgrid`	`4.24.0`	`EventGridEvent` deserialization in `KeyVaultSecretEventConsumer` for the `SecretNewVersionCreated` webhook.
`com.github.ben-manes.caffeine:caffeine`	`3.2.4`	The two bounded caches inside `SessionCookieKeyCache` (per-version key material, and the negative-result cache for invalid `kid`s).
`org.postgresql:postgresql`	`42.7.11` (runtime)	JDBC driver behind `AdvisoryLockService`'s `pg_try_advisory_lock`/`pg_advisory_unlock` calls.
`io.projectreactor:reactor-core`	`3.8.6`	The reactive (`Mono`) API `CredentialService`, `SessionCookieKeyRotationService`, and `AdvisoryLockService` expose, with blocking Key Vault/JDBC calls offloaded onto `Schedulers.boundedElastic()`.
`com.fasterxml.jackson.core:jackson-databind` (via `spring-boot-starter-webflux`)	`2.21.4`	Serializes/deserializes the `SessionCookiePayload` JSON that gets encrypted into the cookie, and parses Event Grid event data.

Key Components

Class	Responsibility
`CredentialService`	Encrypts/decrypts the compact JWE. Stamps the encrypting key's Key Vault version into `kid`; on decrypt, resolves the version from `kid` rather than trying a fixed set of keys.
`SessionCookieKeyCache`	Per-pod cache: the "current version" pointer (TTL-refreshed) and per-version key material (cached forever, since Key Vault versions are immutable). Hardens `getKeyForVersion` against an attacker-controlled `kid` — format validation, a bounded/short-TTL negative-result cache, and bounded cache sizes.
`KeyVaultSecretClient` (+ `AzureKeyVaultSecretClient` / `LocalKeyVaultSecretClient`)	Abstraction over Key Vault secret operations: get a version's value, get/create the current version, disable a version, list all versions. `AzureKeyVaultSecretClient` runs everywhere except the `local` profile; `LocalKeyVaultSecretClient` is an in-memory stand-in for local dev.
`SessionCookieKeyRotationService`	Owns rotation: generates a new AES-256 key and pushes it as a new Key Vault version when due; disables previous versions once their grace period has elapsed; reacts to externally-detected rotations (Event Grid).
`SessionCookieKeyRotationScheduler`	Two cron schedules: a rotation due-check and a grace-period expiry sweep (see Key Rotation).
`KeyVaultSecretEventConsumer`	Handles `Microsoft.KeyVault.SecretNewVersionCreated` Event Grid notifications (plus the Event Grid subscription-validation handshake) for near-instant pickup of a rotation, instead of waiting on the poll/TTL.
`AdvisoryLockService`	PostgreSQL `pg_try_advisory_lock`-based, cluster-wide, non-blocking lock. Serializes concurrent pods so only one pushes a new key version per rotation. Lock resource: `auth-service:session-cookie-key-rotation`.
`SessionCookieEncryptionSafetyValidator`	Fails startup (all profiles except `local`) if `grace-period` is too short relative to `latest-version-ttl` + the longest-lived protected payload's TTL — see Key Rotation.
`SessionCookieDecryptCli` (`src/test`, dev tooling only)	Decrypts a cookie/JWE outside a running instance for local debugging; never ships in the deployable jar.

Key Vault Configuration

Vault URLs (by environment)

Environment	Key Vault URL
dev / test / stage / reg / perf	`https://fcc-comn-chkt-kv-dev.vault.azure.net/`
prod	`https://ccg-comn-chkt-kv-prod.vault.azure.net/`

Configured via the CCG-AZURE-KEYVAULT-URL env var (per-environment values-*.yaml), bound to ccg.azure.keyvault.url. AzureConfig builds the Key Vault SecretClient from this URL using DefaultAzureCredentialBuilder with the pod's managed identity (ccg.azure.credential.managedidentity.clientid) — no client secret involved.

Secrets / variables

Name	Type	Purpose
`HSID-JWE-SECRET`	Key Vault secret (versioned)	The session cookie / OIDC state encryption key itself. Base64-encoded, exactly 32 bytes (256-bit) decoded. Bound as `oidc.hsid.encryption.secret-name` (a literal secret name, not an injected value — the app reads the key material directly from Key Vault via the SDK, not from an env var).
`HSID-JWE-SECRET-ROTATION-SECONDS`	Key Vault secret, optional	Ops-editable override of the rotation cadence (`oidc.hsid.encryption.rotation-interval-override-seconds`), in seconds. Absent/blank/non-numeric/non-positive falls back to the configured `rotation-interval` (90 days) — a hand-edited typo must degrade safely, not crash-loop the pod. Read once at startup; a pod restart is required to pick up a change.
`HSID-JWE-LEGACY-VERSION`	Key Vault secret, optional	One-time cutover value (`oidc.hsid.encryption.legacy-version`): the exact Key Vault version that was current immediately before this feature's rollout, used as a decrypt fallback for `kid`-less cookies issued by the predecessor implementation. Unset in every environment today — only ever set for the single cutover deploy, then unset again.
`CCG-AZURE-KEYVAULT-URL`	Plain env var (per environment, in `values-*.yaml`)	Key Vault endpoint URL — see table above.

Application properties (`oidc.hsid.encryption.*`, `config/helm/application.yaml`)

Property	Default	Meaning
`secret-name`	`HSID-JWE-SECRET`	Key Vault secret name backing the key.
`latest-version-ttl`	`1m`	How long the cached "current version" pointer is trusted before re-checking Key Vault. Per-version key material itself is cached forever (versions are immutable).
`rotation-interval`	`90d`	Scheduled rotation cadence — a `Duration` (e.g. `90d`, `15m`, `30s`), not day-only, so lower environments can exercise rotation on a short cycle.
`rotation-interval-override-seconds`	unset → falls back to `rotation-interval`	Key Vault-backed override, see `HSID-JWE-SECRET-ROTATION-SECONDS` above.
`rotation-check-cron`	`0 0 2,14 * * *` (twice daily)	How often the rotation-due check runs. A sub-day `rotation-interval` only has effect if this is also shortened — dev overrides both (see below).
`grace-period`	`16m`	How long a superseded key version stays enabled/decryptable after being rotated away, so in-flight cookies/state remain valid. Must be ≥ `latest-version-ttl` + the longest-lived protected payload TTL (checkout session cookie or OIDC state, both 15m today) — enforced at startup by `SessionCookieEncryptionSafetyValidator`.
`grace-period-check-cron`	`0 * * * * *` (every minute)	How often the grace-period expiry sweep runs (disables any enabled, non-current version past its grace period).
`legacy-version`	unset	See `HSID-JWE-LEGACY-VERSION` above.

advisory-lock.max-lease-duration (default 20m) is a separate, safety-net timeout: if a rotation holding the advisory lock hasn't finished within this window, the lock is force-released so a stuck pod can't block rotation on every other pod indefinitely.

Dev-only cadence override

values-dev.yaml overrides OIDC_HSID_ENCRYPTION_ROTATION_CHECK_CRON to run every 10 minutes (was every minute during initial live-rotation testing) as a standing config choice — not a one-off to revert — so the full Key Vault/RBAC/Event Grid rotation path can be re-exercised on demand in dev without a redeploy. dev's HSID-JWE-SECRET-ROTATION-SECONDS can then be set to a short value (e.g. a few minutes) to make a rotation actually become "due" within a reasonable testing window.

Secret format

The Key Vault secret value (every version) must be a Base64-encoded 256-bit (32-byte) AES key:

openssl rand -base64 32

SessionCookieKeyRotationService generates new versions this exact way (SecureRandom, 32 bytes, Base64) when it rotates.

Local development

The local Spring profile uses LocalKeyVaultSecretClient, an in-memory stand-in with no real Key Vault access — versions are just an incrementing local-v<N> counter, so the rotation scheduler can still be exercised end-to-end locally.

Env var	Purpose
`SESSION_COOKIE_ENCRYPTION_LOCAL_KEY_BASE64`	Base64-encoded 256-bit AES key seeded as the initial local key version. Generate with `openssl rand -base64 32`.

Key Rotation

Scheduled rotation

SessionCookieKeyRotationScheduler.checkAndRotate() runs on rotation-check-cron (twice daily by default) and calls SessionCookieKeyRotationService.rotate("SCHEDULER"):

Acquires the auth-service:session-cookie-key-rotation Postgres advisory lock (non-blocking; if another pod already holds it, this pod logs KEY_ROTATION_SKIPPED and does nothing further).
Refreshes the cached current version and checks its age against the effective rotation interval (rotation-interval-override-seconds if set and valid, else rotation-interval).
If due, generates a new 256-bit AES key and pushes it as a brand-new Key Vault secret version (pushNewSecretVersion), then refreshes the local cache.

There is no forced/emergency variant inside this service — see Rotation Triggers above for how an emergency rotation actually happens (Security pushing directly to Key Vault).

Grace-period sweep

SessionCookieKeyRotationScheduler.checkAndDisableExpiredVersions() runs on grace-period-check-cron (every minute by default) and calls disableExpiredPreviousVersions(). This needs no lock — disabling an already-expired version is idempotent — and runs independently on every pod:

Lists every version of the secret from Key Vault (Key Vault never deletes old versions on its own).
For each enabled, non-current version, derives when it was superseded (the creation time of the next-newer version — not the version's own age) and disables it once that supersession age reaches grace-period.
Any version already disabled — by this pod's own earlier sweep, another pod, or Security disabling it directly — is unconditionally evicted from this pod's local cache, so every pod converges within one sweep interval.

Grace is measured from supersession time, not creation time, specifically because the primary 90-day scheduled rotation would otherwise disable the outgoing version almost immediately (it's already 90 days old the instant it stops being current) — defeating the whole point of the grace period.

Cross-pod convergence

A rotation pushed by any means (scheduled push, or Security's emergency push) fires a Microsoft.KeyVault.SecretNewVersionCreated Event Grid event, picked up by every pod's KeyVaultSecretEventConsumer for near-instant currentVersion() refresh.
A pod that misses that notification (webhook misconfigured/down) still self-heals within latest-version-ttl (default 1 minute), since SessionCookieKeyCache.currentVersion() re-checks Key Vault once its TTL expires.

Session Continuity During Key Rotation — Scenario Matrix

The core requirement this whole design exists to satisfy: a session/state encrypted before a rotation must keep decrypting correctly for its full remaining lifetime, regardless of how many rotations happen while it's alive, or which pod happens to serve the request. Every scenario below is exercised by an actual test — this is not aspirational.

#	Scenario	What happens	Why it's safe	Verified by
1	A session is encrypted with the current version; no rotation happens before it expires.	Decrypts normally for its full TTL.	Trivial baseline — no version change involved.	`SessionCookieServiceCryptoRoundTripTest`
2	A session is encrypted with V1; V1 is scheduled-rotated to V2 shortly after; the session is still within its TTL.	Decrypts successfully throughout the grace period, using the exact `kid` stamped at encrypt time.	`kid`-based resolution means decrypt never has to "guess" between current/previous — it asks Key Vault for the exact version, which is still enabled.	`SessionCookieKeyRotationLifecycleTest` (full rotate → decrypt-during-grace → disable → reject cycle)
3	A pod's cached `currentVersion()` pointer is stale (a rotation happened elsewhere, but this pod hasn't refreshed) — it keeps encrypting new sessions with the soon-to-be-superseded V1 for up to `latest-version-ttl` (1m) after V2 became current.	The resulting session, encrypted with V1 at the worst possible moment, must still be decryptable for its entire TTL (15m) after that.	`grace-period` (16m) is sized as `latest-version-ttl + max(payload TTLs)` (1m + 15m = 16m) — exactly covers this worst case with zero slack today. Enforced by a startup check, not a runtime one.	`SessionCookieEncryptionSafetyValidator` + `SessionCookieEncryptionSafetyValidatorTest` (fails startup if the margin is violated)
4	The grace period for V1 elapses; V1 is disabled in Key Vault.	New decrypt attempts for V1 fail with `SecretVersionUnavailableException` → 401. Any session still relying on V1 at this point has, by construction, already exceeded its own maximum possible TTL.	Grace is measured from supersession time (when V2 appeared), not V1's own age — critical since the real 90-day rotation means V1 is already ~90 days old the instant it's replaced.	`disableExpiredPreviousVersions_doesNotDisable_justRotatedVersion_evenThoughItIsNinetyDaysOld`, boundary-parameterized tests at 0/1/5/14 vs. 15/16/30/90 minutes
5	Several rotations happen back-to-back (dev short interval, or an emergency rotation landing right after a scheduled one) before earlier versions' grace periods elapse.	Each version still gets its own full, independent grace period.	Grace is anchored to each version's immediate successor, not to whatever is "current" right now — a chain of N rotations doesn't compress any earlier version's window.	`disableExpiredPreviousVersions_disablesMultipleExpiredVersions_inOnePass`
6	A pod runs its grace-period sweep while its own `currentVersion()` pointer is stale (points at an old version, not the true newest).	The sweep can still never disable the actual newest version by `createdAt`, regardless of what the pod's stale pointer claims is "current".	The newest-by-timestamp version always has no successor in the sort, independent of the `currentVersion` string used only for the skip check.	`disableExpiredPreviousVersions_neverDisablesTheNewestVersion_evenWhenCurrentVersionPointerIsStale`
7	Two versions are created within the same second (Key Vault `createdOn` has 1-second resolution) — e.g. two emergency rotations moments apart.	The predecessor still gets a correctly-ordered successor and a real grace deadline, instead of never expiring.	Sort ties are broken by placing the pod's believed-current version last.	`disableExpiredPreviousVersions_disablesVersion_whenItTiesOnTimestampWithTheCurrentVersion`
8	A version is disabled by a different pod, or by Security directly in Key Vault (not by this pod's own sweep).	This pod's local cache still forgets it within one of this pod's own sweep intervals (≤1 minute), converging without needing to be the one that performed the disable.	The "already disabled → invalidate" check is unconditional on every sweep, independent of who disabled it.	`disableExpiredPreviousVersions_invalidatesCache_forAVersionAlreadyDisabledByAnotherPod`
9	A version's disable call fails (Key Vault error, e.g. throttling).	The version stays enabled and stays in the local cache — not evicted — and is retried on the next sweep.	Only a confirmed disable should evict; a failed API call must not pretend the version is gone.	`disableExpiredPreviousVersions_doesNotEvictFromKeyCache_whenDisableCallFails`, `_continuesProcessingOtherVersions_whenOneDisableCallFails`
10	A `kid`-less cookie shows up (issued by the pre-CC-27017 static-secret implementation) shortly after this feature's cutover deploy.	Falls back to the ops-configured `legacy-version` for one cookie/state lifetime, then the fallback path stops being exercised as those old cookies expire.	Explicit, ops-verified fallback rather than an auto-detected heuristic that could misresolve.	`CredentialService` legacy-`kid` tests; `LEGACY_KID_LESS_COOKIE_*` log events

Known limitations

Zero-slack margin today. With current defaults, grace-period (16m) exactly equals the required minimum (latest-version-ttl 1m + checkout-session/OIDC-state TTL 15m) in every environment — there is no extra buffer. This is intentional (see the code comment history on this value), and it's fail-safe: SessionCookieEncryptionSafetyValidator fails startup if AUTH_CHECKOUT_SESSION_EXPIRY_IN_MINUTES or the OIDC state TTL is ever increased without a matching grace-period increase. But it means any change to those TTLs must be paired with a grace-period change in the same PR, not left to a later cleanup.
Per-pod convergence, not instantaneous. A pod's own disablement/invalidation depends on that pod's own grace-period-check-cron continuing to fire. In practice this is extremely reliable (Spring's @Scheduled keeps firing on schedule even if a single execution throws, and every relevant method here already catches and logs its own exceptions), but it's worth knowing that the "converges within one sweep interval" guarantee is per pod, not a single global event.
Pushing a new key version does not, by itself, revoke the old one within any bounded time better than the configured grace period. See the next section — this is the one gap worth internalizing before relying on "emergency rotation" as an incident-response tool.

Incident Response: Revoking a Compromised Key

Pushing a new HSID-JWE-SECRET version is not the same operation as revoking the old one, and conflating the two is the single most important pitfall in this design for an actual key-compromise incident.

pushNewSecretVersion (a new Key Vault secret version) makes a new version current for future encryptions. It does not touch the old version's enabled state at all. The old (possibly compromised) version is picked up by the normal grace-period sweep like any routine rotation, and — by design, to protect legitimate in-flight sessions — stays enabled and decryptable for the full configured grace-period (16 minutes by default) after being superseded.
disableSecretVersion (an explicit disable of a specific, existing version) is a separate Key Vault operation. Disabling an existing version does not fire a SecretNewVersionCreated Event Grid event (no new version was created), so pods do not learn about it near-instantly the way they do for a new-version push. Instead, every pod's own disableExpiredPreviousVersions() sweep re-reads each version's enabled state from Key Vault unconditionally, on every tick, before any grace-period math runs — so a version Security force-disables converges across every pod within one grace-period-check-cron interval (1 minute by default), completely independent of the 16-minute grace period.

Practical consequence: if a key is genuinely compromised, Security must do both of the following, not just the first:

Push a new version (az keyvault secret set --name HSID-JWE-SECRET --value <new-base64-key>) so new sessions stop using the compromised key.
Explicitly disable the compromised version (az keyvault secret set-attributes --name HSID-JWE-SECRET --version <compromised-version> --enabled false) so every pod stops honoring it within about a minute, instead of the full grace period.

Step 2 does mean any legitimate, still-valid session encrypted with the compromised version is invalidated early too (forcing re-authentication) — that's the correct trade-off during an actual compromise, and is exactly why this isn't done automatically for routine rotations.

Sequences

HSID callback / guest checkout -> SessionCookieService.buildEncryptedSessionCookie(session)
  -> serialize SessionCookiePayload to JSON
  -> CredentialService.encrypt(bytes)
    -> SessionCookieKeyCache.currentVersion()   [Key Vault on TTL miss]
    -> SessionCookieKeyCache.getKeyForVersion(version)   [Key Vault on cache miss]
    -> JWEObject(header{kid=version}, payload).encrypt(DirectEncrypter)
  -> ResponseCookie(__Host-wallet_session, jwe, HttpOnly, Secure, SameSite=Lax, MaxAge=session TTL)

Protected request -> SessionCookieService.validateSessionCookie(cookie, session, merchantId, customerId)
  -> CredentialService.decrypt(jwe)
    -> parse kid from JWE header (fallback to legacy-version if absent)
    -> SessionCookieKeyCache.getKeyForVersion(kid)   [rejects malformed kid before any lookup]
    -> JWEObject.decrypt(DirectDecrypter)
  -> parse SessionCookiePayload JSON
  -> check expiresAt, sessionId, merchantId, customerId claims

Sequence: Scheduled rotation

SessionCookieKeyRotationScheduler.checkAndRotate() [cron: rotation-check-cron]
  -> AdvisoryLockService.tryLockAndExecute("auth-service:session-cookie-key-rotation")
     lock busy -> KEY_ROTATION_SKIPPED, done
     lock acquired ->
       -> SessionCookieKeyCache.refresh() -> current version + createdAt
       -> age >= effective rotation-interval ?
            no  -> KEY_ROTATION_NOT_DUE
            yes -> generate 256-bit AES key
                -> KeyVaultSecretClient.pushNewSecretVersion(HSID-JWE-SECRET, newKey)
                -> SessionCookieKeyCache.refresh()
                -> KEY_ROTATION_COMPLETED
       -> release advisory lock

Sequence: Emergency rotation (Security-triggered, routine — new key, old key not compromised)

Security -> az keyvault secret set --name HSID-JWE-SECRET --value <new base64 key>
  -> Azure Event Grid: Microsoft.KeyVault.SecretNewVersionCreated
    -> every pod's KeyVaultSecretEventConsumer.handle
      -> SessionCookieKeyRotationService.onExternalVersionDetected("HSID-JWE-SECRET")
        -> SessionCookieKeyCache.refresh()   [new version becomes "current" immediately]
  -> next grace-period sweep disables the old version once its grace period elapses [~16m default]

Sequence: Key compromise (Security-triggered, old key must be revoked immediately)

Security -> az keyvault secret set --name HSID-JWE-SECRET --value <new base64 key>
  -> (as above: every pod adopts the new version as current near-instantly)

Security -> az keyvault secret set-attributes --name HSID-JWE-SECRET --version <compromised-version> --enabled false
  -> No Event Grid event fires (no new version was created)
  -> Every pod's own next grace-period-check-cron tick [~1m default]:
       -> disableExpiredPreviousVersions() lists versions, sees compromised-version.enabled=false
       -> SessionCookieKeyCache.invalidateVersion(compromised-version)   [unconditional, before any grace-period math]
  -> Any session still encrypted with the compromised version is now rejected on every pod within ~1 minute,
     not the full 16-minute grace period — see Incident Response above.

Observability (Rotation Logs)

Structured event= log keys emitted by this feature (key versions are masked to their last 8 characters in every log line):

ROTATION_INTERVAL_RESOLVED — the effective rotation interval logged once at startup.
ROTATION_INTERVAL_OVERRIDE_INVALID — HSID-JWE-SECRET-ROTATION-SECONDS was set but non-numeric/non-positive; fell back to the configured default.
ROTATION_INTERVAL_SHORTER_THAN_GRACE_PERIOD — startup warning (not a failure) when the effective rotation interval is shorter than the grace period.
KEY_ROTATION_STARTED / KEY_ROTATION_COMPLETED — a scheduled rotation actually pushed a new version.
KEY_ROTATION_NOT_DUE — scheduled check ran, but the current version hasn't aged out yet.
KEY_ROTATION_SKIPPED — another pod already held the advisory lock.
KEY_ROTATION_EXTERNAL_VERSION_DETECTED — an Event Grid notification refreshed this pod's cached current version.
KEY_ROTATION_PREVIOUS_VERSION_DISABLED / KEY_ROTATION_PREVIOUS_VERSION_DISABLE_FAILED — grace-period sweep outcome for a specific version.
KEY_ROTATION_GRACE_PERIOD_SWEEP_COMPLETED — a sweep disabled at least one version.
KEY_VAULT_SECRET_VERSION_PUSHED / KEY_VAULT_SECRET_VERSION_DISABLED — raw Key Vault write operations (AzureKeyVaultSecretClient).
LEGACY_KID_LESS_COOKIE_FALLBACK_ATTEMPTED / LEGACY_KID_LESS_COOKIE_REJECTED — a kid-less cookie from before this feature was seen; expected only in a short window around the cutover deploy.
ADVISORY_LOCK_ACQUIRED / ADVISORY_LOCK_BUSY / ADVISORY_LOCK_RELEASED / ADVISORY_LOCK_LEASE_EXPIRED — advisory lock lifecycle.

Error Handling

All encrypt/decrypt failures from CredentialService are wrapped in CredentialEncryptionException (unchecked). SessionCookieService translates these into AuthorizationException (HTTP 401) with legacy messages:

Cause	Result
No cookie / blank credential	`AuthorizationException`: "HSID POST Auth credential is required for IDP_REQUIRED sessions"
Decrypt failure (bad/disabled key version, tampered/malformed JWE)	`AuthorizationException`: "HSID POST Auth credential decryption failed"
Valid JWE, unexpected payload shape	`AuthorizationException`: "HSID POST Auth credential format is invalid"
`expiresAt` has passed	`AuthorizationException`: "HSID POST Auth Session Cookie has expired"
Claim mismatch (`sessionId` / `merchantId` / `customerId`)	`AuthorizationException`: "HSID POST Auth credential `<field>` does not match the request"
`kid` version resolves to a Key Vault 4xx (not found / disabled)	`SecretVersionUnavailableException`, negative-cached for 5 minutes
`kid` version resolves to a Key Vault 5xx or 429 (throttling)	Propagated as-is on every call, never negative-cached (a transient blip must not become a multi-minute outage)
Key material not valid Base64, or not exactly 32 bytes decoded	`CredentialEncryptionException`

SessionCookieDecryptCli (src/test/java/.../tools, never ships in the deployable jar) decrypts a compact JWE outside a running instance, using the same nimbus-jose-jwt classes as CredentialService:

mvn -q test-compile org.codehaus.mojo:exec-maven-plugin:3.1.0:java \
  -Dexec.mainClass=com.optum.wallet.authorization.tools.SessionCookieDecryptCli \
  -Dexec.classpathScope=test \
  -Dexec.args="<compact-jwe> <base64-aes-256-key>"

Both arguments can be supplied via SESSION_COOKIE / SESSION_COOKIE_KEY env vars instead (keeps them out of the process arg list visible via ps/CI logs — this does not hide them from shell history).

The key is the raw Base64 AES-256 value for the Key Vault secret version named in the cookie's kid header:

az keyvault secret show --name HSID-JWE-SECRET --version <kid> --query value -o tsv

The CLI never calls Key Vault itself. Treat any key that has passed through a shell/chat/ticket as compromised — rotate it, don't reuse it.

Testing

SessionCookieKeyRotationServiceTest — rotation due/not-due boundaries, grace-period expiry boundaries, advisory-lock skip path.
SessionCookieKeyRotationSchedulerTest — cron-triggered calls into the rotation/sweep services, error swallowing.
SessionCookieKeyRotationLifecycleTest — end-to-end: a 90-day-old key → real rotate() → decrypt succeeds during the grace period → grace period elapses → version disabled → decrypt rejected, wiring the real CredentialService, SessionCookieKeyCache, and SessionCookieKeyRotationService against an in-memory Key Vault stand-in (no mocking of the classes under test).
AzureKeyVaultSecretClientTest — real Azure SDK SecretClient against a WireMock stub of the Key Vault REST API, verifying the actual get/push/disable/list-versions HTTP contract.
KeyVaultSecretEventConsumerTest / EventGridKeyVaultWebhookIntegrationTest — Event Grid payload handling, including real JSON deserialization through the actual controller route.
SessionCookieDecryptCliTest — the local debugging CLI.

Troubleshooting

A key was just compromised — how do I revoke it right now?

Pushing a new version is not enough by itself — see Incident Response: Revoking a Compromised Key. You must also explicitly disable the compromised version (az keyvault secret set-attributes ... --enabled false) for every pod to stop honoring it within ~1 minute instead of the full grace period.

A rotation didn't get picked up by all pods

Check:

Event Grid subscription for Microsoft.KeyVault.SecretNewVersionCreated on HSID-JWE-SECRET is active (validation handshake logged once when the subscription was created).
KEY_ROTATION_EXTERNAL_VERSION_DETECTED in the logs of the pods that should have reacted.
Even without Event Grid, every pod self-heals within latest-version-ttl (default 1 minute) via its own TTL-based refresh.

An old key version isn't being disabled

Check:

KEY_ROTATION_GRACE_PERIOD_SWEEP_COMPLETED / KEY_ROTATION_PREVIOUS_VERSION_DISABLED logs.
Whether the version has a successor yet — the newest version is never a disable candidate, and grace is measured from when a version was superseded, not from its own createdOn.
KEY_ROTATION_PREVIOUS_VERSION_DISABLE_FAILED for a Key Vault-side error (e.g. RBAC).

Customers are getting logged out / 401s right after a rotation

This should never happen within the configured grace period — if it does:

Confirm grace-period >= latest-version-ttl + max(OIDC state TTL, checkout session TTL) — startup would otherwise fail via SessionCookieEncryptionSafetyValidator, so this points at a live misconfiguration bypassing that check (e.g. the Key Vault rotation-interval override was changed without redeploying, decoupling actual behavior from what the safety validator checked at startup).
Check for LEGACY_KID_LESS_COOKIE_REJECTED — a kid-less cookie from before a cutover deploy, with no legacy-version configured for that transition window.

Overview​

Cookie Details​

Encrypted payload (SessionCookiePayload)​

Architecture​

Libraries & Dependencies​

Key Components​

Key Vault Configuration​

Vault URLs (by environment)​

Secrets / variables​

Application properties (oidc.hsid.encryption.*, config/helm/application.yaml)​

Dev-only cadence override​

Secret format​

Local development​

Key Rotation​

Scheduled rotation​

Grace-period sweep​

Cross-pod convergence​

Session Continuity During Key Rotation — Scenario Matrix​

Known limitations​

Incident Response: Revoking a Compromised Key​

Sequences​

Sequence: Issue session cookie (encrypt)​

Sequence: Validate session cookie (decrypt)​

Sequence: Scheduled rotation​

Sequence: Emergency rotation (Security-triggered, routine — new key, old key not compromised)​

Sequence: Key compromise (Security-triggered, old key must be revoked immediately)​

Observability (Rotation Logs)​

Error Handling​

Local Debugging: Session Cookie Decrypt CLI​

Testing​

Troubleshooting​

A key was just compromised — how do I revoke it right now?​

A rotation didn't get picked up by all pods​

An old key version isn't being disabled​

Customers are getting logged out / 401s right after a rotation​