Designing a Secrets Manager: Sealing, Dynamic Credentials, and the Root of Trust

Every company's secrets start the same way: database passwords in config files, API keys in environment variables, a TLS key someone copied to three laptops. A secrets manager centralizes them — which immediately creates the more interesting problem: you've now built the single most attractive target in your infrastructure, and it has to answer a genuinely philosophical question — what encrypts the encryptor? Designing one (Vault being the reference architecture) is a security-systems interview that stays a systems interview: trust bootstrapping, availability of a thing that must fail closed, and credentials as a lifecycle rather than a string.

The seal: solving "who holds the master key"

Secrets at rest are encrypted with a data encryption key, itself encrypted by a master key — standard envelope encryption, enabling key rotation without re-encrypting the world. The design question is the master key. Vault's answer: at rest, the master key exists nowhere. The server boots sealed — storage present, unreadable. Unsealing reconstructs the master key via Shamir secret sharing: the key was split into N shares (say 5) held by different humans, any K (say 3) suffice.

sealed:   storage ciphertext only; no key material in memory
unseal:   3 of 5 keyholders submit shares -> master key reconstructed in RAM only
auto-unseal: cloud KMS/HSM wraps the master key -> trust moves to the KMS root

No single person can unseal it; compromise of the host at rest yields ciphertext. Auto-unseal (KMS) trades ceremony for availability and relocates the root of trust rather than eliminating it — turtles all the way down until a hardware module; the honest interview answer is "the root of trust is wherever you stop, and you choose where consciously." Seal-on-intrusion is the corresponding kill switch: one command re-seals every node, freezing the blast radius.

Dynamic secrets: credentials as leases, not strings

The deepest idea in the design: stop storing most secrets and start minting them. A static database password shared by forty services is unrotatable in practice and unattributable in an incident. Instead, the secrets manager holds a privileged root credential per backend and generates short-lived, per-consumer credentials on demand:

app requests db access -> manager CREATEs db user 'app-x-7f3a', TTL 1h, grants role
                       -> returns creds bound to a LEASE
lease expires/revoked  -> manager DROPs the user

Every consumer gets unique credentials (audit trail attributes every query), compromise has a TTL, and revocation is real — drop the user, not "rotate and pray." The same lease machinery drives PKI (the manager as internal CA, issuing certificates with hours-long TTLs — which is what makes mTLS-everywhere operationally possible) and cloud credentials (STS-style). The systems consequence: the manager is now in the runtime path of credential issuance, so leases need renewal loops in every client, expiry storms need jitter, and mass revocation (kill every lease from service X) is a first-class operation. Encryption-as-a-service ("transit") rounds out the offering: apps send plaintext, get ciphertext, and key material never leaves the manager — solving the "every service reimplements crypto badly" problem with an API.

Authn, authz, and the bootstrap problem

How does a machine prove identity to get its first secret? The secure introduction problem, and the answer is: lean on an existing trusted attestor — cloud instance identity documents (AWS IAM auth: the platform vouches), Kubernetes service-account tokens (the API server vouches), CI OIDC tokens. The secrets manager verifies the platform's signature and maps identity → policies (path-based, deny-by-default: payments/* readable by payment services only). Human access rides SSO/OIDC with short tokens. The anti-pattern to name: an "app token" baked into an AMI or repo is just the secret-zero problem wearing a costume — the whole point is that identity comes from the platform, not from a pre-shared string.

Operationally, the manager must be more available than everything it feeds — it's in the boot path of every service. Raft-replicated storage, active/standby with fast failover, performance replicas for read scale, and one sharp trade: it fails closed (a sealed/partitioned secrets manager stops issuing — availability engineering here is security engineering), so clients cache leased credentials to ride through brief outages, and lease TTLs set your maximum tolerable manager downtime. Audit logging is non-optional and blocking — if the audit sink is down, Vault refuses operations, a deliberate "no unwitnessed access" stance worth quoting. Break-glass paths (offline recovery keys, printed and vaulted) exist because "we locked ourselves out of everything" is a real failure mode.

Interview probe	Answer sketch
Why not KMS alone?	KMS encrypts blobs; this adds identity brokering, dynamic minting, leases/revocation, PKI, and audit — KMS is the seal, not the product
Manager compromised while unsealed?	Assume key material in RAM is gone: seal the fleet, mass-revoke leases (short TTLs cap the damage), rotate roots — the design's value is that this playbook exists
Secret sprawl into env vars/logs?	Sidecar/agent injection with in-memory files, template rendering, and log-scrubbing; the manager can't fix apps that exfiltrate their own secrets — say the boundary
Multi-region?	Replicated clusters with local read/issue paths; revocation must propagate globally fast — the one operation you never want eventually-consistent

The distilled design: envelope encryption with a consciously-chosen root of trust, platform-attested machine identity, secrets minted as short-lived leases instead of stored strings, and fail-closed availability engineering with a break-glass path. The product isn't storage — it's the ability to answer "who can access what, right now, and how do we make that false in sixty seconds."