Secrets Management Expert

A secret in source control is compromised the moment it is pushed, and it stays compromised after deletion because history is forever. Everything below follows from that.

Core Concepts

The Hierarchy

Ranked by how much damage a compromise causes:

No secret at all — workload identity. The platform proves who the workload is; no credential exists to steal. The target state wherever it is available.
Dynamic, short-lived credentials — minted per session, expiring in minutes. Theft has a short window.
Static secret in a manager, rotated on a schedule, accessed by authenticated identity, audited.
Static secret in an environment variable, injected at deploy.
Secret in a config file on disk.
Secret in source control. Compromised.

Move each secret up this list rather than aiming for perfection everywhere.

Envelope Encryption

Encrypt data with a data key; encrypt the data key with a key that never leaves the KMS. Store the wrapped data key beside the ciphertext.

plaintext --[AES-256-GCM, data key]--> ciphertext
data key  --[KMS master key]---------> wrapped data key   (stored together)

This is what lets you rotate the master key without re-encrypting terabytes, and what keeps the master key material inside a boundary you can audit.

Rotation Only Works If It Is Routine

A rotation procedure exercised once a year fails when you need it during an incident. Rotate on a schedule so the mechanism is known to work — that, not the interval, is the point.

Getting Secrets Out of Code

import os

# Wrong: a default makes a misconfiguration silently insecure
API_KEY = os.environ.get("API_KEY", "dev-key-123")

# Right: fail closed, at startup, with a clear message
def required(name: str) -> str:
    try:
        return os.environ[name]
    except KeyError:
        raise RuntimeError(f"{name} is required and not set") from None

API_KEY = required("API_KEY")

Validate every required secret at startup, not at first use. A service that boots successfully and fails at 3 a.m. on the first payment is worse than one that refuses to start.

Never log them, never put them in error messages, and never pass them as command line arguments — the process table is world-readable.

class Secret:
    """Wrapper that resists accidental disclosure."""
    __slots__ = ("_value",)

    def __init__(self, value: str) -> None:
        self._value = value

    def reveal(self) -> str:
        return self._value

    def __repr__(self) -> str:
        return "Secret(***)"

    __str__ = __repr__

This stops the common leak: an exception rendering a config object, or a structured logger serialising a settings dataclass.

Secret Managers

HashiCorp Vault

# Authenticate as a workload identity, not with a static token
vault write auth/kubernetes/login role=payments jwt="$(cat /var/run/secrets/.../token)"

# Dynamic database credentials: created on request, expire automatically
vault read database/creds/payments-readonly
# username: v-kubernet-payments-x7Kq  password: A1a-...  lease_duration: 1h

Dynamic secrets are the reason to run Vault. The application receives a database user that did not exist a second earlier and will not exist in an hour. A leaked credential expires on its own, and every issuance is attributable.

path "database/creds/payments-readonly" {
  capabilities = ["read"]
}
# No wildcards. A policy granting "secret/*" is not a policy.

Cloud managers

aws secretsmanager get-secret-value --secret-id prod/payments/stripe --query SecretString
az keyvault secret show --vault-name prod-kv --name stripe-api-key --query value
gcloud secrets versions access latest --secret=stripe-api-key

Cheaper to operate than Vault and well integrated with the platform's identity system. Use rotation Lambdas or the managed rotation feature; a secret manager holding a five-year-old static key provides storage, not management.

Cache retrieved secrets in memory with a short TTL. Calling the manager on every request adds latency and cost, and makes the manager a hard dependency of every transaction.

Workload Identity

The best secret is the one that does not exist.

# Kubernetes service account bound to a cloud role - no key material anywhere
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payments-service

# GitHub Actions to AWS via OIDC - no long-lived access keys in CI
permissions:
  id-token: write
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/github-deploy
      aws-region: eu-west-1

Long-lived cloud access keys in CI are among the most frequently leaked credentials in existence. OIDC federation removes them entirely.

Kubernetes

A Kubernetes Secret is base64-encoded, not encrypted. Anyone with read access to the namespace, and anyone with etcd access, has the plaintext.

Options, in increasing strength:

Enable encryption at rest for etcd — the minimum.
External Secrets Operator — syncs from a real manager into the cluster.
Sealed Secrets — encrypted manifests safe to commit; only the controller can decrypt.
CSI Secrets Store — mounts secrets from the manager directly, never creating a Kubernetes Secret object.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: stripe
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets-manager, kind: SecretStore }
  target: { name: stripe-credentials }
  data:
    - secretKey: api_key
      remoteRef: { key: prod/payments/stripe, property: api_key }

Secrets in Git

sops encrypts values while leaving keys and structure readable, so a diff is meaningful and a merge is possible.

# .sops.yaml
creation_rules:
  - path_regex: secrets/prod/.*\.yaml$
    kms: arn:aws:kms:eu-west-1:123456789012:key/abcd-1234
  - path_regex: secrets/dev/.*\.yaml$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

sops --encrypt --in-place secrets/prod/payments.yaml
sops --decrypt secrets/prod/payments.yaml | kubectl apply -f -

Access control becomes KMS policy: revoking someone's key access revokes their ability to decrypt every past commit — which plain file encryption cannot do.

Detection

Scan before the commit, and scan history.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks: [{ id: gitleaks }]

gitleaks detect --source . --redact --log-opts="--all"   # full history
trufflehog git file://. --only-verified                  # verifies liveness

Enable the platform's push protection (GitHub secret scanning, GitLab secret detection) so a secret is blocked at push rather than found afterwards.

When a Secret Leaks

Order matters. Rotation first; cleaning history is cosmetic by comparison.

Rotate immediately. Assume compromise from the moment of exposure. This is the only step that actually removes risk.
Revoke the old credential — do not merely replace it. An unrevoked old key still works.
Audit for use. Search provider logs for use of that credential from unexpected addresses or at unexpected times, over the whole exposure window.
Assess the blast radius. What could that credential reach? Was there data access? Notification obligations may follow.
Then purge history if required (git filter-repo), knowing that forks, clones, caches and CI logs may retain it.
Fix the cause. Add the scanner, the pre-commit hook, the review step.

Deleting the file and force-pushing without rotating is the most common wrong response, and it leaves the secret fully valid.

Best Practices

One secret, one purpose, one environment. Never share a credential between production and staging.
Least privilege on the secret itself — a read-only key where reads suffice.
Short TTLs, dynamic where possible.
Audit every access. Who read which secret, when, from where.
Alert on anomalies: a secret read from a new workload, out of hours, or at an unusual rate.
Automate rotation and verify it works by watching it happen.
Keep a break-glass credential offline, heavily audited, alerting on use.
Never let a secret transit a CI log. Mask it, and check the masking works.

Anti-Patterns

Committed .env files — add to .gitignore before the first commit, and commit a .env.example with empty values.
Default values for secrets in code — turns a misconfiguration into a silent vulnerability.
Base64 mistaken for encryption.
One shared credential for all services — no least privilege, and rotation becomes an outage.
Secrets in container images — they persist in layers and in every registry.
Secrets as command-line arguments — visible in the process table.
Rotation without revocation.
Long-lived cloud keys in CI when OIDC federation is available.

Reference Documentation

Encryption and Key Management — envelope encryption, KMS patterns, key rotation, field-level encryption, and post-quantum migration planning

secrets-management-expert