Kubernetes Secrets Aren't Actually Secret: Plugging the Leaks Before They Sink Your Cluster

Let's start with a fact that should make every platform engineer at least a little uncomfortable: Kubernetes Secrets are, by default, stored as base64-encoded strings in etcd. Not encrypted. Not hashed. Just encoded — which is about as secure as writing your password in a language you think your coworkers don't speak.

Base64 is reversible in seconds. Anyone with read access to etcd has access to every secret your cluster holds. And yet, teams ship to production every day without ever addressing this, because the word "Secret" in the name implies a level of protection that simply doesn't exist out of the box.

This isn't a niche gotcha. It's one of the most commonly misunderstood parts of the Kubernetes security model, and it shows up repeatedly in post-mortems from companies that should have known better.

The etcd Problem Is Real and Underappreciated

Etcd is the key-value store that backs the entire Kubernetes control plane. Every object your cluster knows about — Pods, Deployments, ConfigMaps, and yes, Secrets — lives there. If an attacker gains access to etcd, they own your cluster. Full stop.

The default behavior is to store Secret values in base64 inside etcd with zero additional encryption. Kubernetes does support encryption at rest through its EncryptionConfiguration API, but you have to explicitly configure it. It doesn't come pre-enabled, not even on managed services like EKS or GKE in their default configurations.

Enabling encryption at rest means creating an EncryptionConfiguration manifest that tells the API server to encrypt Secret data using a provider like AES-CBC, AES-GCM, or — for the most security-conscious shops — a KMS integration that offloads key management to something like AWS KMS or Google Cloud KMS. The KMS approach is worth the extra setup because it removes the key from your cluster entirely, which is the whole point.

If you're running a managed cluster and you haven't explicitly turned this on, go check. Right now. We'll wait.

ConfigMaps Are Not a Safe Fallback

Here's a pattern that's surprisingly common: a team reads about the etcd problem, decides Secrets are too complicated, and starts shoving sensitive values into ConfigMaps instead. The logic being, apparently, that if Secrets aren't truly secret, why not just use the thing that doesn't pretend to be?

This is worse. ConfigMaps have even fewer guardrails than Secrets. They're not base64-encoded (not that encoding helps much), they don't have the same RBAC defaults, and they're far more likely to get logged in full during debugging sessions or printed out in CI/CD pipelines during dry-run operations.

ConfigMaps are the right tool for non-sensitive configuration — feature flags, environment labels, application tuning parameters. The moment you put a database password or an API key in a ConfigMap, you've made a mistake.

Where Secrets Actually Leak in Practice

Etcd access is a serious concern, but in practice, most secret exposure happens at a much more mundane level. Here are the three places teams consistently get burned:

CI/CD Pipeline Logs It's alarmingly easy to accidentally echo a secret in a pipeline. A kubectl describe command on a Pod that has environment variables sourced from a Secret will print those values. A misconfigured debug step that dumps environment variables will do the same. GitHub Actions, GitLab CI, and Jenkins all capture stdout, and that output often gets retained for weeks or months in artifact storage.

Audit your pipelines. Search your log storage for known patterns — things that look like API keys, connection strings, or JWT tokens. You might be unpleasantly surprised by what you find.

Application Logs Applications themselves are often the culprit. A startup error that prints the full configuration object, an unhandled exception that logs request headers, a debug mode that was never turned off in production — all of these can surface secret values in application logs that flow into your centralized logging stack.

RBAC Misconfigurations Kubernetes RBAC has a get and list verb for Secrets. Any service account or user with those permissions on the secrets resource in a namespace can read every secret in that namespace. Audit your role bindings. The principle of least privilege is not optional here.

External Secret Management Is the Real Fix

The honest answer to the Kubernetes Secrets problem is that you probably shouldn't be using native Kubernetes Secrets as your primary secret store for anything sensitive in a production environment.

Tools like External Secrets Operator (ESO) let you define a ExternalSecret resource that pulls values from AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or GCP Secret Manager at runtime and syncs them into native Kubernetes Secrets. This means your source of truth lives outside the cluster in a system purpose-built for secret management, with proper audit trails, rotation support, and access controls.

HashiCorp Vault with the Vault Agent Injector is another popular approach, especially for teams that want a cloud-agnostic solution. The agent sidecar pattern injects secrets directly into the Pod's filesystem, so values never have to live in a Kubernetes Secret object at all.

Both approaches add operational complexity. That's a real tradeoff. But for any organization handling user data, payment information, or regulated workloads, the complexity is justified.

Audit Logging: Know When Something Goes Wrong

No security control is perfect, which is why you need visibility into who's accessing your secrets. Kubernetes API server audit logging can capture every read operation against Secret resources — who made the request, from which IP, and when.

Configuring audit policy to log get and list operations on Secrets at the Metadata or Request level (avoid RequestResponse for secrets to prevent logging the actual values) gives you a trail that security teams can use for incident response and compliance reporting. Ship those audit logs to your SIEM or a log aggregation platform where they'll be retained and searchable.

This isn't glamorous work. It doesn't show up in a demo. But it's the difference between knowing your secrets were accessed and finding out about it from a breach notification.

A Realistic Starting Point

If you're looking at this and feeling overwhelmed, here's a prioritized list:

1. Enable encryption at rest for Secrets in etcd. If you're on a managed service, check the docs — this is usually a checkbox or a configuration flag.
2. Audit your RBAC and remove any overly broad permissions on the secrets resource.
3. Scan your CI/CD logs for leaked values using tools like truffleHog or gitleaks.
4. Evaluate External Secrets Operator or Vault for your most sensitive workloads.
5. Enable API audit logging and route it somewhere useful.

Kubernetes Secrets aren't broken. They're just not doing what most people assume they're doing. Closing that gap between assumption and reality is the work — and it's absolutely achievable with the right controls in place.