KubeTW All articles
Architecture & Strategy

Why Kubernetes Hates Your Database (And What to Do About It)

KubeTW
Why Kubernetes Hates Your Database (And What to Do About It)

Let's be honest: Kubernetes is kind of a jerk to your database. Not intentionally—it's just that everything about the way K8s thinks about workloads (ephemeral, replaceable, stateless) is basically the opposite of what a database needs (persistent, careful, consistent). And yet, teams keep trying to force the relationship, usually with painful results.

This isn't a hit piece on running stateful workloads in Kubernetes. It absolutely can work. But it requires dropping some assumptions that work fine for your Node.js API and don't work at all for Postgres.

Why Stateless Apps Feel Right at Home

When you deploy a stateless service—say, a REST API or a frontend server—Kubernetes is genuinely great at its job. Pod dies? Reschedule it. Need more capacity? Scale out. Rolling update? No problem, just swap containers. The app doesn't care which node it lands on or whether it's the same instance that handled the last request.

This is the happy path K8s was designed for. The scheduler treats every pod like a fungible unit of compute, and for stateless workloads, that's exactly right.

Stateful services break this mental model almost immediately. A database pod isn't fungible. It has data. It has an identity. If Postgres gets rescheduled to a different node and its persistent volume doesn't follow, you've got a bad time. If two replicas both think they're the primary, you've got a worse time.

The Three Ways Teams Usually Get This Wrong

Treating PersistentVolumes like magic. PersistentVolumes solve the "my data disappears when the pod dies" problem, but they introduce their own complexity. Volume binding modes, storage classes, ReadWriteOnce vs ReadWriteMany—these aren't just config knobs, they have real implications for availability and performance. A lot of teams slap a PVC on a database deployment and call it a day, then wonder why failover doesn't work the way they expected.

Using Deployments instead of StatefulSets. This one's common with teams that are newer to K8s. Deployments are great, but they don't give pods stable network identities or ordered startup/shutdown. StatefulSets exist specifically for workloads that need both. If your database cluster needs to know that db-0 is always the primary and db-1 is always a replica, a Deployment won't reliably give you that. StatefulSets will.

Skipping operator patterns. Raw StatefulSets get you part of the way there, but they don't know anything about your database's internal logic—things like leader election, failover promotion, or backup scheduling. That's what operators are for. The Postgres Operator from Zalando, the MongoDB Community Operator, the Strimzi Kafka Operator—these exist because running a database in K8s isn't just a scheduling problem, it's an operational one.

A Real-World Failure Worth Learning From

Here's a scenario that plays out more often than anyone wants to admit. A team migrates a MySQL instance into their K8s cluster. They use a Deployment (first mistake), attach a PVC, and everything looks fine in staging. In production, the node running MySQL gets drained for maintenance. Kubernetes reschedules the pod, the PVC re-attaches, and... the app can't connect for four minutes while the volume mounts and MySQL initializes. No one had tested that path.

Then, six weeks later, they try to scale to two replicas for "high availability." Except MySQL with a ReadWriteOnce volume can't actually be scheduled on two nodes simultaneously. Now they have a pending pod and a confused on-call engineer at 11pm on a Friday.

None of this is mysterious in hindsight. But it happens because the team applied stateless thinking to a stateful problem.

When to Containerize, When to Use Managed Services

Here's the honest framework: running a database in Kubernetes makes sense when you have operational maturity to match the complexity, when you need tight integration with your existing K8s-native tooling, or when cost constraints make managed services a non-starter.

For most teams—especially smaller ones—managed services are the right answer for primary databases. RDS, Cloud SQL, ElastiCache, Amazon MSK—these services handle the operational burden that K8s operators only partially abstract away. You still need to think about backups, replication, and failover, but you're not also managing the scheduling layer.

Where running stateful workloads in K8s starts to make more sense:

Practical Patterns That Actually Work

If you're committed to running stateful workloads in K8s, here's what the architecture should look like:

Use StatefulSets with headless services so pods get stable DNS names. Your database cluster members need to find each other reliably, and db-0.db-service.default.svc.cluster.local is a lot more predictable than whatever IP gets assigned to a Deployment pod.

Invest in a storage class that fits your workload. For databases, you generally want low-latency block storage (EBS gp3, GCE Persistent Disk SSD) rather than network-attached file storage. The difference in I/O performance is significant.

Use pod disruption budgets to prevent Kubernetes from doing something well-intentioned but catastrophic, like draining two nodes simultaneously during a cluster upgrade and taking out your entire database quorum.

Deploy via an operator whenever one exists for your technology. Operators encode the operational knowledge that StatefulSets alone don't have. They're not perfect, but they're a lot better than writing your own runbooks and hoping the on-call engineer reads them.

And please—test your failure scenarios. Drain a node. Delete a pod. Simulate a volume detach. If you haven't seen how your database recovers from these events in a controlled environment, you will eventually see it in production, and that's a much worse time to learn.

The Bottom Line

Kubernetes isn't actually hostile to stateful workloads—it's just indifferent to them. It doesn't know or care that your database has opinions about identity and persistence. That context has to come from you, through the right primitives, the right operators, and the right architectural decisions.

The teams that succeed with stateful K8s workloads aren't the ones who are most optimistic about what the platform can do. They're the ones who are most clear-eyed about where the complexity lives—and who plan accordingly.

All Articles

Keep Reading

From Helm Beginner to Production Engineer: Your Realistic Kubernetes Learning Roadmap

From Helm Beginner to Production Engineer: Your Realistic Kubernetes Learning Roadmap

Burned Out and Overbooked: The Real Cost of the Kubernetes Talent Crisis

Burned Out and Overbooked: The Real Cost of the Kubernetes Talent Crisis

Running Kubernetes Everywhere: A No-Nonsense Guide to Multi-Cloud Deployments in 2024

Running Kubernetes Everywhere: A No-Nonsense Guide to Multi-Cloud Deployments in 2024