Your Cluster Upgrade Is a Ticking Clock—PDBs Are the Only Thing Standing Between You and Data Loss

Here's a scenario that plays out in engineering teams across the country more often than anyone wants to admit: it's a Tuesday morning, your cloud provider starts rolling a node upgrade across your managed Kubernetes cluster, and thirty minutes later your on-call Slack channel is on fire. A stateful workload lost quorum. A batch job died mid-write. Your API is returning 503s because too many pods got evicted simultaneously.

The postmortem will eventually surface the same uncomfortable truth: you had the tools to prevent this. You just didn't configure them.

Pod Disruption Budgets—PDBs—are Kubernetes' built-in mechanism for telling the cluster how much voluntary disruption your workloads can tolerate. They're not complicated to understand. They are surprisingly easy to misconfigure, ignore entirely, or set up in ways that create new problems while solving old ones. Let's fix that.

What "Voluntary Disruption" Actually Means

Before getting into configuration, it's worth being precise about what PDBs protect against—and what they don't.

Kubernetes distinguishes between two kinds of disruption. Involuntary disruption is hardware failure, a kernel panic, a node getting terminated by your cloud provider unexpectedly. PDBs don't help here. Voluntary disruption is anything the cluster chooses to do: draining a node for maintenance, evicting pods to make room during a cluster upgrade, or the Cluster Autoscaler scaling down a node to save money.

PDBs operate exclusively in the voluntary disruption space. When something tries to evict a pod—the kubectl drain command, the eviction API, the autoscaler—it checks whether the eviction would violate any active PDBs. If it would, the eviction is blocked until the budget allows it.

This is powerful. It means your stateful application can tell the cluster: "You can restart me, but only if at least two of my three replicas are healthy first." The cluster will wait. Your data survives.

The Two Knobs and How to Use Them

A PDB gives you two configuration options, and you pick one:

minAvailable sets the minimum number (or percentage) of matching pods that must remain available during disruption. If you have a three-node Kafka cluster and set minAvailable: 2, the eviction API will never take down more than one pod at a time.

maxUnavailable sets the maximum number (or percentage) that can be down simultaneously. It's the inverse framing—often more intuitive for teams already thinking in terms of replica counts.

Both accept integers or percentage strings. minAvailable: "50%" on a ten-replica deployment means at least five pods must stay up. Simple enough.

Here's where people get tripped up: percentages round down for minAvailable and round up for maxUnavailable. On a deployment with three replicas, minAvailable: "50%" means minAvailable: 1—which might be less protection than you intended. Always verify what your percentage actually resolves to at your current replica count.

The Incident Scenarios That Should Scare You

Scenario one: the quorum killer. A team runs a three-node etcd cluster (outside the managed control plane—don't ask). They set minAvailable: 1 thinking "at least one node stays up, we're fine." During a node pool upgrade, the eviction API happily takes down two nodes simultaneously. Etcd loses quorum. The cluster is effectively down until a human intervenes. The fix would have been minAvailable: 2—one line of YAML.

Scenario two: the autoscaler trap. A cost-conscious team enables the Cluster Autoscaler with aggressive scale-down settings. No PDBs anywhere. On a quiet weekend, the autoscaler decides to consolidate workloads and drains three nodes in quick succession. A stateful job writing to a persistent volume loses its pod mid-transaction. The volume is fine; the application state is not. Corrupt data, weekend on-call, bad time.

Scenario three: the PDB that blocks everything. This one goes the other direction. A team sets minAvailable equal to their total replica count—effectively maxUnavailable: 0. No pod can ever be voluntarily evicted. Node drains hang indefinitely. Cluster upgrades stall. The ops team ends up force-deleting pods to unblock maintenance, which defeats the entire purpose and can cause the very data loss they were trying to prevent.

The lesson from scenario three is real: an overly restrictive PDB is almost as dangerous as no PDB. It creates pressure to bypass the protection mechanism entirely.

A Framework for Setting Budgets That Actually Protect You

Instead of guessing, work through these questions for every stateful or availability-sensitive workload:

1. What's the minimum number of replicas needed to serve traffic or maintain consistency? For a quorum-based system, this is usually ceil(n/2) + 1. For a stateless API, it might be one or two. This number becomes your minAvailable baseline.

2. How long can your workload tolerate being at minimum capacity? If the answer is "not long," set up alerting on PDB disruption counts, not just pod availability. You want to know when you're running at the edge of your budget, not after you've fallen off it.

3. Does your PDB selector actually match your pods? This is a gotcha that bites more teams than it should. If your label selector doesn't match the pods you think it does, the PDB is silently doing nothing. Verify with kubectl get pdb -o yaml and cross-reference against kubectl get pods --show-labels.

4. Are you accounting for voluntary evictions from multiple sources? The Cluster Autoscaler, node pool upgrades, and manual drains all go through the same eviction API. Your PDB covers all of them—but make sure your budget accounts for the possibility that multiple maintenance operations could be happening simultaneously.

Don't Forget PodDisruptionBudget v1

As of Kubernetes 1.21, policy/v1beta1 PDBs are deprecated and were removed in 1.25. If you're running manifests that reference the old API version, they'll stop working after an upgrade. Audit your configs now. The policy/v1 spec is functionally identical—it's just a find-and-replace, but you need to actually do it.

The Bigger Picture

PDBs are one part of a broader disruption management story. Pair them with terminationGracePeriodSeconds long enough for your app to actually shut down cleanly, preStop lifecycle hooks that give your load balancer time to drain connections, and readiness probes that accurately reflect whether a pod is actually ready to serve traffic.

None of this is glamorous work. It doesn't show up in a demo. But when your cloud provider rolls a security patch across your node pool at 3 AM on a Saturday, you'll be glad someone on your team took the time to think through exactly how much disruption your workloads can handle—and told Kubernetes in terms it actually understands.

Set your PDBs. Check your selectors. Sleep better.