Pod Placement Roulette: Taking Back Control of Your Kubernetes Scheduler

Here's a scenario that plays out more often than anyone wants to admit: you've carefully configured node selectors, written affinity rules that felt airtight, and sprinkled in a few taints for good measure. You ship your workload, watch the pods spin up, and everything looks fine. Then three weeks later, during a traffic spike, you realize half your latency-sensitive API pods landed on the same overloaded node while two perfectly healthy nodes sat nearly idle across the cluster.

Welcome to Pod Placement Roulette—a game nobody signed up to play.

The Kubernetes scheduler is genuinely impressive engineering. But it operates on a set of priorities and heuristics that don't always align with what your team has in mind. Understanding that gap is the first step toward actually fixing it.

How the Scheduler Actually Makes Decisions

At a high level, Kubernetes scheduling happens in two phases: filtering and scoring. Filtering narrows down the list of eligible nodes based on hard constraints—things like resource requests, node selectors, taints and tolerations, and affinity rules. Scoring then ranks the remaining candidates using a weighted set of priority functions.

The problem is that most teams focus almost entirely on the filtering phase. They write their node selectors, define their requiredDuringSchedulingIgnoredDuringExecution affinity rules, and call it done. But the scoring phase is where things get quietly weird.

By default, the scheduler uses functions like LeastRequestedPriority and BalancedResourceAllocation to spread pods around. These sound sensible in isolation. In practice, though, they're working off requested resources, not actual utilization. A node that's technically undercommitted on paper might be getting hammered in reality, and the scheduler has no idea.

Add in things like pod topology spread constraints that interact unexpectedly with affinity rules, or taints that partially overlap with tolerations in ways you didn't fully think through, and you've got a recipe for placement decisions that feel almost random—even when they're technically deterministic.

The Affinity Rule Trap

Affinity rules are one of the most powerful tools in the Kubernetes scheduling toolkit, and also one of the easiest to misconfigure in ways that are hard to notice.

The classic trap is leaning too hard on preferredDuringSchedulingIgnoredDuringExecution. Preferred affinity is essentially a hint—the scheduler will try to honor it, but if scoring works out differently, your pods end up wherever they land. Teams often write preferred affinity thinking it behaves like a soft guarantee. It doesn't.

Another common footgun: stacking multiple affinity rules without thinking through how they interact under constrained conditions. If you've got zone-aware affinity, node-type affinity, and a pod anti-affinity rule all running simultaneously on a cluster with limited capacity, the scheduler might satisfy two out of three and quietly ignore the third. No error, no warning—just pods in places you didn't expect.

Required affinity solves the ambiguity problem, but it introduces its own risk: if no node satisfies all your constraints, pods sit in Pending forever. You've traded unpredictable placement for potential scheduling deadlock.

Debugging When Placement Goes Wrong

The first tool most engineers reach for is kubectl describe pod, and it's genuinely useful. The Events section will tell you if a pod failed to schedule and why, at least in broad strokes. But for deeper investigation, you want to look at the scheduler's own logs.

If you're running a self-managed cluster, bumping the scheduler's verbosity (--v=10 if you're feeling brave) gives you a firehose of detail about why each node was filtered or scored the way it was. On managed services like EKS or GKE, you're more limited, but the Events API and tools like kubectl get events --sort-by=.lastTimestamp can still surface a lot.

For teams dealing with persistent placement confusion, kube-scheduler-simulator is worth knowing about. It's an open-source project that lets you replay scheduling decisions in a sandboxed environment, which makes it much easier to test configuration changes before rolling them into production.

Also worth checking: kubectl get nodes -o custom-columns with resource capacity and allocatable fields side by side. It's a quick way to spot whether the scheduler's view of available capacity matches your mental model.

Practical Strategies for Deterministic Placement

So how do you actually get control back? A few approaches that work well in practice:

Lean into topology spread constraints. Introduced in Kubernetes 1.18 and now stable, topologySpreadConstraints gives you much more expressive control over how pods distribute across zones, nodes, or custom topology keys. Unlike affinity rules, they're designed specifically for spread scenarios and behave more predictably when capacity is constrained.

Use requiredDuringScheduling sparingly but intentionally. Reserve hard requirements for workloads where placement genuinely matters for correctness—GPU workloads, compliance-scoped nodes, that kind of thing. For everything else, accept that preferred rules are hints and design your architecture to be resilient to imperfect placement rather than trying to force perfection through configuration.

Invest in node labels that reflect your actual topology. The scheduler can only make good decisions based on the metadata it has. If your node labels are stale, inconsistent, or don't capture meaningful distinctions about your infrastructure, your affinity rules are working with bad inputs. A regular audit of node labels pays dividends.

Consider the Descheduler. The Kubernetes Descheduler is an often-overlooked project that runs as a separate component and proactively evicts pods that have drifted into suboptimal placements over time—because of node changes, scaling events, or just scheduler decisions that made sense at the time but don't anymore. It won't fix a broken scheduling configuration, but it's a useful safety net for long-running clusters.

Test your scheduling logic before you need it. This sounds obvious, but most teams never actually validate that their affinity rules produce the distribution they expect under realistic load conditions. Spinning up a staging cluster with representative node diversity and running placement tests before a big deployment is a low-cost way to catch surprises early.

The Bigger Picture

The Kubernetes scheduler isn't broken—it's doing exactly what it's designed to do. The mismatch most teams experience comes from assuming it thinks about placement the way humans do: with context, intent, and an understanding of what actually matters for a given workload.

It doesn't. It runs a deterministic algorithm over the information it has, and that algorithm's priorities aren't always yours.

The good news is that the tooling to close that gap has gotten genuinely good over the last couple of Kubernetes releases. Topology spread constraints, improved scheduler profiles, and better observability all make it more tractable to express your actual placement intent in a way the scheduler can honor.

But none of that helps if you're not regularly auditing where your pods actually land versus where you expected them to. Build that check into your operational practice, and you'll spend a lot less time losing at Pod Placement Roulette.