The Gatekeeper Nobody Configured: Getting Admission Controllers to Actually Do Their Job

You've got RBAC locked down. Network policies are in place. Your team follows the runbooks. And yet, somehow, a container running as root just made it into production. Again.

This is the part where admission controllers should have saved you—and didn't, because they were either misconfigured, too permissive, or just... not there.

Admission controllers are one of the most powerful and most underutilized security mechanisms in the Kubernetes ecosystem. They sit between the API server and whatever actually gets written to etcd, giving you a last-chance opportunity to validate, reject, or even mutate incoming requests before they become reality in your cluster. When they work well, they're invisible. When they're missing or broken, you find out the hard way.

Let's talk about what most teams get wrong—and how to build something that actually holds.

What Admission Controllers Actually Do (And Why You Should Care)

At a high level, there are two kinds of webhooks you'll be working with: ValidatingWebhookConfiguration and MutatingWebhookConfiguration.

Validating webhooks look at an incoming request and give it a thumbs up or thumbs down. They can't change anything—they just decide whether the request is allowed to proceed. Mutating webhooks, on the other hand, can actually modify the request before it gets stored. Want to automatically inject a sidecar? Add a default resource limit? Slap a label on every pod? Mutating webhooks are how that happens.

What makes these powerful is that they operate after authentication and authorization but before the object gets persisted. That means even if a developer has the RBAC permissions to create a deployment, your webhook can still block that deployment if it violates policy. RBAC tells you who can do something. Admission controllers tell you whether it should be done at all.

That's a meaningful distinction that a lot of teams miss.

The Most Common Misconfiguration: Failure Mode Settings

Here's where things go sideways fast. Every webhook configuration has a failurePolicy field. It can be set to either Fail or Ignore.

Set it to Ignore and your webhook becomes purely advisory—if it's unreachable or returns an error, the request just goes through anyway. You've essentially built a security control with an off switch that activates whenever your webhook has a bad day.

Set it to Fail and you get the opposite problem: if your webhook service is down, nothing gets admitted. No new pods, no config updates, no deployments. Your entire cluster's ability to accept changes is now tied to the availability of your webhook service.

Most teams default to one extreme or the other without thinking through the implications. The right answer is context-dependent. For security-critical policies—like blocking privileged containers or enforcing image signing—Fail is correct, and you need to treat your webhook service with the same availability requirements as the API server itself. For lower-stakes mutations like injecting labels, Ignore might be acceptable.

The mistake is applying the same failurePolicy to everything.

Scope Creep: When Your Webhook Watches Too Much

Another pattern that causes real problems is over-broad webhook scope. If your namespaceSelector or objectSelector isn't properly configured, you can end up with a webhook that intercepts requests in namespaces it has no business touching—including system namespaces like kube-system.

This is how teams accidentally cause self-inflicted outages. A webhook that intercepts requests in kube-system can block critical system components from updating. If that webhook is also set to failurePolicy: Fail, you now have a scenario where a webhook service restart can take down cluster-critical functionality.

Always explicitly exclude system namespaces unless you have a very specific reason to include them. Use namespaceSelector with a label-based match to target only the namespaces your policies apply to. It's a small thing, but it's the difference between a security control and a footgun.

What RBAC Misses That Admission Controllers Catch

Let's get concrete. Here are three real scenarios where admission controllers fill gaps that RBAC can't:

Privileged container enforcement. A developer has permission to create pods in a namespace. Nothing in their RBAC policy says they can't set securityContext.privileged: true. Without a validating webhook checking for this, that container runs with host-level access. A well-scoped webhook catches this at admission time and rejects it before it ever schedules.

Image registry allowlisting. RBAC doesn't know or care what container image you're pulling. A validating webhook can check that every image reference points to your approved internal registry, blocking pulls from public Docker Hub or other unapproved sources. This is especially valuable in regulated industries where supply chain security is a compliance requirement.

Resource limit enforcement. Kubernetes won't stop you from creating a deployment with no resource requests or limits. Your cluster will, eventually, in the form of noisy neighbor problems and OOM kills. A mutating webhook can inject sensible defaults; a validating webhook can reject requests that don't specify limits at all. Either approach keeps your cluster from becoming a free-for-all.

Latency Is a Real Concern—Here's How to Handle It

The knock on admission webhooks is that they add latency to every API server request. That's true, and it's worth taking seriously. But it's also manageable if you approach it thoughtfully.

First, keep your webhook services lean. These aren't the place for complex business logic or database calls. Each webhook should do one thing, do it fast, and return a decision. Aim for sub-10ms response times under normal load.

Second, use timeoutSeconds to set an explicit cap on how long the API server will wait for your webhook. The default is 10 seconds, which is way too long for a path that every API request goes through. Set it to something like 3-5 seconds and make sure your service can reliably respond within that window.

Third, be surgical with your rules. Every rules entry in your webhook configuration is another pattern the API server has to match against incoming requests. Don't use * wildcards unless you genuinely need to intercept everything. Target specific resources, specific API groups, specific operations.

Finally, consider running your webhook service with a pod disruption budget and multiple replicas. If your webhook is a single pod, a node drain or a rolling update creates a window where your security controls are degraded. Treat it like infrastructure, not an application.

A Practical Starting Point

If you're starting from scratch, don't try to boil the ocean. Pick two or three high-value policies—privileged container blocking, image registry enforcement, and resource limit requirements are good candidates—and implement those first with tight scoping and proper failure modes.

Tools like OPA Gatekeeper and Kyverno give you policy-as-code frameworks that make this significantly easier to manage at scale. They abstract away a lot of the webhook plumbing and let you focus on writing policies in a human-readable format. If you're managing more than a handful of policies, using a framework like these is almost always worth it.

The goal isn't to create an impenetrable fortress of admission rules. It's to catch the predictable mistakes before they become incidents. Admission controllers, done right, are the difference between a policy that lives in a wiki and one that actually runs in your cluster.

Stop leaving that door unguarded.