Why Your Kubernetes Readiness Probes Are Lying During Rolling Updates

TL;DR — Key Takeaways

• A Kubernetes rolling update can look completely healthy while real traffic is failing if readiness probes check only process health rather than true application readiness.
• In a 5G signaling service, HTTP probes passed before SIP registration completed, causing an 18% spike in session-establishment failures during the update window.
• Protocol-aware readiness checks should verify the conditions the application actually needs to serve production traffic, such as peer registration, handshakes, warmed connection pools or service-mesh integration.

Rolling updates are one of Kubernetes’ most useful features. Configure them correctly, and your application deploys without downtime. Traffic shifts gradually. Old pods drain. New pods take over. The dashboard stays green throughout.

That is the theory. Here is what can happen in practice.

During a rolling update across a Kubernetes-based 5G signaling service, every readiness probe passed. New pods came up healthy. The old pods drained cleanly. From Kubernetes’ perspective, the update was textbook.

Session establishment failures jumped 18% during the update window.

The new pods were running. They were not ready. Nothing in the standard probe configuration knew the difference.

What Readiness Probes Actually Check

Kubernetes readiness probes exist to solve a real problem: A pod that starts successfully might not be able to serve traffic immediately. The application might still be loading configuration, warming caches or establishing database connections. Readiness probes give Kubernetes a way to hold traffic back until the pod signals it is prepared.

Most readiness probes are HTTP checks. Kubernetes sends a GET request to a specified endpoint. If it gets a 200 response, the pod is marked ready. Traffic flows.

That works well for stateless web services, where ‘the application responds to HTTP’ and ‘the application can handle requests’ are effectively the same thing.

The assumption starts to crack the moment your workload communicates through something other than HTTP.

Where the Assumption Breaks

Telecom network functions — anything speaking SIP, Diameter or GTP — don’t just start up and serve requests. They register with upstream systems. They complete protocol handshakes. They establish peer relationships with gateways, authentication services and session management functions that may themselves take time to acknowledge the new pod.

That registration process takes time. Depending on the environment, it can take anywhere from several seconds to over a minute after process startup.

During that window, the pod is genuinely running. The HTTP health endpoint responds with 200. The readiness probe is satisfied. Kubernetes marks the pod ready and starts routing traffic.

But the pod hasn’t finished registering with the upstream signaling infrastructure. Requests that depend on that registration start failing — not with obvious errors, but with subtle degradation. Session establishment attempts that should succeed don’t. Retries increase. Latency climbs.

The infrastructure dashboards show nothing unusual. The pods are healthy. The update completed successfully.

The failures are happening in the gap between what Kubernetes can see and what the protocol actually requires.

This isn’t unique to telecom. Any workload that maintains stateful connections, registers with upstream systems or depends on protocol-level handshakes before it can serve real traffic faces the same exposure. Message queue consumers, gRPC services with warm-up dependencies and database-backed services with connection pool initialization all have variants of the same problem.

What Happened in Production

In the 5G deployment, the readiness probe was configured against a lightweight HTTP endpoint that confirmed the process had started. It returned 200 within two seconds of pod startup.

What it couldn’t confirm was whether the pod had completed SIP registration with the upstream proxy and re-established its session state. That took considerably longer — long enough that traffic was being routed to pods that were live but not yet part of the signaling flow.

The symptom was an 18% increase in failed session establishment attempts during the update window. It resolved on its own as registration completed, which made it easy to miss the first time. The second time it happened, during a peak traffic window, the impact was more visible.

The fix wasn’t complicated. The diagnosis took longer than the solution.

What Protocol-Aware Readiness Actually Looks Like

The standard HTTP probe checks whether your application is alive. A protocol-aware probe checks whether your application is integrated.

For the signaling service, that meant replacing the HTTP probe with an exec probe that queried internal registration state before declaring readiness:

readinessProbe:

  exec:

    command:

    – /bin/sh

    – -c

    – “check-registration-status.sh && exit 0 || exit 1”

  initialDelaySeconds: 10

  periodSeconds: 5

  failureThreshold: 12

The script checked whether the pod had successfully registered with upstream peers. Until it had, the pod stayed out of the traffic rotation — regardless of what the HTTP endpoint reported.

The initialDelaySeconds gave the registration process time to begin. The failureThreshold gave it time to complete without triggering unnecessary restarts.

For workloads outside telecom, the same principle applies. If your application needs to complete a handshake, warm a connection pool or register with a service mesh before it can handle production traffic, your readiness probe should verify that — not just confirm that the process started.

Running Isn’t the Same as Ready

Kubernetes gives you the tools to express what ‘ready’ actually means for your workload. The default HTTP probe is a starting point, not a definition.

In stateless, HTTP-native services, the gap between ‘running’ and ‘ready’ is small enough that it rarely matters. In stateful, protocol-aware or registration-dependent workloads, that gap is exactly where rolling updates fail quietly — passing every infrastructure check while degrading real traffic.

The probe should reflect what your application actually needs to serve requests — not what is easy to check; not what works for the example in the documentation.

If you are routing traffic based on an HTTP 200 from an endpoint that says nothing about your application’s actual readiness state, you are not doing zero-downtime deployments. You are doing zero-visible-downtime deployments — and the difference shows up in your error rates.

Frequently Asked Questions

Why can a Kubernetes readiness probe pass when the application is not truly ready?

A standard HTTP readiness probe often confirms only that the process is running and responding. It may not verify that the application has completed upstream registration, protocol handshakes or other dependencies required to handle real production traffic.

What is a protocol-aware readiness probe?

It is a readiness check that verifies the application’s actual operational state. For a telecom workload, that might mean confirming SIP registration; for other systems, it could mean checking service-mesh registration, connection-pool initialization or another protocol-level dependency.

When should teams go beyond a standard HTTP readiness check?

Any workload that depends on stateful connections, upstream registration, warm-up procedures or protocol handshakes should consider a readiness check that reflects those conditions rather than relying solely on a generic health endpoint.