The Hidden Cost of “Just Works” Load Balancing in a Service Mesh
If you’re running a multi-AZ Kubernetes cluster with a service mesh on top, there’s a good chance you’re paying a tax you never signed up for, and it won’t show up as a line item on any dashboard you’re already watching.
The Default Nobody Configures
Kubernetes Services and Istio’s Envoy sidecars both default to distributing traffic randomly across all healthy endpoints, with zero awareness of which availability zone a pod happens to live in. That’s a perfectly reasonable default for a single-AZ deployment. But once you spread pods across three AZs for resilience (which is more or less table stakes at this point), that same default quietly becomes a liability. A pod in us-east-1a calling a downstream service now has roughly a two-in-three chance of landing on a pod in a different AZ. Multiply that across a request path touching three or four internal services plus a database replica, and a single user request can rack up multiple cross-AZ hops before a response ever goes out.
AWS makes this worse by default, too. Cross-zone load balancing on an NLB sitting in front of an Istio ingress gateway will happily route an incoming connection to a target in any AZ, even when a perfectly healthy target is sitting right next to the load balancer node that received the request in the first place.
What It Actually Costs
Two things, concretely.
Latency. In a mid-sized production cluster running roughly 3,500 RPS across three AZs, tracing data showed same-AZ requests landing at 15-18ms p50, while cross-AZ requests to the same service ran 25-30ms p50, 40% to 65% slower. With about two-thirds of requests crossing AZs, the weighted p50 sat around 24ms instead of the ~17ms it could have been. That gap compounds fast on any request path with multiple internal hops.
Money. AWS bills $0.01/GB for traffic crossing AZs within a region. That sounds trivial on a per-request basis, but service-to-service traffic (the kind that doesn’t show up in your external bandwidth numbers) tends to run 5x to 10x the volume of external traffic once you count internal API calls, database replica reads and Kafka consumer traffic. In the environment analyzed here, a conservative accounting of ingress, service-to-service, database and Kafka cross-AZ traffic landed around $600/month before any fix, and that’s before factoring in automatic Aurora cross-AZ replication and log shipping, which push the real number higher in practice. The exact dollar figure depends heavily on your own traffic mix, so pull your own numbers from AWS Cost Explorer’s DataTransfer-Regional-Bytes usage type before treating any of this as gospel, including mine.
The Fix: Locality-Aware, Not Locality-Only
Istio supports locality-aware load balancing through the DestinationRule traffic policy. The naive fix is to route 100% of traffic to the local AZ and 0% everywhere else. Don’t do that. It removes your ability to absorb a bad deploy or a hot pod in the local AZ, and it makes a full AZ failure much harder to recover from cleanly.
The better pattern is a weighted distribution: 80% same-AZ, split 10/10 across the other two, combined with outlier detection so unhealthy endpoints get ejected automatically.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: checkout-service-locality-lb
spec:
host: checkout-service.prod.svc.cluster.local
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
distribute:
- from: us-east-1a/*
to:
"us-east-1a/*": 80
"us-east-1b/*": 10
"us-east-1c/*": 10
outlierDetection:
consecutiveErrors: 5
interval: 30s
baseEjectionTime: 30s
maxEjectionPercent: 50Three things have to be true for this to actually work, and all three are easy to skip:
- Even pod distribution across AZs. If one AZ holds 50% of a service’s pods, an 80/10/10 policy sends that AZ far more than half of all traffic. Use
topologySpreadConstraintswithmaxSkew: 1to keep pod counts balanced. Otherwise the locality policy just relocates the imbalance instead of fixing it. - Ingress gateway pods spread across all AZs, too. Locality-aware routing at the service level doesn’t help if every request enters through ingress pods concentrated in one AZ to begin with.
- NLB cross-zone load balancing disabled, via the
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false"annotation. Skip this and the load balancer itself reintroduces the exact randomness you just eliminated at the mesh layer.
What You Get, and What You Give Up
Latency improves, but the more interesting change is in failure blast radius. Under the old random-distribution model, losing an AZ means the remaining two suddenly absorb 50% more load than they were running a moment earlier, a fast path to cascading saturation. Under an 80/10/10 locality policy, losing an AZ only redistributes the 80% slice that AZ was already handling for itself; the other AZs see a much smaller relative increase, and outlier detection reroutes around the failure automatically.
The tradeoff is operational complexity. Traffic distribution stops being an intuitive 33/33/33, and anyone debugging “why is this AZ getting more traffic than that one” now needs to know the locality policy exists. Document it before you ship it, not after someone gets paged asking why a dashboard looks lopsided.
Before You Roll This Out
Validate in a lower environment first, with realistic load. A k6 or Locust run against a staging cluster is usually enough to surface load imbalance caused by uneven pod counts. Canary on your lowest-traffic services in production before touching anything customer-critical, and watch error rate and p95 latency, not just the cross-AZ percentage, before calling it done.
Locality-aware load balancing isn’t exotic. It’s an Istio feature that’s existed for years, and it gets skipped mostly because the mesh works fine without it, right up until someone reconciles the AWS data transfer line item or gets paged during an AZ event. Both are worth checking before you assume your multi-AZ cluster is actually behaving like one.
