When Kubernetes Meets Real Data Center Networking: Building a Routable, Resilient On-Prem Cluster with BGP and ECMP
TL;DR — Key Takeaways
- On-prem Kubernetes networking is fundamentally different from cloud networking because operators own the switches, uplinks, routing and failure domains.
- Excessive NAT and overlays make troubleshooting harder by hiding original pod identities and forcing operators to reconstruct flows across translation and encapsulation layers.
- A routable design gives each pod an address the data center can reach directly, with BGP advertising pod prefixes into the fabric.
- BIRD handles eBGP while kube-router manages pod networking, IPVS service routing and policy, separating routing from Kubernetes control-plane dependencies.
- Dual uplinks with ECMP and BFD improve resilience, allowing traffic to move quickly when a link or switch fails.
- The payoff is simpler incident response: operators can inspect standard routing state instead of untangling NAT tables, tunnels and hidden overlays.
I have spent a lot of time staring at packet captures that should have been simple. A connection fails between two services, but tcpdump shows translated addresses instead of the real source and destination. You end up cross-referencing NAT tables and overlay headers to reconstruct the flow.
That experience pushed me toward a routable network for our on-prem Kubernetes cluster. The goal was a simpler data path with fewer translations and better visibility.
On-Prem Kubernetes is a Different Problem
In a cloud environment, the provider handles cross-node routing and load balancers. On-prem, the operator owns the switches, cables and failure domains. Adding a node does not tell the rest of the network where its pods live, and traffic will not reroute around a failed switch port unless the design supports it.
Routing protocol, uplink design and placement of the network daemon determine how the cluster behaves during an incident. These decisions have to be made before the first failure.
The data center also needs an answer for services that cloud platforms normally supply. A node needs a route for its pod CIDR. A failed uplink needs a withdrawal mechanism. The API endpoint needs to move when a control plane node goes down. Treating these as routing problems makes recovery behavior explicit instead of depending on hidden state in several networking layers.
The Problem With Excessive NAT
NAT is useful when address translation is required, but making it the default inside an on-prem cluster obscures packet identity. A typical kube-proxy Service path rewrites the destination address. Some paths also use SNAT or MASQUERADE, hiding the source. Add an overlay, and the underlay sees node traffic instead of the pod-level flow.
Encapsulation also adds overhead where the physical network can already route pod traffic. VXLAN adds about 50 bytes per packet, depending on how Ethernet framing is counted, and increases CPU use. Direct L3 routing lets the Linux kernel and data center switches forward packets with their normal routing tables.
The operational cost is more important than the header size. During an incident, the switch may see only traffic between node addresses, while a capture inside the cluster shows translated addresses. Neither view identifies the original flow on its own. The operator has to correlate conntrack state with the overlay and Service rules before testing the network path.
What Routable Kubernetes Networking Means
In a routable cluster, pods are reachable through explicit routing decisions. Each pod has an IP address that is reachable across the data center without translation. A routing protocol tells the network which node owns each pod prefix, so traceroute and packet capture expose the real path.
Default CNI configurations often keep the cluster opaque to the fabric. The switches witness node traffic while Kubernetes manages pod reachability separately. When the same team operates the cluster and the network, that boundary makes cross-layer failures harder to diagnose. A routed design lets standard network tools work across both.
This does require the network team to accept pod prefixes into the data center routing domain. Therefore, export filters need to be narrow. A node should announce only the pod CIDR assigned to it and any approved virtual or loopback addresses. It should not become a general transit router or leak unrelated host routes.
The Architecture
kube-router is the CNI plugin. It manages pod networking, replaces kube-proxy with IPVS-based service routing and enforces network policy through iptables and ipset.
Bird Internet Routing Daemon (BIRD) runs as a systemd service on each node and manages eBGP sessions to the data center switches. Each node has two uplinks to independent top-of-rack switches, with ECMP across both paths. BIRD announces each node’s pod CIDR to the fabric.
Cross-node pod traffic follows visible IP hops: The source node routes the packet to a ToR switch, the fabric sends it to the destination node and that node delivers it to the pod.
BGP: Announcing Routes Into the Fabric
Every node runs BIRD and peers with both ToR switches. Each session exports a small set of routes, most importantly the pod CIDR assigned by Kubernetes IPAM. Selected service VIPs or loopback addresses can also be exported through strict filters.
BIRD is independent of the Kubernetes control plane and uses the host routing table. That separation is useful during partial failures: The fabric can retain reachability to healthy pods even when the API server is unavailable, and an operator can inspect BGP state without depending on kubectl.
A worker-node configuration looks like this:
router id 10.19.0.2;
# kube-bridge: pod gateway for pods on this node
# kube-dummy-if: Kubernetes Service Ips assigned by kube-router
# lo: routable node management address
filter accept_kubernetes {
if ifname = “kube-bridge” then accept;
if ifname = “kube-dummy if” then accept;
if ifname = “lo” then accept;
else reject
}
filter accept_default {
if net ~ [0.0.0.0/0] then accept;
else reject;
}
protocol direct kube_router_dummy {
ipv4;
check link on;
interface “kube-dummy-if”;
}
protocol direct lo {
ipv4;
check link on;
interface “lo”;
}
protocol device {
scan time 10;
}
protocol kernel {
learn all;
scan time 10;
merge paths 8;
persist on;
ipv4 {
import all;
export all;
};
}
protocol bfd {
interface “e*” {
min rx interval 100 ms;
min tx interval 100 ms;
idle tx interval 1000 ms;
multiplier 3;
};
}
protocol static default_backup {
ipv4;
route 0.0.0.0/0 via 10.19.2.1 { preference = 50; };
}
protocol bgp leaf_a {
local 10.19.2.2 as 65001;
neighbor 10.19.2.1 as 65000;
hold time 90;
bfd on; ipv4 {
next hop self on;
import filter accept_default;
export filter accept_kubernetes;
};
}
protocol bgp leaf_b {
local 10.19.3.2 as 65001;
neighbor 10.19.3.1 as 65000;
hold time 90;
bfd on;
ipv4 {
next hop self on;
import filter accept_default;
export filter accept_kubernetes;
};
}
The switches install the announced pod CIDRs like any other routed prefix. Pod-to-pod traffic crosses no overlay tunnel and uses no NAT in the data path. On the node, kube-router manages pod interfaces and IPVS rules, while BIRD handles BGP sessions and route advertisements. A failure in one component does not automatically stop the other.
The backup default route in the example has a lower preference than routes learned from the two BGP peers. It is there for controlled fallback, not to replace ECMP. The export filter is equally important: It admits routes tied to the Kubernetes interfaces and loopback while rejecting everything else.
ECMP and BFD Across Dual Uplinks
ECMP uses both uplinks on each node at the same time. The switches distribute flows across equal-cost paths using a hash. L3 hashing normally uses source IP, destination IP and protocol. L4-aware hashing also includes the ports. Both links carry production traffic instead of keeping one idle for failover.
As ECMP works per flow, it does not split individual packets arbitrarily across both links. This avoids packet reordering within a connection under normal conditions. The result depends on the hashing capabilities and configuration of the ToR switches, so the exact behavior should be tested with the production fabric.
When a switch fails, its routes are withdrawn and traffic moves to the surviving path. Default BGP hold timers can leave a dead peer undetected for 90s, causing traffic to be blackholed during that interval.
BFD detects forwarding failures in sub-second timeframes and tells BGP to tear down the affected session. The configuration above uses 100ms receive and transmit intervals with a multiplier of three. Therefore, a failed link can be removed before most applications notice it.
Kube-router and the Operational Model
kube-router handles pod interfaces, IPVS service routing and network policy without a separate data store or custom kernel module. It reads from the Kubernetes API.
kube-router can advertise pod CIDRs itself, but I assigned that job to BIRD. If kube-router fails, the BGP sessions remain up. If BIRD fails, local pod networking and service routing continue.
There is a tradeoff: kube-router enforces network policy with iptables chains and ipsets, not eBPF. Large rule sets with complex policies should be benchmarked. Environments that need eBPF-based policy or deep per-flow visibility may be better served by Cilium in BGP control-plane mode, though its operational model differs.
Control Plane and Worker Node Resilience
Worker nodes and control plane nodes have different failure modes. For a worker, the main risks are physical: A NIC, cable, switch port or ToR switch. Dual uplinks to separate switches keep one path available, while BFD-triggered BGP withdrawal removes the failed path.
For the control plane, the main risk is the loss of quorum. etcd needs a majority of members for writes, so control plane nodes should be placed in separate physical failure domains. One hardware failure should not affect more than one member.
I used the same L3 design for control plane networking. BIRD runs on each host, with two uplinks to separate switches, and announces only addresses present on that node. The Kubernetes API is exposed through a virtual IP. Keepalived runs in unicast VRRP mode, moves the VIP between control plane nodes and checks the local API server.
Separating the control plane from workers also limits the blast radius. Worker routes do not depend on the API VIP, and the control plane does not share a switch failure domain with a large group of workload nodes. Physical placement is part of the quorum design, not an afterthought.
Only the current keepalived master advertises the VIP through BIRD. If the API server or its node fails, keepalived moves the VIP and BIRD announces it from the new owner. If a link fails, BFD and BGP remove that path while the other uplink continues to carry the traffic.
BIRD runs on the host rather than as a DaemonSet. A networking daemon inside Kubernetes creates a dependency loop: Pods need networking to start, but the networking pod needs the scheduler and cluster network. Running BIRD under systemd breaks that cycle. A control plane outage also leaves worker BGP sessions and pod-to-pod traffic unaffected.
What You Can See During an Incident
The routed design gives operators a short diagnostic chain:
ip route show
birdc show route
birdc show protocols
birdc show bfd sessions
tcpdump -i any host 10.19.64.23
On the switch, show ip bgp summary shows which nodes are peered and what they advertise. A failure is usually visible as a down BGP session, a missing route or blackholed traffic. Each step can be checked at the layer where it occurs.
The checks follow the packet path. First confirm the host route, then verify what BIRD is exporting and whether both peers are established. BFD shows whether the forwarding path is alive. A packet capture then confirms that traffic carries the expected pod addresses without an encapsulation layer.
With NAT and overlays, the same symptom may come from address translation, iptables state or tunnel encapsulation. A routed architecture reduces that search space and preserves the packet identities needed to debug the incident.
The Result
This design runs BIRD on the host, peers each node with two ToR switches, uses ECMP across both uplinks and relies on BFD for fast failure detection. kube-router remains responsible for pod networking and IPVS service routing. Pod traffic crosses the fabric without an overlay or NAT.
On-prem Kubernetes requires the cluster to participate in the data center routing fabric. With BGP and ECMP, failures appear in routing state that operators can inspect with standard tools instead of reconstructing the path through translation and encapsulation layers.
Frequently Asked Questions
Why use BGP for on-prem Kubernetes networking?
BGP lets Kubernetes nodes advertise their pod CIDRs directly into the data center fabric, so the network knows which node owns each pod prefix and can route traffic without relying on overlays or internal NAT.
Why avoid unnecessary NAT and VXLAN overlays?
They can hide the true source and destination of traffic and add encapsulation overhead, making packet captures and network troubleshooting more complicated.
How does this design improve Kubernetes incident response?
Operators can follow a clear diagnostic chain using host routes, BIRD state, BGP sessions, BFD status and packet captures, with the original pod addresses preserved.
