Not Your Grandpa’s Packet Filter: eBPF in Cloud-Native Networking
eBPF has become an important tool in the systems programming and infrastructure software ecosystems. It is a revolutionary technology that allows safe and efficient modification of the Linux (and now Windows) kernel’s behavior without altering its source code or loading extra modules.
While its origins lie in packet filtering and networking, eBPF has evolved into a versatile technology that goes far beyond its initial purpose and has emerged as a transformative technology for the cloud-native ecosystem, including applications built using containers and running on platforms like Kubernetes. Its runtime programmability has opened the doors to a wide range of use cases, empowering developers and system engineers and administrators to rethink observability, security and performance optimization, among other areas.
This article explores how eBPF has evolved from a simple packet filter into a general-purpose compute engine within the kernel, and how it adds functionality, flexibility and performance to a decades-old technology. The best part is that eBPF is already running in billions of machines around the world, even though we are just scratching the surface of its potential. We’ll start with a short history of where eBPF came from before diving into a few different use cases eBPF is tackling today and looking at the projects in the cloud-native landscape that leverage eBPF.
eBPF: A Swiss Army Knife
eBPF traces its origins back to BPF (Berkeley Packet Filter), primarily focused on packet filtering and network operations in the Linux ecosystem.
BPF first began its evolution beyond just packet filtering in 2014, when engineers Alexei Starovoitov and Daniel Borkmann presented a proposal to the Linux kernel networking sub-system maintainer, David S. Miller. The proposal aimed to improve the Linux kernel by incorporating a new bytecode language and virtual machine, extending the existing BPF. The vision for eBPF was quite straightforward—a 64-bit solution for BPF that is easily extended with built-in methods that allow seamless interaction with the kernel.
This proposal was eventually accepted, and e(xtended)BPF was finally in the Linux kernel.
After its inclusion, eBPF continued to evolve; today, it is a true Swiss Army Knife for cloud-native developers, sysadmins and security experts.
“eBPF has revolutionized the way infrastructure platforms are built and has enabled many new cloud-native use cases that were previously difficult or impossible to implement,” said Bill Mulligan and Daniel Borkman. “With eBPF, platform engineers can safely and efficiently extend the capabilities of the Linux kernel, allowing them to innovate quickly. This allows for creating new abstractions and building blocks tailored to the demands of the cloud native world, making it easier for developers to deploy software at scale.”
The power of eBPF lies in its runtime programmability, allowing developers to dynamically execute code within the kernel, enabling deep visibility and control over various aspects of system behavior. This capability has led to a rapid expansion of eBPF’s use cases, transcending its initial networking-centric roots.
In the Wild: How the World is Using eBPF
eBPF is a technology with programmability at its core that enables developers to customize and extend the kernel’s behavior, opening up a world of possibilities. From continuous profiling and tracing to security enforcement and compliance auditing, eBPF has found its way into numerous areas of modern cloud-native computing. Its ability to capture, analyze and act on events across the system provides unparalleled observability and flexibility, enabling developers to gain insights into performance bottlenecks, troubleshoot issues and optimize resource utilization. Let’s explore how eBPF’s programmability has given rise to many powerful use cases.
The first prominent use case for eBPF is networking, which led to significant advancements like improved packet filtering, load balancing and secure routing. For example, Cilium’s eBPF-based load balancer enabled Seznam.cz to switch from IPVS, reducing CPU consumption by 72x while still seamlessly handling their production traffic.
The success of Cilium served as a catalyst, inspiring engineers to explore eBPF’s potential in other areas, with observability emerging as the next domain where eBPF shines. In the world of cloud-native applications, observability is crucial for monitoring and troubleshooting complex distributed systems. eBPF’s observability capabilities allow developers and system engineers to gain deep visibility into application behavior, network communication, and resource utilization.
By attaching to various kernel and user-space events, eBPF facilitates detailed monitoring and debugging, enabling faster issue resolution and better resource optimization. Attaching to various kernel and user-space events facilitates detailed tracing, too. bpftrace is a dynamic tracing tool that leverages eBPF. With bpftrace, developers can create custom tracepoints in their code to capture various events, such as function calls, system calls and I/O operations.
eBPF provides a low-overhead and flexible solution, allowing developers to gain deep insights into the performance characteristics of their applications. Traditionally, profiling involved significant overhead and often required specialized tools or manual configuration. However, eBPF’s low overhead and programmability shifted what was possible. With eBPF, developers can collect profiling data in real-time, with minimal impact on the application’s performance, opening up new possibilities for understanding application behavior under various workloads and scenarios.
The programmability of eBPF extends observability into the area of security, too. Within the Linux Security Modules (LSM) framework, eBPF empowers administrators to implement custom security policies, advanced access control and fine-grained permissions. Developers can also enforce data protection mechanisms, perform encryption and decryption and implement custom security policies with eBPF, as it offers several advantages that make it stand out in data security compared to other pre-existing tools. Its runtime programmability enables the implementation of custom security policies with fine-grained control, ensuring tailored protection for specific needs.
Efficient process scheduling is critical for system performance and responsiveness. With eBPF, system engineers can enhance kernel scheduling policies by adding custom logic and fine-tuning process management. By leveraging eBPF’s capability to efficiently profile scheduling latencies, Google got insights into queueing delays and CPU usage. Their ghOSt kernel scheduler was able to drastically reduce the time between a CPU going idle and task assignment down to a mere 0-1 microseconds.
Related to security, intrusion detection systems (IDS) play a vital role in identifying and mitigating security threats. Compared to other pre-existing IDS tools, leveraging eBPF and its runtime programmability in IDS enables efficient and real-time analysis of network traffic, enabling faster detection of malicious patterns and potential security threats. Its ability to attach to various kernel and user-space events facilitates comprehensive monitoring and troubleshooting. Moreover, eBPF’s seamless integration with IDS frameworks allows for easy adoption and scalability.
Finally, eBPF isn’t just limited to the kernel context, it also extends its capabilities to managing human interface device (HID) events, enabling advanced input handling and customization. This use case finds applications in gaming, accessibility and enhancing user experiences. An example of where this use case can flourish is in devices like the Microsoft Surface Dial. The Linux kernel recognizes the Surface Dial but doesn’t handle its haptic feedback. With eBPF, the device can be transformed into a mouse, converting dial events into wheel events.
Through eBPF’s remarkable programmability, many diverse and impactful use cases have been realized, surpassing what was previously possible. The ability to customize the kernel opens up endless possibilities, with the potential for even more applications in the future.
eBPF: Empowering the Cloud-Native Ecosystem
Developers have long faced problems such as limited observability, complex networking and performance limitations, and these challenges only increased with the move toward microservices and distributed computing. eBPF enables the development of powerful new tools that offer enhanced observability, efficient networking and improved performance management, revolutionizing the way cloud-native applications are built, run and operated.
In this section, we’ll highlight some open source applications in the cloud native ecosystem that use eBPF to tackle different use cases.
Cilium
Cilium is a networking, observability, and security project that leverages eBPF to provide enhanced networking capabilities and advanced security features, particularly in the context of Kubernetes. Using eBPF, Cilium achieves high-performance packet filtering, routing and load balancing at the kernel level. This level of network control, performance and security would be challenging to achieve without eBPF’s programmability. Cilium’s use of eBPF has significantly boosted its performance above previous solutions like iptables. With eBPF, Cilium is moving towards 100 Gb/s Kubernetes clusters.
Falco
Falco is a cloud-native runtime security project designed to detect and alert on unexpected application behavior and potential security threats in real-time. By leveraging eBPF, Falco can dynamically instrument the Linux kernel to capture and analyze system events. This level of runtime security monitoring is made possible by eBPF’s observability and runtime programmability.
Hubble
Hubble is a network visibility project under Cilium that utilizes eBPF to provide advanced network monitoring and analysis capabilities in Kubernetes environments. Hubble uses eBPF to capture and analyze network traffic at the kernel level. By attaching eBPF programs to network interfaces, Hubble can intercept and inspect network packets without adding significant overhead to the system. This allows Hubble to gather valuable insights into network flows between Kubernetes pods and services.
With eBPF’s observability power, Hubble enabled Ascend to reduce their debugging time from four to 16 hours down to 20 seconds and also enabled them to detect issues in less than three seconds.
Katran
Katran is a C++ high-level Layer 4 load balancer. With eBPF, Katran can perform efficient packet processing directly in the kernel, allowing for high-performance load balancing and efficient network traffic management. It has processed every packet coming into Meta data center since 2017.
KubeAmor
KubeArmor is a cloud-native runtime enforcement system. It provides control over the actions of pods, containers and nodes (VMs) at the system level, like process execution, file access and network operations. eBPF’s programmability allows KubeAmor to provide developers with detailed alerts and telemetry events that include precise container, pod and namespace identities.
Parca
Parca is a continuous profiling tool for the analysis of CPU and memory usage, down to the line number and throughout time. Using eBPF allows Parca to collect profiling data, save infrastructure costs, improve system performance and increase reliability. Parca was used to audit Cilium, resulting in a reduction of the baseline CPU usage of Cilium Agents by 3%-4% through the elimination of unnecessary allocations. Additionally, Parca played a crucial role in enhancing Hubble decoding performance for trace events in runtime, achieving an impressive improvement of 92%-95%.
Pixie
Pixie is an observability tool for Kubernetes applications. Pixie uses eBPF to automatically capture telemetry data without the need for manual instrumentation. eBPF’s observability capabilities enable Pixie to provide real-time insights into the behavior and performance of Kubernetes applications, simplifying observability workflows.
Tetragon
Tetragon, a project under Cilium, provides eBPF-based transparent security observability combined with runtime enforcement. The deep visibility is achieved without requiring application changes and is provided at low overhead thanks to smart in-kernel filtering and aggregation logic built directly into the eBPF-based kernel-level collector.
eBPF: Beyond Networking
eBPF is no longer just your grandpa’s packet filter. Born from a visionary proposal by engineers Alexei Starovoitov and Daniel Borkmann, eBPF has opened up a world of possibilities for developers, sysadmins and security experts.
eBPF’s runtime programmability has allowed the kernel’s capabilities to evolve, making it a Swiss Army knife for developers, operators and administrators. This evolution has opened up an array of use cases, empowering developers, sysadmins and security experts to achieve high levels of observability, security and performance optimization in Linux and across the cloud native ecosystem. Many cloud-native applications now rely on eBPF to provide core functionality to meet the demands of dynamic distributed computing.
As the eBPF ecosystem expands and developers explore its capabilities, the future holds even more exciting possibilities for the technology.
