Kubernetes Security: Attachment Issues

As a product manager, my main focus is on the value my customers will get from products. I often meet with customers and talk about pain points and problems; offer a solution, see how the product can help address those. Over the last few years, one thing that always comes up isn’t related to any pain point or feature requirement – it is the attachment method when dealing with Kubernetes security.

Customers ask, “Are you using eBPF?” “Is your product agentless?” – and while these are classic, technical, “how” questions, it doesn’t surprise me that customers are asking. We use these terms and terminology all over our website: “Agentless,” “eBPF,” “nano-agent.” But what do those terms mean, and is one attachment method better than another?

TL;DR: there is no right and wrong! Each attachment option gives you different capabilities and benefits – there is no magic! For the purposes of simplicity, I’ll only focus on attachment in Kubernetes. That said, basically, there are the following attachment points (for a Kubernetes security/visibility product):

  • API (often called agentless)
  • Host agent (today’s trend is extended Berkeley Packet Filter, or eBPF),
  • Application-level agent (can be a sidecar or an agent inside the application/process).

That’s it! No other options exist. Putting a physical device outside of the cluster (a firewall, WAF, IPS) does not really work in Kubernetes, as you then lose the context, which is the very essence of security and visibility.

Attachments and Kubernetes Security

Let’s look at each attachment option and the pros and cons of each.

API-Based or Agentless Attachment

The API option is all about using Kubernetes’ native security capabilities (in managed Kubernetes cases like AKS and EKS, it also includes the cloud vendors’ APIs). You use these capabilities to get audit logs, pods logs, Kubernetes configuration, etc. Later, you apply some logic that will generate security value.

Is it seamless? Not entirely. In most cases, the product must have the highest-privileged access to your environment, which means it can do anything and may even become an additional point of vulnerability. Some vendors will argue that their product “just” needs read-only access. If this is, in fact, the case, then that product is a detection, recommendation-based product. It can alert if something is wrong, but that’s it – you need to go the extra mile, sometimes manually, to further analyze or remediate the issue. In addition, if a malicious actor knows your configuration, they will be able to find a way into your environment; this is the perfect kind of intelligence they need to plan an attack. Lastly, you are bound to only what functionality the API provides. For example, if you use Kubernetes’ API to generate a pod-pod access list, you can’t do basic things like logging specific connections or blocking a connection.

That said, agentless is a very good option for trying out products, and will probably get the least objection from DevOps teams or from whichever person is managing the environment. If, after trying it out it does not provide the value you need, remove it – no harm done.

Host Agent (Is it eBPF?)

Host agent is not a new idea; antivirus companies have used it for years (think user mode and/or kernel mode). It is, as the name suggests, an agent, usually privileged, that you install on each worker node in your cluster. In Kubernetes, it is called daemonset, and Kubernetes did an amazing job scaling this agent.

Over the last few months, the Kubernetes community has amplied the use of a new “magic” called eBPF. I, personally, became acquainted with this magic two or three years ago, when I was trying to help customers solve visibility challenges in Kubernetes – specifically, knowing which pod communicated with which pod.

I cannot call myself an eBPF expert; I am far from it. Years ago, at Check Point, we developed a Linux kernel driver, which was intended for fast packet processing. It was an agent on every host. The issue with this agent was that the smallest problem would cause the entire machine to die. Back then, the only other option was to go to user mode. In user mode, if you have a bug, the agent dies, but the host remains functional. You can then spin up a new agent, and start from where you left off – worst-case scenario, you lost a few connections. The larger benefit was performance – the kernel performed way better. One important note, though, is that kernel attachment is not really an option in the cloud; in some clouds, you cannot change the Linux kernel.

Enter eBPF, which solves this problem. In eBPF, you write a small program in userspace that runs in a sandbox in the kernel. You don’t need to install kernel modules; you just write code that runs in a sandbox and changes your kernel.

This sounds great. Can I order two? Joking aside, it is great. The only issue is that, now, you have an agent with root privileges on every host in your environment. If someone breaks into the product, they are able to go anywhere in your environment. If there is a bug in your program, it might take down the entire host. Another issue is that you must, as the name suggests, run the host agent on the host. That’s not an option in cases where you use container-as-a service (CaaS) offerings like ACI or Fargate.

Application-Based Agent or Workload-Based Agent

The idea behind this attachment option is that something is running in the pod. There are two main options to cover: sidecar, which is getting more popular, mainly due to service meshes like Linkerd and Istio, and micro or nano agent, which is an agent that runs in the container itself (or in processes inside the container).

There are some general pros and cons with application- or workload-based agents. In most cases, you don’t need to change your application, and, at least in theory, the agent connects seamlessly to your environment. The main benefit of this approach is that you get a lot of value without the need to rewrite your application. The value you get from it can vary from canary deployment to network security and to various other use cases. It leaves your development team focused on what they do best – developing a product for your customers, and delivers added value from a third party who is an expert in security. Another important point is that this method should not require any excessive privileges; it should inherit the same privilege levels as your existing pod.

In today’s security and observability landscape, when the industry is pushing for end-to-end traffic encryption, most of the existing solutions (choke points, proxy etc.) will become blind to the traffic. Opening the traffic for inspection or visibility with man-in-the-middle (MiTM) techniques comes with a huge performance penalty. That leaves only one option, which is for observability to be in the application.

The issue with the container-based attachment point is that, if you remove it, it might require extra adjustment; that means it is not part of your application, but your application requires it to function. This is no different than, say, having a network firewall in your data center and taking it out. Once you take it out, you need to adjust routing.

Another concern that I often hear from customers is the added CPU, latency and memory overhead. They are afraid that adding this agent will degrade workload performance massively.

Sidecar

As the name implies, it is usually a container that runs in your pod. This container has some special features and capabilities of its own; some might be network security-oriented, but this is not mandatory.

A sidecar is a full container that is added to your application; since best practices dictate that you should run a single container in a pod, by adding the sidecar container, you are potentially increasing the resources required for your workload. In cloud environments, where you pay for the compute resources – it means you’re paying more money.

Another problem is the performance and latency. While some claim the impact on performance is small, the sidecar is acting as a transparent proxy, and even if it is super effective, it is still a proxy; a choke point for every communication destined to and from the workload. If you add a sidecar to any workload, you pay for this latency twice for pod-to-pod communications. The sidecar can handle network security problems, but it does not solve runtime malware detection and prevention.

Lastly, most of the people I have talked to will add it to the entire application (namespace, cluster, etc.). They did not find value by adding it to just a few microservices rather than the entire application.

The benefit of a sidecar approach is that, if it has a bug that causes it to crash, it does not influence your workload. To be honest, it will not kill your workload, but it also will not be able to operate.

Nano/Micro Agent

This is an agent, usually a library (.SO in Linux and DLL in windows). Adding it to your workload should be as easy as adding any other library to your workload. These micro-agents usually have a relatively small footprint (CPU and memory), and since they instrument operating system-native capabilities, if done right, they should not add major CPU, memory usage and latency.

The main issue is that, when the library has a bug, it might break your workload. As we’ve seen before, this is not a special case – if the agent has issues, in all cases, it will break your application. This is a major concern for most customers; they don’t want to add a third-party piece of software that might break their application. I would argue that, in the microservices and DevOps-oriented landscape we live in, our development teams take so many libraries and code from unknown third parties anyway that an agent should be the least of our concerns. At the end of the day, whichever vendor you choose is betting the success of their business (to some degree) on the quality of that agent, and they will support customers if issues arise.

There is no right and wrong when it comes to choosing the right attachment point, either. Leave it to the vendor to choose where to attach to solve your issues and give you added value. At the end of the day, you are not looking for an attachment point; you are looking for a solution to a headache.

Amir Kaushansky

Amir Kaushansky is an innovative and creative technology leader with 20+ years in all the areas of product life cycle: Product Management, Business Development, Technical Services , Quality Assurance, IT and Software Engineering. He Excels in bridging technical and business worlds, excellent communication skills and interpersonal relationships with peers, customers and suppliers and has a proven track record in development and deployment of complex products and solutions, especially in the field of cybersecurity.

Amir Kaushansky has 3 posts and counting. See all posts by Amir Kaushansky