The Kubernetes Annotation Pitfall: The One Word That Puts Your AWS Load Balancers at Risk
Misconfiguring just one word in Kubernetes can expose your AWS environment to the internet, putting your data and applications at serious risk. Kubernetes and AWS are essential tools for managing scalable applications, yet their complexity can sometimes lead to critical misconfigurations. As DevOps and security professionals, we understand the importance of staying ahead of these challenges and ensuring our systems are both robust and secure, but with the introduction of large language models (LLM) the challenges have grown.
There is no question that LLMs are redefining how we perform tasks across the organization. From marketing to software engineering, there is not a single aspect that hasn’t been impacted by the proliferation of tools like ChatGPT and Gemini.
However, along with the obvious benefits come certain risks that must be managed. Thanks to the powers of LLMs, what was previously not an issue may now become one.
An example of a misconfiguration is managing security groups for AWS load balancers using Kubernetes annotations, specifically classic load balancers (CLB) and network load balancers (NLB).
What we have found in our research is that ChatGPT may sometimes suggest one annotation:
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups
and at other times another annotation:
service.beta.kubernetes.io/aws-load-balancer-security-groups
The difference between the annotations that the LLM suggests depends on how you prompt the model, and the wrong choice can dramatically amplify your exposure. In this post, we will dive into this issue with Kubernetes annotations, understand what can go wrong and figure out how to avoid this pitfall.
Potential Scenario: Load Balancer Security Misconfiguration
Consider a scenario where a developer wants to set up a simple test environment using an AWS CLB in Kubernetes with the help of ChatGPT. To quickly get started, the developer creates a security group that allows traffic from their home IP address — 203.0.113.1/32 in our example — with the following cloud formation configurations:
AWSTemplateFormatVersion: “2010-09-09”
Resources:
MySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: “Allow HTTP from specific IP”
VpcId: “vpc-12345678”
SecurityGroupIngress:
– IpProtocol: “tcp”
FromPort: 80
ToPort: 80
CidrIp: “203.0.113.1/32”
SecurityGroupEgress:
– IpProtocol: “-1”
CidrIp: “0.0.0.0/0”
After creating the security group, the developer receives the following security group ID: “sg-0123456789abcdef0”
Then, the developer asks ChatGPT to create a load balancer that sets its security group. ChatGPT provides a YAML with the following configuration:
apiVersion: v1
kind: Service
metadata:
name: my-loadbalancer-service
annotations: service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: “sg-0123456789abcdef0”
spec:
selector:
app: my-app
ports:
– protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
Unaware of the implications, contrary to the intention, the developer exposes the load balancer to the entire internet, thus creating a significant security risk.
So, What Happens Next?
When AWS EKS creates a load balancer (for example, through a Kubernetes service of type: LoadBalancer), it also creates a default security group that allows access from IP addresses anywhere in the world (0.0.0.0/0) on the port specified in the service configuration. In the example above, this port is 80.
While the developer created a restricted security group to allow traffic only from a specific IP address, the issue arises with the word ‘extra’ in the annotation. Instead of replacing the default security group, this annotation adds the developer’s specified security group on top of the default one. That means that the load balancer allows traffic from everywhere (0.0.0.0/0), even though the developer meant to configure only the security group, which allows traffic from the specific IP. In short, this one word can leave your load balancer open to the world.
It’s not a bug; it’s a feature.
Understanding the Differences Between Key Annotations
aws-load-balancer-security-groups vs aws-load-balancer-extra-security-groups
When configuring AWS classic/network load balancers in Kubernetes, there are two kinds of annotations developers can use for managing security groups:
- service.beta.kubernetes.io/aws-load-balancer-security-groups
- Purpose: Specifies the primary security groups for the load balancer and replaces the default security group.
- When to use: Use this annotation when you need to define a new set of security groups to replace the default security group created by AWS for your load balancer.
- service.beta.kubernetes.io/aws-load-balancer-extra-security-groups
- Purpose: Allows you to add additional security groups to the load balancer in addition to the default security group.
- When to use: Use this annotation when you want to retain the default security group and simply add more security groups for additional control and security policies.
Three Types of Annotations in Kubernetes
There are three types of annotations for AWS load balancers in Kubernetes, which relate to our use case:
| Annotation | Applies To | Default SG Behavior |
| service.beta.kubernetes.io/aws-load-balancer-extra-security-groups | Classic load balancer
Network load balancer |
Adds additional security groups to the default one. Default SG is created and used. |
| service.beta.kubernetes.io/aws-load-balancer-security-groups | Classic load balancer
Network load balancer |
Specifies exact security groups for the load balancer.
Default SG is not used. |
| alb.ingress.kubernetes.io/security-groups | Application load balancer
|
Specifies security groups for an ALB |
Three Types of Annotations in Kubernetes
Unlike the CLB and NLB that we discussed in the above scenario, using AWS application load balancer (ALB) is straightforward since there is only one annotation you can use:
alb.ingress.kubernetes.io/security-groups
- Purpose: Specifies the security groups for the ALB. When you use this annotation, the default security group created by AWS for the load balancer is replaced by the specified security groups. There isn’t a separate annotation for extra security groups; simply list multiple security groups within this annotation to configure your ALB’s security groups.
- When to use: Use this annotation for ALB to define one or more security groups that will replace the default security group for enhanced security management.
Best Practices for Avoiding Security Groups Misconfigurations
There are numerous practices, some of which you might already be familiar with, to avoid this misconfiguration:
- Specify Primary Security Groups: It’s generally safer to use beta.kubernetes.io/aws-load-balancer-security-groups. But you should choose the one that best fits your use case.
- Review Security Groups Rules: Regularly audit and adjust AWS’s security group rule configurations to ensure they meet your security standards.
- Automate Audits: Use tools like AWS Config to automatically check for and alert for security group misconfigurations.
- Stay Informed: Continuously learn and stay updated with AWS and Kubernetes best practices to avoid common pitfalls.
The Risks of Not Securing Your Configurations
Understanding the distinction between security groups annotations is crucial for securing your AWS load balancers in Kubernetes. The risks of not configuring these annotations correctly can be critical and include:
- Data Breaches: Overly permissive security groups or misconfigurations can expose sensitive data to unauthorized access, leading to data theft, privacy violations and financial loss.
- Service Disruption: Improper security configurations can leave your applications vulnerable to attacks, resulting in downtime and loss of availability.
- Compliance Violations: Failing to secure your infrastructure properly may lead to non-compliance with industry regulations, potentially causing legal penalties and reputational damage.
Final Thoughts
It is essential to validate your code — especially if you are copying it from sources like Google or ChatGPT. Even seemingly small changes in Kubernetes annotations can expose your environment to unnecessary risks. It is important to familiarize yourself with the different kinds of annotations, what each one is doing and the implications of even small changes.
The other key factor is understanding how LLMs are changing the game. LLMs are becoming increasingly popular across various aspects of work, and this Kubernetes annotation scenario is just one of many emerging with the widespread use of these models. While LLMs provide useful suggestions, they may also suggest something that might not fit the situation and could introduce an unnecessary security risk you didn’t count on.
Stay driven, keep learning, and stay caffeinated.
