Securing GenAI Workloads: Protecting the Future of AI in Containers 

Generative artificial intelligence (GenAI) has taken the world by storm, and nearly every company is racing to introduce some form of AI into its offerings. A McKinsey Global Survey reveals that one-third of survey respondents regularly use GenAI in at least one business function, and 40% expect to increase AI investments due to advancements in GenAI. With the advent of new models, agentic AI and more, the race to apply AI to business is just beginning. For many in the AI race, security has been left behind as experimentation and speed of creation are prioritized ahead of protection. We have seen it before with groundbreaking technologies that create a competitive advantage. As the world tries to figure out how to create and manage these projects, security is an afterthought. According to the National Institute of Standards and Technology (NIST), AI systems are susceptible to a wide range of attacks, both old and new. Given the unique nature of these applications with their access to sensitive data, it is crucial to protect these fledgling systems. While there are nuances, we can have a profound impact on the success of AI applications by securing a technology we already know well: Containers.  

Prevalence of Containers in AI Deployments 

AI workloads — the tasks and processes that AI systems perform — can range from data processing and model training to real-time inference and decision-making. These workloads require scalable, sophisticated infrastructure capable of handling large-scale data processing and analysis. As AI models become more complex and datasets become larger, systems require continuous updates and tuning to ensure they remain effective at scale. 

Similar to the most modern, rapidly deployed applications, AI applications are developed in small, ephemeral, self-contained portable units known as containers. According to IDC reports, AI applications will drive the creation of one billion—yes, billion—new containers in the next three years. Why such a large number? Because containers and large language models (LLMs) are a powerful combination. Benefits of containerized AI applications include: 

• Portability: Containers are small and portable, meaning AI models can move seamlessly across environments to align with platforms and business goals, including private clouds, public clouds and even on mainframes. 

• Scalability: Containers enable efficient scaling of AI workloads by dynamically adjusting to demand faster deployment methods. 

• Efficiency: Containers optimize resource utilization, ensuring you get the most out of hardware investments. 

• Flexibility: Containers provide a consistent and flexible environment for AI/machine learning (ML) workload deployment. 

Cloud providers and other infrastructure organizations are reporting exponential growth in container-based AI workloads owing to the benefits mentioned above. One report shows more than 70% of GenAI applications are deployed in containers. Gartner shows similar predictions, expecting more than three-quarters of all AI deployments will use container technology by 2027.  

In this world where AI is tied into our most critical systems and data, AI security must include a lens of container security. 

Securing Containers Across the AI Application Life Cycle 

AI workloads and resources introduce novel risks — AI model poisoning, data integrity threats and other AI abuse. It is important to consider that while businesses can deploy various tools to help secure CI/CD pipelines, these solutions don’t typically cover ML pipelines, which use different workflows and tools. Supply chain vulnerability management should be part of a comprehensive AI security plan. Without proper security, from code commit to runtime, AI-powered applications become high-value attack targets with readily exploitable security holes.  

If AI runs on containers, then securing AI requires securing containers and preventing, identifying and remediating attacks against AI systems, starting with the same cloud native security best practices. Important first steps include performing a risk assessment to identify security weaknesses in AI systems and strengthening access controls to restrict unauthorized access to AI services and tools. Applications that connect to AI systems must be scanned for vulnerabilities as well. Compromised apps linked to AI services highly attract threat actors, who can use them to compromise the AI services themselves. 

When it comes to securing AI applications, companies should incorporate the following strategies into their security plans: 

1. Secure the Code From First Commit

• Employ code scanning to identify and mitigate unsafe use of LLM in application code, including unauthorized data access, misconfigurations and vulnerabilities specific to LLM-powered applications. Evaluate against the OWASP Top 10 LLM Risks to identify security gaps.  

• Scan infrastructure as code (IaC) and AI model repositories for security vulnerabilities and compliance issues. This helps identify and address potential issues before they are deployed to production. Using trusted sources ensures that your application is free from vulnerabilities, leading to safer and more reliable deployments. 

• Implement software bill of materials (SBOM) for visibility into software supply chains, enabling detection of insecure dependencies and other risks. 

• Building on SBOM, implement the concept of the AI bill of materials (AIBOM), which records the resources used to develop and deploy an AI application or service, including data sources, AI model, parameters and training procedures that a team relied on when developing an AI solution. 

2. Secure the Build Process

• Scan container images for vulnerabilities, misconfigurations, malware and secrets before deployment to prevent teams from unintentionally introducing vulnerabilities, bad configurations or secrets into container images. 

• Create and enforce policy-based security controls based on container behavior and ensure that containers only do what they are supposed to do in the application context.  

• When possible, automate these actions to prevent security gaps, leverage technology to detect and prevent activities that violate policy and defend against container-specific attack vectors. 

3. Secure Deployment and Runtime

• Secure applications against new and emerging AI attack vectors with runtime protection. Ensure you can monitor and respond to threats in real-time, leveraging real-world threat intelligence to identify and block malicious container activity automatically. 

• Implement runtime anomaly detection specifically for AI-driven workloads to prevent unauthorized actions that LLMs might attempt, such as executing malicious code due to prompt injection attacks. 

4. Secure the Interactions of the Application With the Model 

• Enforce strict access controls to limit interactions with the LLM to authorized users and applications, protecting against unauthorized use and data breaches. 

• Leverage secure execution environments, which isolate LLMs from potentially harmful external influences and provide a controlled setting for AI operations. This is crucial for protecting the integrity of AI processes and preventing the exploitation of vulnerabilities within the operational infrastructure. 

5. Implement Continuous Observability and Compliance

• Deploy monitoring mechanisms to detect anomalous behavior or potential security breaches in real-time. This includes monitoring data inputs and outputs, model performance metrics and system logs for suspicious activity. Additionally, there should be vulnerability management processes in place to promptly address and remediate security flaws as they are discovered. 

• Be proactive and employ AI-Specific Security Posture Management (AI-SPM) to mitigate AI-related risks and ensure LLM-powered applications are meeting the latest compliance standards. 

• Implement AI/LLM assurance policies to serve as guardrails for developers, preventing unsafe usage of LLMs. 

Because traditional security tools don’t understand container-native behavior, companies should employ specialized tools that address both container and AI-specific security challenges. Some capabilities to consider are threat intelligence for AI-focused attacks and runtime security for AI workloads. 

The Business Value of AI-Optimized Container Security 

It can almost be understood that proper container security for LLM apps benefits the business, but it is worth repeating. AI container security will reduce business risk, increase time to deploy new apps and reduce mean-time-to-remediate issues. All of this reduces the risk profile of deployed applications in a measurable way. Further, real-time protection of the app in production will prevent successful workload attacks. In doing so, they can avoid AI-driven breaches and compliance violations while ensuring their teams have the resources and tools they need to scale security for these complex applications. Companies will gain speed and agility without slowing down AI innovation. With more secure, reliable AI, they will see greater competitive advantages and more positive business outcomes.  

AI technology has introduced new capabilities for applications, empowering businesses to deliver personalized experiences and increase efficiency. However, these innovations create risks that demand proactive strategies to mitigate potential security compromises. Because more AI applications run on containers, the best approach spans the entire life cycle of AI applications, from code commit to runtime, with a focus on container security. Employing this approach will ensure that the business remains competitive and innovative and achieves business outcomes without compromising security.