Auditing and Compliance in the Cloud-Native Era

Once upon a time, auditing and compliance were relatively straightforward—albeit tedious—tasks. Fast-forward to the cloud-native age in which we currently live and you’ll find the auditing and compliance landscape has changed dramatically. Infrastructure is largely ephemeral, applications use complex, distributed architectures and cloud environments are managed via complicated “shared responsibility” models dictated by cloud providers. Being able to simply determine which types of data to collect—let alone collect them—have become monumentally challenging tasks.

Traditional approaches to auditing and compliance no longer suffice in the cloud-native age. Businesses must fundamentally rethink the way they collect, interpret and report on the data that drives auditing and compliance.

DevOps Dozen 2023

There’s no one-size-fits-all auditing toolset or process that can solve every business’s compliance challenges. However, understanding these seven essential concepts can form the foundation for developing an auditing and compliance strategy that addresses the complex realities of the cloud-native era. No matter which types of applications you run, how your cloud environments are designed or which compliance mandates your business must meet, this guide can help you build an auditing and compliance strategy today.

Understanding the Compliance Landscape

The first step in devising an auditing and compliance strategy is understanding which compliance requirements your business is legally required to meet, as well as which ones it voluntarily chooses to meet.

These are complex issues for three main reasons:

  • New and emerging compliance mandates: The regulatory compliance landscape has changed significantly in recent years, with new frameworks like the GDPR and the California Privacy Rights Act coming online or set to do so soon. As a result, businesses must evolve their auditing and compliance strategies to meet additional—and, in many cases, stricter—rules that simply didn’t exist just a few years ago.
  • Complex applicability rules: While some compliance frameworks are straightforward in that they apply to all businesses within a certain industry or jurisdiction, others don’t take effect until businesses reach a certain size or extend their operations to a particular geographic region. This means that compliance rules that don’t apply to your business today could become relevant in the future as the business grows.
  • Regulatory versus Non-Regulatory Requirements: Not all compliance frameworks are regulatory frameworks with which businesses are legally required to comply. Some, like the SOC 2 and ISO 27001 rules, are essentially just security benchmarks that businesses may voluntarily adopt to help guide their auditing efforts. The challenge here is that businesses must identify which non-regulatory compliance rules it makes sense to meet while avoiding overburdening themselves with voluntary work.

As you can see,  the modern compliance landscape is more complicated than ever and there is no simple solution. Instead, businesses must do the hard work of understanding which rules apply to them—now or in the future—and devising strategies for adhering to them. This requires close alignment between legal teams who understand compliance mandates and IT organizations responsible for operationalizing those rules within their environments.

Understanding Which Data Matters

Once you know which compliance rules you need to meet, the second step in building a cloud-native auditing and compliance strategy is identifying which data to collect.

As a general rule, it’s a best practice to collect as much data as possible from as many sources as possible. This includes data—such as ticketing system logs and CI/CD pipeline events—that teams may not conventionally think of as being relevant for auditing and compliance purposes.

In some ways, this advice may seem counterintuitive. Given that cloud-native environments expose virtually limitless volumes of data, doesn’t it make sense to be strategic about which data you collect and analyze for auditing and compliance purposes? Isn’t it unrealistic to try to track every data point from every source? And, doesn’t more data collection increase the risk of data leakage and compliance violations?

For most businesses, the answer to those questions is a definitive no. Here’s why: The more data you have and the more diverse your data sources, the greater your ability to correlate discrete data points. This allows you to gain deep context on events that matter for auditing and compliance purposes that you otherwise might not be able to.

For instance, if an application bug accidentally exposed sensitive customer data, you could correlate your customer support tickets with your application logs to find out who was impacted and how they were impacted. Likewise, one could map access logs from a cloud IAM service with IAM policies to determine who accessed that sensitive data and possibly identify the cause—given it was a faulty IAM policy.

Auditing Ephemeral Infrastructure

A common challenge that IT teams and developers face when collecting data for auditing and compliance purposes is how to collect data from infrastructure that is ephemeral. Whether it be containers or serverless functions, these types of infrastructure have short lifespans—days, if not hours—and don’t persist data beyond their life cycle.

The key is to collect data continuously and in real-time. Pulling or pushing data won’t cut it, as various infrastructure components like containers don’t always live for long; therefore, if you don’t collect data before they shut down, it may be gone forever.

Of course, the challenge teams face to continuously collecting performance data from ephemeral infrastructure is orchestrating it. It’s no small feat to ensure that every container, serverless function and the like within cloud-native infrastructure is continuously observed.

To meet this challenge, teams must build observability into their infrastructure strategy from the very start. Just as important is to understand that the monitoring tools—like those offered by cloud providers—that may be available in ephemeral environments by default aren’t typically sufficient to collect all data in real-time. You’ll need to make sure to embed other observability tooling into your environments from the moment they are launched.

Auditing for Microservices

The main reason why auditing distributed, microservices-based applications is such a challenge is that each microservice needs to be observed independently. At the same time, teams must correlate data from the various microservices to audit the state of the application as a whole.

Here again, the key to solving this challenge is to bake observability into your application’s architecture from the very start by ensuring that you can continuously collect data from each microservice instance. In many cases, that will mean deploying ‘sidecar’ containers alongside the containers that host application microservices or configuring microservices observability through your service mesh, if your app uses one.

Either way, the point is that observability should be a first-order priority when designing and deploying your microservices app, not something you tack on once the app is running.

It’s just as important to ensure that you can relate data from each microservice to other microservices to gain a holistic understanding of your application. Doing so is important not just for managing the performance and availability of your application, but also for assessing compliance effectiveness.

The Shared Responsibility Model and Modern Auditing

Yet another challenge that teams face in the context of modern cloud-native auditing is interpreting cloud providers’ shared responsibility models for auditing and compliance purposes.

Virtually all cloud providers adopt shared responsibility architectures that assign some compliance and security responsibilities to customers while leaving it to cloud providers to manage others. In many cases—especially with conventional IaaS resources like cloud VMs or object storage—these models are fairly straightforward and easy to understand.

Shared responsibility becomes trickier, however, in the context of managed services like hosted Kubernetes platforms. These services blur the lines between backend infrastructure (which cloud providers secure) and customer-managed resources (which customers must manage) more than standard IaaS services.

It may be easy for customers to assume that cloud providers assume responsibility for securing the Kubernetes clusters that they offer. But in fact, core security responsibilities like API access control and cluster software upgrades typically fall to customers. This matters from a compliance perspective because it means that customers can’t claim that their Kubernetes environment is secure and compliant based on the cloud provider’s guarantees alone. Customers must do more to demonstrate that they have sufficient compliance safeguards in place.

This is why comprehensively observing cloud services to the fullest extent possible forms the basis for successful cloud-native compliance. Even if your cloud provider guarantees compliance and security standards, collecting and analyzing as much data as possible from your cloud environments puts you in the strongest position to meet and demonstrate your own compliance goals.

Understanding Cloud Providers’ Auditing Tools

In addition to confusion surrounding the shared responsibility model, businesses may run into auditing and compliance challenges because they lack a full understanding of what their cloud providers’ auditing and monitoring tools can and cannot do.

Most of the major public clouds offer basic compliance assessment tools, such as AWS Audit Manager. However, these tools don’t suffice for complex, large-scale auditing and monitoring needs. Most of them can only detect compliance violations based on a limited selection of specific compliance frameworks, which may not include all of the rules that your business needs to meet.

The takeaway is that teams should look beyond their cloud providers’ tools for auditing and compliance purposes. While the providers’ native tools may be useful in certain contexts, they don’t offer the comprehensive, cloud-agnostic observability and data correlation functionality necessary to meet all compliance requirements across all types of environments.

Embracing Continuous Auditing

A final key lesson for cloud-native compliance and auditing is to embrace continuous auditing. Continuous auditing allows for the detection of potential compliance violations and the generation of auditing report data on a continuous and ongoing basis.

Historically, this was not how most teams approached auditing. Audits were traditionally performed periodically — a practice that worked well enough when infrastructure didn’t change rapidly and the data you collected at one point in time reliably represented the state of your environment.

However, in a cloud-native world where environments evolve constantly, periodic audits aren’t sufficient. Today’s businesses must be able to observe all of their IT resources in real-time and detect compliance issues as they appear. Continuous data collection and correlation make this possible.

Just as important, teams should also be able to track the historical state of their environments to determine exactly when a compliance issue occurred and which resources it may have impacted over time. Identifying the issue is one thing, but knowing how long it has existed and what its scope was are what matters.

The Brave New World of Auditing and Compliance

In short, auditing and compliance success in cloud-native environments hinges on an observability strategy that is:

  • Comprehensive, in terms of the types of data collected.
  • Continuous, meaning that data is collected from ephemeral resources in real-time.
  • Correlative, because correlating disparate data sets is the only way to achieve the context necessary to assess complex compliance requirements.
  • Rooted in tools other than those that cloud providers offer natively.

By embracing these principles, businesses can thrive on the auditing and compliance front even as their environments grow more complex and as the compliance landscape becomes more and more complicated.

Knox Lively

Knox Lively is the lead technical evangelist at Observe. An accomplished writer, published author and former DevOps Engineer with over 12 years behind the keyboard, Knox builds community, drives engagement and educates the public about the company's product.

Knox Lively has 1 posts and counting. See all posts by Knox Lively