Best of 2023: Why Container Runtimes Still Matter

As we close out 2023, we at Cloud Native Now wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.

Peter Hunt co-authored this article.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Today, Kubernetes and associated projects are the focus of most attention in cloud development—and the word “containers” is almost forgotten except as an implementation detail. Nevertheless, the CRI-O container runtime project is growing and frequently implementing new features to improve cloud flexibility, security and performance. Importantly, this is work Kubernetes users need and can only be done at the container runtime level.

The CRI-O container runtime project has reached graduated status in the Cloud Native Computing Foundation (CNCF), showing its maturity alongside another major container runtime project, containerd. This shows the completeness and stability of both container options for Kubernetes. However, major work to improve container runtimes continues, including managing devices and plugins, securing supply chains and creating a new container monitor written in Rust. Much of this work is done in coordination between both projects, whose relationship can be described as a “friendly rivalry.”

“The collaboration between containerd and CRI-O maintainers allows us to develop complex Kubernetes features or maintain projects like cri-tools together,” said Sascha Grunert, CRI-O maintainer and Kubernetes SIG-Release Chair.

What’s a Container Runtime?

Most people know Kubernetes as a service that runs containers on your server cluster. But Kubernetes doesn’t actually run containers; instead, it schedules them and then sends requests to container runtime programs on each individual server node using the Container Runtime Interface (CRI). The runtimes handle the actual downloading and execution of the containers. There are two mainstream projects, CRI-O and containerd, both of which are open source projects under the CNCF.

“Folks can think of Kubernetes (or more specifically, in this context, the Kubelet), CRI and CRI-O as the boss, the language they speak, and the employee. Kubelet delegates the actual work of the container/pod/image operations to CRI-O and delegates those responsibilities via the CRI protocol,” explained Peter Hunt, CRI-O maintainer.

In the early days of Kubernetes, it ran containers using the Docker runtime. Docker is a heavyweight, interactive developer environment, though, and not really intended to be a small part of a larger system. As such, in 2016, Kubernetes contributors like Antonio Murdaca and Mrunal Patel started work on a container runtime that would fulfill the CRI and nothing more and named it CRI-O. A year later, Docker developers separated the container runtime kernel of Docker, named containerd, and contributed it to the CNCF. Both projects enjoyed rapid development and in 2022, the Kubernetes project was able to drop support for using full Docker as its container runtime, instead offering a choice of CRI-O and containerd.

In addition to the CRI implementations of CRI-O and containerd (sometimes also called high-level container runtimes or container managers), the OCI and CNCF ecosystem is constantly evolving, inventing new ways to run containers and customize behavior. Kata Containers, for instance, is an OCI runtime (sometimes called a low-level runtime) that runs a pod within a virtual machine (VM). Many users want even further customization of resources and container creation, especially when they are supported by both CRI-O and containerd.

NRI Adds Plugins to Container Runtimes

While the alternate container runtimes are exciting, ultimately, they represent a fragmentation problem for container technology. Having dozens of container runtimes, particularly some designed to support special hardware devices, would be confusing and problematic for users. The specification team at the Open Container Initiative (OCI) recognized this in 2017 and started working on a way to extend the container runtimes. Runtime developers tried several different things, and eventually, this resulted in the Node Resource Interface (NRI), a specification and API that allows vendors to build container runtime plugins. NRI has been a completely collaborative project between containerd and CRI-O developers. It was released as version 0.5 in September, and the project expects to reach 1.0 stability at the end of 2023.

NRI allows plugin authors to modify the behavior of the container runtime to support things like pinning the container execution to a particular CPU core or supporting specialized hardware. AI and machine learning systems can run containers on NVIDIA GPUs. This allows system vendors and owners to write device managers that optimize container behavior for their specific hardware, even edge devices.

As Kubernetes expands into edge computing and AI use cases, being able to run in those special hardware environments is increasingly in demand. Kubernetes cannot enable them on its own; it needs the support of container runtime functionality. NRI will deliver that.

Re-Writing Conmon in Rust

Even container runtimes involve multiple components. One of these is conmon, short for “container monitor,” which dispatches execution requests and reacts to container events. Conmon dates back to the creation of CRI-O and, as a result, is weighed down with older code and technical debt. In 2022, the CRI-O maintainers launched a rewrite of conmon in Rust. Rust, a newer programming language, has been thrust into the public eye because of its memory safety, which caused the U.S. Cybersecurity and Infrastructure Security Agency to give the language special attention.

“Conmon, while serving faithfully for the lifetime of CRI-O, has become unwieldy to work with and difficult to extend,” said Hunt. “Further, there are more processes in the mix than need to be. Conmon-rs benefits from being written in a modern language and will be much easier to extend.”

Securing the Image Supply Chain

Aside from the rewrite of conmon, the CRI-O project is working on security elsewhere, particularly the integration of Sigstore signature verification. Sigstore is a project that permits cryptographically verifiable signatures on software components, ensuring that they actually originate where they’re supposed to. Kubernetes, which publishes its own official container images with signatures, supports security policies that let administrators determine which verified sources they trust.

“Sigstore is a critical piece in supply chain security and CRI-O is incorporating first-class support for Sigstore verification working in conjunction with Kubernetes upstream,” explained Mrunal Patel, CRI-O maintainer.

By evaluating the signature policy directly in the container runtime, CRI-O and Kubernetes are able to ensure that an untrusted image will not be downloaded or run regardless of how the container is scheduled. In Kubernetes 1.28, CRI-O is even able to support per-namespace policies so that users can have different levels of trust for different tenants of their cloud. Full support for the feature is planned for the upcoming 1.29 release.

More Work to Do in Container Runtimes

In addition to the above, the CRI-O team is working on checkpoint restore for containers, a new metrics interface and event-based updates. containerd is codifying the container sandbox API and creating a long term support (LTS) release. Both are adding features and APIs to support new Kubernetes features like Dynamic Resource Allocation (DRA).

Container runtimes are still actively in development. While mature, they are not finished.

Runtime features are essential for innovation and feature development for the Kubernetes cloud.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2023 – November 6-9, 2023.