As we close out 2023, we at Cloud Native Now wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.
A year ago, I told you that OTel was at the core of Sumo Logic’s collection strategy and that we were betting our future on OpenTelemetry. Back then, I said that unified collection was a unicorn idea that the industry was trying to pursue. Today, I am happy to share that we have launched fully unified Kubernetes monitoring for all signals (logs, metrics, traces, events)! With the advent of our latest Helm chart release, we not only have fully-unified collection, but the project includes auto-instrumentation and more efficient collection than prior Helm chart releases.
“But what is a Helm chart, and why does unified collection matter for Kubernetes monitoring?” you ask.
Understanding Helm Charts and Unified Collection
Think of Helm charts as the package manager of the Kubernetes universe. For those working with Linux-based systems, it’s akin to the “apt” or “yum” for Kubernetes. They are designed to encapsulate and simplify the deployment, installation and upgrade processes for Kubernetes applications.
Practitioners still need to monitor and manage their Helm chart versions while adhering to a semantic versioning convention. This method offers significant manageability, allowing operators and developers to precisely deploy specific application versions, ensuring both compatibility and stability. Furthermore, if a new chart version presents breaking changes or encounters issues, operators have the flexibility to effortlessly roll back to a prior stable version. Additionally, Helm charts can be interdependent on other charts, and this structured versioning ensures that dependent charts harmoniously align with their correct versions.
OpenTelemetry has now expanded beyond unifying receivers and making a unified collection of signals. Now, there is an increase in automation via auto-instrumentation and a relatively new working group for semantic conventions. Each of these working groups works with Helm charts quite closely to achieve the original goal of the OTel project (which is to unify collection).
The Advent of Unified Collection for Kubernetes Monitoring
The image on the left shows the chaos of disparate and separate collectors. You have a hodgepodge of vendor-provided collection, instrumentation and backends. This is what data collection looks like without using the OTel standard. Your telemetry pipeline is backed up with myriad agent-specific code, and you are likely still using manual instrumentation held together by various ad-hoc scripts.
On the right, you can see what the architecture for a fully unified OpenTelemetry pipeline looks like. This is what the latest Helm chart release is able to provide for Sumo Logic users. Sumo Logic’s commitment to open source projects further solidifies the potential of unified metrics, events, logs and traces (MELTs) collection processes in the foreseeable future. This enables correlated telemetry, auto-instrumentation and a single agent to allow for a smaller, more efficient data collection footprint.
The transition to a unified monitoring approach with Kubernetes marks a pivotal moment in how we approach and manage data collection. This cohesive framework, backed by auto-instrumentation and semantic conventions, will not only unify collection but begin to define the future of the OTel project.
The Future of OpenTelemetry
Now that unified collection is possible through a single agent, what’s next for the OTel project and community? Is OTel still relevant?
As we step into the future, OTel will have to grow beyond unifying receivers and collection agents. The new wave of work coming from the project seems to be more focused on ease of use and automation. It is already starting to focus on standards for agent collector management with Open Agent Management Protocol (OpAMP) compliance.
The OTel standard seems to be trending toward remote management of vast fleets of data collection agents that align with the OpAMP protocol. This ensures ease of use via efficient agent management by facilitating real-time agent status reporting and configuration management en masse. Additionally, OpAMP touts a variety of security features, including auto-updating capabilities that support both upgrades and downgrades, TLS certificate rotation and a robust connection credential management system. These automation features enable secure communication between agents and servers, thereby minimizing potential threats.
The most notable feature OpAMP unlocks for observability vendors is streamlined remote configuration. OpenTelemetry agents have intricate functionalities and demand precise configurations to capture data accurately. Unified collection on fewer agents is great, but more is needed for ease of use and configuration for agents themselves. Dynamic adjustment capabilities from OpAMP based on agent status reports and the ability for these agents to report to an OTLP-compatible backend means that users can keep track of agent-specific metrics dynamically (such as resource usage and data processing rates). This ensures that agents function within desired thresholds and fosters proactive management without practitioners experiencing burnout.
OTel agents often undergo updates or require new packages to enhance their data collection capabilities. OpAMP simplifies this process, making the distribution and installation of these packages straightforward and reduces the need for SRE acrobatics due to extensive manual intervention. Standardized protocols like OpAMP will become indispensable to unlocking ease of use for the future of simplified, secure, and efficient agent management.
If you would like to learn more about Kubernetes monitoring best practices, check out my e-book. By reading that, you can learn more about the history of Kubernetes, a primer on Kubernetes architecture, and how to leverage OTel for K8s infrastructure monitoring. I also include a section on future trends and where the cloud observability industry seems to be heading.
To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2023 – November 6-9, 2023.