Running Production With Zero Stored Logs

I run a Kubernetes platform without a centralized log archive. When I say that, people often imagine something else: short-term retention, logs with sensitive information removed, or something similar. Let me explain it in simpler terms. This platform has no centralized mechanism for collecting, forwarding or archiving application logs. We’ve been running this configuration in production long enough to describe what it’s actually like.

The platform itself is standard. More than 15 microservices run on K3s. The nodes are located on VPS servers in different locations and communicate over a private WireGuard network. With Ansible, new nodes can be integrated into the cluster from a bare VPS in minutes, without manual intervention.

There’s one thing that’s different. It’s a constraint I set during the design phase: Observability cannot become a means of collecting information about users. The wording matters here. If you use the word “store,” you can create a system that allows you to “delete later,” as is typical. “Collect” removed most of this mechanism before the initial service was released.

Much of what’s called privacy engineering in the industry is actually retention management. First, you aggregate all the data. Then you promise to handle it carefully, including deletion schedules, rules and who can request what. All these promises depend on processes that must continue working. Alternatively, they may depend on lifecycle rules set by someone in 2021 or access policies that need to survive the next organizational restructuring.

Seeing how these systems deteriorate over time, I’ve lost faith in such guarantees. If sensitive data isn’t recorded in a centralized logging system, it can’t leak from that system because of an engineer’s mistake or a misconfigured log shipper. The question then became whether a true multi-node platform could function without a centralized log archive at all.

Every SRE I explained this configuration to initially reacted the same way. Nobody believed it was possible to analyze incidents without logs.

Metrics Instead of Logs

For observability, the nodes send metrics rather than logs, using technical labels for things such as load, latency, queue depth and error rate. This rule is enforced when metrics are ingested, not during later review.

Each application scrape job has a short list of allowed label names in its Prometheus configuration. If an exporter creates labels outside this list, those labels are removed before the samples enter metric storage. Cluster infrastructure jobs retain labels such as namespaces and pods, but do not retain user data. Application series are protected by the following configuration:

metric_relabel_configs:
  - action: labelkeep
    regex: __name__|job|instance|service|route|method|code|le|quantile

__name__ is not just for show. At this relabeling stage, the metric name is itself represented as a label. Without it, the metric name would be removed, making the affected samples invalid.

There is an important limitation: Metric relabeling occurs after scraping, so the configuration prevents disallowed labels from being stored but does not prevent Prometheus from receiving them in the first place. Application instrumentation must therefore avoid exposing sensitive user information in its metrics.

The allowlist also cannot verify label values. Therefore, there is one privacy rule in code reviews: The route label must be a template (e.g., /users/:id) and not a raw path.

Alerts fire based on user-visible symptoms and are calculated from the same metric series, such as route latency or work queues where processing has stalled. Therefore, the alert system did not require logs.

Debugging Without a Log Archive

The main objections are always about debugging. Whenever something goes wrong, there’s no centralized log archive to search with grep, and every reviewer asks what’s going to happen next.

In reality, metrics allow me to do far more than conventional logging habits might suggest. Metrics were designed for this task, rather than being added later alongside a large volume of logs. High-resolution series typically allow us to narrow down an incident to a single service and a single behavior within minutes.

If that’s still not enough, which is rare, I temporarily switch one of the services to verbose mode for as long as it takes to identify the error. I change the LOG_LEVEL variable in one deployment, reproduce the problem and use kubectl logs -f to inspect the real-time output. After that, I roll back the change.

There is one important distinction regarding the nodes themselves: Container standard output is still captured locally by the container runtime and made available through Kubernetes. That output is subject to the node’s log rotation and cleanup behavior. Rolling back a deployment does not, by itself, guarantee that all local log files are immediately deleted. Additionally, journald on each node is configured to use volatile storage.

This is not an environment in which log lines never exist. It’s an environment without a centralized system for collecting and retaining them.

Independent Review

An external engineer reviewed the platform, but didn’t take the diagrams at face value. He traced the entire node path, from a bare VPS to a cluster member. Next, he checked traffic within the WireGuard network and traffic originating from the nodes.

He then looked for the gaps that typically exist between design specifications and the actual system. Before starting the work, he told me there might be blind spots in incident visibility.

The telemetry traffic he observed matched the scraping configuration: Metric series were collected with the allowed labels, without a separate centralized log-forwarding pipeline.

The Trade-Offs

Such choices come with their own costs. Metrics must be carefully considered beforehand, because there is no log archive to answer questions that weren’t anticipated.

Diagnosis also requires discipline. Verbose logging that remains active after an incident can accumulate information that the architecture was designed to avoid retaining. Temporary diagnostics must remain temporary.

Furthermore, large teams with compliance audits have legitimate needs that this model cannot address. If you need to provide auditors with an immutable audit trail, you should build an audit pipeline, not this architecture.

However, this approach may become increasingly useful for teams facing data minimization requirements. The default solution of collecting everything and managing it later becomes costly and vulnerable as systems grow. The advent of cloud-native tools has made alternatives practical. High-resolution metrics storage has become more accessible, and encrypted networks like WireGuard have become commonplace.

By changing a single line in the declarative configuration, I can enable or disable verbose diagnostic logging when it’s needed.

This platform has never required a choice between visibility and privacy in its routine monitoring. The constraint was built into the architecture from the beginning. Incidents and capacity issues are detected as expected. And the user information that might otherwise accumulate in a centralized log archive isn’t protected by encryption or retention schedules. The platform was designed not to collect that information through its logging system in the first place.