Most technology pundits agree that Kubernetes won the container orchestration wars, triumphing over alternatives like Docker Swarm and Mesos. Google even showcases a featured snippet for the exact moment of victory—November 29, 2017—the day AWS announced their Elastic Container Service for Kubernetes (EKS).
With growing adoption and a strong community foothold, Kubernetes appears to be the clear industry-standard container orchestrator. Yet, Kubernetes is no cure-all. Users cite mounting complexity with new features, developer experience qualms and security misconfigurations as potential roadblocks to implementation. It’s not that these drawbacks can’t be overcome, but most stakeholders admit Kubernetes’s inherent complexity makes onboarding to the platform and securing it rather cumbersome.
For these reasons, some still use alternative container orchestrators. I recently spoke with Chang Li from HashiCorp on why she foresees more organizations dropping Kubernetes in exchange for more usable options. Below, we’ll consider some of the drawbacks to Kubernetes and see how it stacks up against Nomad as an alternative.
The State of Kubernetes
Nearly 90% of container deployments are now orchestrated, research from Datadog finds. Half of container deployments are running Kubernetes, either self-hosted or in a managed service such as EKS, GKE, or AKS. “The orchestration market matured rapidly,” describes Li, with Kubernetes becoming the dominant player.
Yet, Kubernetes introduces much complexity, which is heightened as new features are introduced. And, managing multiple tools and workflows could be a burden for some teams. “The more feature-rich ones offer more fine-grained control, but at the expense of operational complexity,” writes Cindy Sridharan.
Some organizations are turning to a full-cycle approach, wherein a developer owns their code through to production. For this new paradigm to work, developer experience for DevOps tools is crucial. Yet, the operational overhead associated with Kubernetes is very significant, describes Li. “We see a trend of K8s getting more complex as it grows in more features,” Li says.
In the rush to adopt Kubernetes, some red flags were ignored. Issues could include a lack of security best practices for this new environment, as 47.9% of Kubernetes security violations were due to insecure defaults, a recent study finds. Some teams lack good transparency into container workload utilizations. Most Kubernetes instances are underutilized, causing potential overspending. Kubernetes is also primarily focused on containers, leaving out scheduling for alternative formats.
As a result, Li now sees a renewed multi-orchestrator pattern emerging. With the ubiquity of container scheduler usage, Li believes there is room in the market for alternative orchestrators to cater to specific needs. Multiple orchestrators may even coexist within the same organization.
At organizations heavily using Kubernetes, Li notes the growing adoption of Nomad for new projects among specific enterprise business units. These situations may require less fine-grained and more simplistic orchestration needs. Small and medium sized organizations without the staff to support Kubernetes may also turn to Nomad. For example, Roblox, Q2 Software, Cloudflare and Bowery Farming all use Nomad in production.
Nomad also is more flexible than Kubernetes when it comes to technology support. Whereas Kubernetes only supports certain container types (Docker, containerd, CRI-O, etc.), Nomad also supports non-container and legacy formats, making it a more generalized solution.
Kubernetes vs. Nomad
“Kubernetes has become the de facto standard for container orchestration,” says Datadog. Though Kubernetes is mature and versatile, it requires many components to manage. Here’s a general, high-level comparison between Kubernetes and Nomad:
- Fine-grained control.
- Superior community and support.
- Kubernetes has become a source of truth for some organizations, who are using it for other capabilities like RBAC.
- Expanding tooling community.
- Google, AWS, Microsoft all offer managed K8s versions.
- High complexity and difficulty to onboard.
- K8s itself has many interoperating components to deploy and manage.
- Requires new security forethought, especially for self-hosting.
- Potential for runaway costs.
- You must learn new networking models (node IP, pod IP, internal/external IP).
- May require additional tools to manage.
- Simpler and easier to get started with.
- Less configuration may be better for smaller teams who can’t handle the complexity.
- Helps deploy, manage different workloads: Nomad supports workloads outside of containers, like Java, Windows apps and even binary.
- No new overlaid networking model to learn.
- Immature in comparison to Kubernetes.
- Lacks diverse, strong community support.
- Narrower tooling community.
- Vendor-biased: intimately tied with HashiCorp ecosystem.
Never One Tool Fits All
When comparing divergent technologies, it’s good to recognize the complementary strengths. “There is always a space for a second solution; an alternative solution,” says Li. Since Nomad is “exactly the opposite of K8s,” Li believes it could fill some gaps that K8s won’t be able to offer in the long run.
If your team is already using containers in production, Kubernetes is a mature and versatile option. However, if you’re a small group or independent business unit wanting to get something together fast, especially one that supports various workloads, Nomad could be an excellent generalist approach.
When Kubernetes and Nomad, think of it this way:
“Kubernetes is primarily a platform for distributed containerized applications that happens to have a scheduler built in, whereas Nomad is primarily a flexible scheduler that can be used to build a platform comprised of hybrid deployment artifacts,” writes Cindy Sridharan.
When it comes down to it, the end goal is supporting an excellent application running on top, not the orchestration means, Li describes. Therefore, she foresees orchestration as becoming further abstracted from engineers in the future. Managed services have already taken a step in this direction. “Managed services alleviates a lot of operational complexity,” says Li.
Yet, the unique flavors of managed services make multi-cloud challenging to support. It requires substantial reconfiguration to migrate a workload from EKS to GKE, Li says. With some layers over Kubernetes, when something goes wrong, it could require double the effort to figure out problems, she adds. Kubernetes managed services or other container orchestration tools that hide operational complexity mustn’t cause additional troubleshooting.
“Organizations tend to chose an orchestrator based on their product, budget and timeline,” said Li. “It’s never one tool fits all.”