Why is Standardizing on Container Best Practices so Hard?

If you are the technology leader of your organization, you are likely driving the shift to cloud-native or even multi-cloud architectures to capitalize on the scalability, elasticity and agility it offers. And you are not alone. Gartner predicts that by 2022, more than 75% of global organizations will be running containerized applications in production, and three-fourths of respondents to Hashicorp’s 2021 State of the Cloud Strategy Survey are already deploying multi-cloud. As we embrace cloud-native, it is important to remember that ‘we are what we eat.’ The foundation of cloud-native is containers, and the quality of our containers ultimately determines the quality of our cloud-native approach.

Adopting containers is much like purchasing your first sports car or high-end home theater system—you don’t really experience the full power of the technology until you learn how to configure the settings and adopt the lessons learned by experts that came before you. Container best practices (CBP) are essentially a catalog of configurations, tweaks and maintenance steps that can optimize the size, performance and security of containers. 

The benefits of CBP include improvements to:

  • Build times (higher velocity, increased innovation speed) 
  • Developer experience (better tooling, less technical debt)
  • Stability (fewer bugs and dependencies, less risk)
  • Security (reduced attack surface, software bill of materials)
  • Cost savings (reduced cloud storage, bandwidth and startup times)

Practicing CBP sounds like a no-brainer, and in an ideal world, few would argue against implementing CBP. But in reality, it’s more complicated than that. For starters, whose job is it to implement CBP?

Today’s CBP Champions are Doing It the Hard Way

Many of those currently implementing container best practices are a rare breed of legitimate container experts and power users. They have coveted job titles in software engineering today—cloud architect, DevOps engineer, principal infrastructure engineer—but what sets them apart is their deep understanding of how their containers are built. They understand that container best practices are more than aesthetics. They are about optimizing containers for size, build speed, attack surface and performance—and they understand how these factors have dramatic, positive implications for developer velocity and security.

Here’s a typical scenario: A developer starts a new project by pulling an image from a public repo or building from a Dockerfile found on GitHub. They add code and run the container. They ship to production, passing security scans on the way. Simple, right? But baked into that simple process are the types of questions that keep developers awake at night. 

Is the image up to date? Is it well maintained? Why is it so big? Why does it take so long to build and scan? Does it have everything it needs to run (and nothing more)? Is there anything in the image that poses a risk? Will it even run?

Answers to these questions are not always easy to find.

Some experienced developers with an eye toward production or delivering base images to their teammates may tackle building and optimizing the image themselves. Perhaps they use Alpine Linux, distroless images, or multi-stage builds alongside open-source tools like Dive or DockerSlim. They know the perils of being unaware of what’s in their images and have enough accrued expertise to know CBP are worth the effort. The challenge is that optimization often means trade-offs between images that provide the tools needed for a great developer experience and optimized, lean, secure, slim images that may be production-ready. Striking that balance is frequently taken for granted.

Unfortunately, many organizations don’t have a container expert on hand who cares enough to do all that manual work, so they just default to using bloated, vulnerable containers and hope for the best. And while most developers and DevOps engineers would rather not think about CBP at all, a small number of big companies that care deeply about developer experience are building their own internal tools to ease the strain and friction. By and large, there is a huge gap between the CBP capabilities of those few, large companies and the rest of the companies in the world.

How do we Shift Left Without Piling Work on Developers? 

Savvy CTOs are actively thinking about how they “shift left” the onus of container performance and security in their organizations—without impacting developer productivity. And that’s a hard thing to solve. Priceline CTO Marty Brodbeck described the challenge this way: 

“It is the age-old problem where the bug gets into production and then you have to go back all the way to the design in order to figure out where the issue was. It would be nice if you understood that problem at the design level and could iterate the testing of it before having it in the production.”

Or, as the saying goes, “An ounce of prevention is worth a pound of cure.”

If your DevOps teams are not using CBP today, you’re probably struggling with a millstone of technical debt—even if you’re a proponent of SRE and error budgeting. You’re ultimately going to have to pay the piper when bloated, insecure containers become a drag on speed, expose you to threats, drive up costs and drive away your developers.

To be clear, I am not saying that your application developers should be held accountable for manually optimizing containers. Ditto for your ops teams.

What I am saying is, let’s not do this manually at all. Let’s automate everything. Let’s create a way to give our developers, ops teams and related stakeholders the CBP outcomes we all want in an automated, rules-based way so that no one has to take on the manual task of CBP. 

And while we’re at it, let’s make sure we make this solution available to every startup and every small business too.


To hear more about cloud-native topics, join the Cloud Native Computing Foundation and cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

Josh Viney

Josh Viney has a sincere love for creating products that make people's lives better. A 20-year veteran of product design and strategy, Josh is a product leader and entrepreneur who has created products used by hundreds of thousands of technologists and teams around the world. He led the developer-experience team at Digital Ocean and now serves as Head of Product for Slim.AI, where he's focused on helping developers create and run their cloud-native apps with zero friction, complexity or waste. 

Josh Viney has 1 posts and counting. See all posts by Josh Viney