Ephemeral, Idempotent and Immutable Infrastructure

DevOps works best when the infrastructure is engineered for developers as a flexible “elastic” service.

Ephemeral and Elastic Resources

If a small, low-risk software change is committed to the trunk branch, the continuous integration (CI) stage of a DevOps pipeline needs minimal infrastructure (compute, storage and network) to do the work of building, testing and packaging the change. When a large or risky change is committed, the CI stage needs significantly more infrastructure. In either case, when the tasks are complete, the infrastructure is no longer needed.

The same level of work demand variance exists throughout all stages of the pipeline. Bear in mind that for any one application, there may be many developers committing code frequently but asynchronously. These different code commits may be of various sizes and risk levels, so the infrastructure needed to build and test them must be highly flexible.

Real infrastructure consists of real, physical computers, storage and network devices that do not magically appear and disappear depending on work demands. But with modern infrastructures suited for DevOps, those real systems exist in distributed, highly configurable and secure resource pools that are dynamically configured on demand. From a developer’s point of view, these resources appear to be ephemeral and elastic.

Idempotency and Immutable Infrastructure

I have a recurring nightmare in which I drive on the same road, but it leads me to a different destination every time. Same car, same driver. What happened to the road? This nightmare also can happen to computer systems infrastructure. When the same program is run on computing infrastructure, you can get a different result if the computer system configuration (hardware settings, operating system environment variables, versions of code in the software stack, etc.) has changed between executions. This can happen when one server is patched but not others or the patch process is not completed correctly. Over the course of multiple patches, the configurations drift away from what is intended. There are two words to describe these characteristics that you need to know to understand elastic infrastructure—idempotent and immutable.


Configuration management systems are designed to prevent infrastructure configuration drift. They purport to guarantee idempotency—the system configurations for infrastructures remain consistent so that you get the same result each time they are used. Configuration management systems, such as Puppet and Chef, were designed around the concept of idempotency. They scan infrastructure components to validate that their configurations match what they are programmed to expect and update configurations that are not compliant. These systems are a boon for infrastructure managers who need to ensure the infrastructure does not drift between uses. However, there are several problems with this approach. The configuration update process can take a long time if a large stack of code is imaged. And every time a configuration image is built, there is a risk that unintended changes can creep in. Unintended changes can cause functional, performance or security problems. If this occurs, the idempotent tools ensure bad configurations are applied to every server!


Immutable infrastructure refers to managing services and software deployments on IT resources wherein containerized components are replaced rather than changed. An application or service is effectively redeployed each time any change occurs.  Immutable infrastructure benefits include lower IT complexity and fewer failures, improved security and easier troubleshooting than on variable infrastructure. It eliminates server patching and configuration changes because each update to the service or application workload initiates a new, tested and up-to-date instance. There is no need to track changes. If the new instance does not meet expectations, it is simple to roll back to the prior known good instance. Since you are not working with individual components within the environment, there are far fewer chances for unpredictable behaviors or unintended consequences from code changes.

What This Means

This blog explained how ephemeral, idempotent and immutable concepts, together with containers, can provide the flexible elastic infrastructures needed for variable DevOps workloads. Those who are serious about DevOps would do well to learn about the advantages that containers offer as part of a DevOps solution. To learn more about immutable containers, refer to my book Engineering DevOps.

Marc Hornbeek

Marc Hornbeek, a.k.a., DevOps-the-Gray esq. is a globally recognized expert for DevOps, DevSecOps, Continuous Testing and SRE. He is CEO and Principal Consultant at Engineering DevOps Consulting , author of the book "Engineering DevOps", and Ambassador and Author for The DevOps Institute . Marc applies his unique, comprehensive Engineering Blueprints, Seven-Step DevOps Transformation Blueprint and 9 DevOps Pillars discovery and assessment tools, together with targeted workshops skills to create actionable and comprehensive DevOps transformation roadmaps and strategic plans. Marc is an IEEE Outstanding Engineer, and 45-year IEEE Life member. He is a DevOps leadership advisor/mentor. He is the original author of the Continuous Delivery Ecosystem (CDEF) and Continuous Testing Foundations (CTF) certification courses that are offered by the DevOps Institute. He is a Blogger on DevOps.com and cloudnativenow.com. He is a freelance writer of DevOps content including webinars, and white papers. He is a freelance trainer for DevOps, DevSecOps and SRE courses offered by partners of the DevOps Institute.

Marc Hornbeek has 21 posts and counting. See all posts by Marc Hornbeek