Why You Should Run Your Pipelines in Kubernetes

So, you saw the title of this article and I can hear you thinking: “No, thank you, I’m running fine in <insert Git server platform here>,” or “Then why did I build this expensive machine to run them on?” or even “Wait?! That’s an option?!”

Hi, my name is Fabian Met, and I’m writing this while drinking some coffee and having a laugh. You see, I’m having flashbacks to the good, the bad and the ugly of the story I’m about to tell, even though I do look back at it fondly.

If I write this correctly, I hope you learn something and, if nothing else, have a good time reading this. I hope you have a small chuckle or snort of recognition.

So why should you run your pipelines in Kubernetes or at least consider doing so? That’s a good question. This insight came while working for a customer a few years back and has since been bolstered by additional arguments.

Strap in and get ready because there will be many opinions, but in the end, I’m hoping you’ve learned something or even gotten to the point where you want to try out some possible solutions.

The History

Let’s talk about that customer I mentioned. The customer hired me as part of a team to build their Kubernetes implementation. The implementation’s ins and outs don’t matter here, but we had to maintain code that automated all our work. That’s what this post is about.

They had the following situation:

They ran an on-premises GitLab server. They also ran an on-premises Bitbucket server. And to encompass the whole spectrum of possibilities, they even ran an on-premises Jenkins server.

Now, if you’re familiar with these tools, you know they can all be used to automate your work. (If you were unfamiliar, now you know they can all be used to automate your work ;-).)

It was quite a situation, three different flavors of automation backed by two flavors of Git servers.

Side note: Before you start feeling smug that your situation is not as bad and your company has everything sorted out, remember that this is just what got me to my opinion. There are extra arguments further on!

The Start

I started my work with this customer using Bitbucket with Jenkins while maintaining legacy repos hosted in GitLab with GitLab Pipelines.

I’m a simple guy. I like simple things. And maintaining two types of automation with very different approaches to automation is a complex thing.

The teams were running into certain limitations with their setups, and we got to a point where we were going to reevaluate where we hosted the Git repositories and what tool we would use for automation.

A solution was reached internally: Let’s move to GitHub! That’s right. We moved to the cloud! Where everything would be better, the sun was always shining and the grass was permanently greener!

We just had to rewrite our pipelines to GitHub Actions!

How hard could it be, right? I had no prior experience with GitHub actions, but it’s a pipeline tool, so it shouldn’t be difficult.

Famous last words.

Moving to GitHub

Now, I want to make one thing clear. This post is not to bash GitHub! It was my own misunderstanding of how GitHub Actions worked that caused the adoption to proceed slower than expected.

However, we finally got it working! I learned a thing or two and, together with the team, we rewrote the pipelines to GitHub Actions.

So, now, the sun was always shining! Everything was better! And the grass was so much greener there!

Things were improved by using GitHub; not because GitHub is a better tool but because the company had finally consolidated the tooling. This way, we never had to ask ourselves again if a pipeline would go into GitLab or Jenkins.

But Wait…

What’s that in the distance? Do I see thunderclouds?!

The security officer for the company opposed the decision to use GitHub. I will not go into why, but that’s what happened.

We had to move everything back to the on-premises solutions.

Gone were the days of greener pastures and sunny weather.

But what were we going to move to? We had outgrown the on-premises solutions. Git servers aside, the decision was how we would run our automation on-premises in a cloud-native way.

We had some investigation to do!

The Investigation

Before making any decision, it is essential not to get distracted by the promises and the marketing of each possible solution. Instead, distill the choice based on the actual requirements you need and then you’ll make a good decision.

Too much reading of the marketing information will make you believe each solution can move mountains with the click of a button, and a deity of your choice will come down or up from wherever they reside to congratulate you.

We looked at the Docker executor of Jenkins, GitLab and the idea of GitHub Actions. These were the solutions we had used before. It turns out they all generally work in the same way:

1. Be able to define a set of steps to automate tasks.

2. Get triggered by a code change.

3. Start a container with a specified image.

4. Copy over some files or artifacts to the container.

5. Run a command.

6. If successful, move on to the next step in the pipeline.

This didn’t make the decision much easier because, it turns out, each would be workable since they all basically do the same thing. (Well, except for GitHub because we were forced away from that.)

But, hang on, there must be tools that are really good at orchestrating containers…?

Wait, weren’t we building a Kubernetes platform for this customer?!

Kubernetes, as it turns out, is great at orchestrating containers. And thus, it was a good fit for the use case we had.

Now we needed a solution for the peripheral matters; defining a pipeline, triggering on code change, exchanging data, etc., etc.

As usual, when it comes to these things, someone else already thought of this and built something awesome to help out!

There are a few open source solutions that can help with this; here are the two we looked at: Tekton and Argo Workflows.

Now, we were already using ArgoCD, which we loved, so we went with the obvious choice: Tekton.

Don’t ask me why—both seemed very much alike, and we decided to try out Tekton first, which turned out to be such a big help that we stuck with that one.

In general, what Tekton should have done was simple: Take a pipeline definition and run it when code changes.

How hard could it be, right?

Famous last words.

Now, don’t get me wrong. Just as with GitHub actions, this was due to my lack of knowledge about how Tekton worked.

So to get started, I first booted up a local Kubernetes cluster on my laptop and installed Tekton.

I started messing around and learning more and more about how Tekton worked and how it didn’t.

After I finished my local development, we deployed it to our admin cluster, which we used to administer other clusters.

Since I had already learned and built a local test pipeline, getting started in our admin cluster was really easy. Just deployed some yaml, and we were golden.

After the team got more involved, the adoption snowballed, and before we knew it, everyone was using Tekton.

Conclusion

To conclude, using many tools that do the same thing is not cool. Using one tool that you are really good at is cool.

Using Kubernetes to run that tool is also cool if you really like running things in Kubernetes.

However, now that you’ve read the above section, I hope you see one really, really important thing that was suddenly possible: We could test our pipelines!

We could now locally develop new steps in our pipelines, test them locally and then, when we had a good grasp, deploy them to our admin clusters.

No longer were we doing a Git push just to see the pipeline fail in an interface at step three out of five. We were testing our logic on our own local Kubernetes clusters first, and since Kubernetes can run basically anywhere, there was nothing holding us back.

It was awesome; the grass was greener and the sun was always shining! It took quite a bit of trial-and-error learning and switching tools to find something that fit us best.

There is one really awesome bonus thing. Now, I can use this knowledge for any customer or situation! I’m not bound by the choice of Git server!

Pick a tool for pipelines in Kubernetes and give it a shot! Pick one and you can use it anywhere Kubernetes is available!