The GitOps approach: best practices for reliable and efficient cloud development processes

‍

At Blackbird, we believe it doesn’t make sense to repeat the same mistakes over and over again. That’s why we want to share our experience and knowledge with developers – to help them learn from our experiences, and get the best results from their hard work. Cloud environments can get complex fast; GitOps can help you with change management for your infrastructure, and help you build a more reliable infrastructure that’s automated and easier to operate and maintain. 

‍

To get the full benefits GitOps can bring, we recommend following some best practices. By following these, you can avoid many of the common problems that people experience with this approach, while also getting the most positive effects from it.

‍

The basics: What is GitOps?

‍

In case you’re new to GitOps (or want a quick refresher), here’s a brief overview: GitOps is a methodology as much as a specific toolkit. The key takeaway is that GitOps allows teams within an organization to manage and deploy changes to infrastructure using Git as the single source of truth.

‍

And what is Git? Git is an open-source, distributed version control system (VCS). It’s incredibly popular, and is used by more than 90% of Fortune 100 companies and over 94 000 000 developers. This VCS was created by the same person who came up with the Linux operating system - so it’s pretty good. 

‍

Data stored in Git is held in repositories, or ‘repos’. We can think of your Git repository as a sort of vault, with all your project information stored in it; files, and all the changes you make to them.

‍

Essentially, GitOps creates a better DevOps process that is oriented around developers using Git as your Infrastructure as Code (IaC), to accelerate delivery and increase reliability.

‍

How flexible is GitOps?

‍

GitOps is a flexible methodology that lets you use a variety of setups, platforms, and tools in combination – but with the Git repository always as the reference for configuration.

‍

The majority of developers use push based CI/CD pipelines like GitHub actions and GitLab pipelines with GitOps, and these two systems work very well together. Perfectly, in fact.

‍

However, it’s also possible to use it with pull based systems, such as Flux Terraform controller and/or ArgoCD, and with Flux Terraform controller , you can keep using your Terraform resources in the GitOps way. This allows you to unify Terraform resources and k8s workloads, using the Git repository as your single, ultimate source of truth.

‍

Why use GitOps?

‍

Ultimately, developers use GitOps because it makes their job easier. It enables the automation of infrastructure, wrapped up with compliant processes, version control, collaboration, and Continuous Integration/Continuous Deployment (CI/CD).

‍

As a result, the common tasks of committing new updates, merging, branching, and comparing previous versions are all made easier and are optimized for performance.

‍

It’s important to realize that GitOps is more of a ‘methodology’ or ‘approach’ than a specific set of tools – because when you don’t follow the system things can start to go wrong, no matter what tools you’re using.

‍

However, when it’s used properly, you gain a single platform for version control that also enables developers to work on updates or fixes transparently.

‍

Here are the biggest benefits of GitOps:

‍

• Developers can’t make unilateral changes, due to the in-built approval process.
• If a problem emerges, you just roll back to the previous version while you work on it.
• The development process becomes more reliable, and you have total transparency over changes with a single source of truth.

‍

What are the downsides to GitOps?

‍

Some people find GitOps to be limiting. But that’s kind of the point. While developers may feel that their territory is being taken away, it results in far fewer problems down the road.

‍

Mostly, the downsides to GitOps come from not using it properly, or trying to find ways ‘around it’. However, there are a couple of areas where GitOps currently falls short of ideal.

‍

Potential challenges using GitOps:

‍

Sensitive information

Sensitive data like passwords or keys aren’t the perfect combination with GitOps. You can secure objects in your Git repository using the SHA1 hashing algorithm, and this helps make them more secure. However, it’s also a process that’s open to accidental misuse. Git will store every change made, and these will be carried with every branch or future iteration of the repository. If secret information is not properly secured at any point, then it’s a potential security hole for the future.

Low observability

While Git tracks every change you make, there’s a lack of observability about the real state of the system, and very little overview. This limits scalability, and becomes more of an issue as projects grow in scope, complexity, or size.

Excessive repositories

As projects evolve, the number of repositories can easily grow (this is also linked to low-observability, above). While creating more repos often makes things easier in the short term, it leads to a growing number of repositories that need to be managed and kept in sync.

For the most part, people experience problems with GitOps when they’re using it the wrong way. Unfortunately there are many wrong ways to use it, so we advise sticking to a tried and tested system until you really get to grips with it (and see the benefits of using it the right way).

‍

The 3 pillars that support GitOps

‍

Together, these 3 pillars make up the core of the GitOps approach:

‍

1. Infrastructure as Code (IaC) – Information about infrastructure and configuration are stored as code in your Git repository. This acts as a single source of truth that tracks changes over time, keeping a record of all previous states.
2. Pull Requests and Merge Requests – These are the mechanism for triggering change, for instance when updates occur. Pull Requests focus teams on making changes with formal approvals, and becomes an audit log for future reference. GitOps automatically uses a merge request to deploy and verify infrastructure changes.
3. CI/CD – As with DevOps, Continuous Integration and Continuous Deployment (CI/CD) are an essential ingredient. GitOps automates this process, so changes are deployed continuously. Any differences or configuration drift is overwritten by GitOps so it matches your Git repository.

‍

Best practices for implementing GitOps

1. One team at a time – too many different teams working on the same Git repos can cause problems.
2. Recording – everyone on the team needs to follow the required processes for recording things like merge requests and issues.
3. Stick with it – it’s important to discourage developers from trying to ‘work around’ proper practices.
4. A minimum of repos – try to use a ‘minimum’ number of GitOps repositories, as this limits the time needed for provisioning and on management. One per cluster is usually enough.
5. Use a secure vault – try to keep secret data like private keys and other secure values in a centralized secure vault. You can also encrypt them in Git, but they’ll always be there in your Git history, and this makes it hard to keep track over time.
6. Security first – keep your Git repositories secure, using rigorous authentication and encryption wherever needed. Use backup and recovery tools, and ensure processes are compliant by design.
7. Use a harmonious toolkit - there are numerous options for different elements in your cloud infrastructure, and GitOps helps you manage these with transparency. However, some work better together than others. Kubernetes is a natural choice for GitOps, for example.
8. Manage the change - it can be a big shift for developers to start using the GitOps approach. This methodology can seem like a restriction is being placed on them, so make sure they understand how it will benefit them with more productive working and faster results with fewer problems down the road. Spend time on training and getting used to it.

Is GitOps for you?

‍

Using proven best practices can help you avoid huge problems much later in your projects. GitOps is therefore the ideal approach for managing your cloud development, because it includes the most important best practices as built-in features.

‍

It’s a highly portable methodology that enables your teams to work flexibly, with clear pathways for all the major processes. Changes and updates can be made with confidence, using a reliable approval and deployment process. And, best of all, you have zero downtime while you make fixes, updates, or other changes.

‍

Implementing GitOps with your team might seem like a challenge, but the benefits far outweigh the effort involved.

‍

If you want some help, we’re here for you. Get in touch, and let’s talk about how we can make your project fly!

‍

‍