Implementing a GitOps Workflow with Terraform Cloud & GitLab

Project Outcome

• Design a Terraform GitOps workflow solution
• Create a GitLab Project to manage all Terraform modules
• Develop Terraform to properly and securely provision new GitLab Projects and Terraform Registry modules
• Automate the tagging of new module release through a GitLab Pipeline, merge request approvals, and git tagging

Architectural Diagram

Project Description

In this end-to-end project, we’re going to design and build a GitOps workflow for Terraform Cloud and GitLab. GitOps addresses the challenges of manually managing infrastructure by developing configuration files to define and manage all your infrastructure based on DevOps best practices, which include version control, collaboration, compliance, and CI/CD.

Problem

In this project, we’re going to work on a problem I was tasked to solve when I worked as a Senior Cloud DevOps Engineer. As part of our workflow, the Cloud Operations team developed and stored terraform modules in the Terraform Cloud private registry. All of this work was being done manually each time a new Terraform module needed to be created. This includes:

• Manually creating a new GitLab Project to store the terraform code. This was problematic as each GitLab project differed in settings and constantly caused issues with not complying with company standards
• Engineers would forget to tag their terraform code updates. This would cause additional overhead as they would reach out to our team for support and it would turn out that the code just needed to be tagged with the next incremental tag, so that the Terraform Cloud registry would register it.
• Performing this setup took up additional time as the engineers would have to point and click through a lot of the setup manually. So by the time they remembered how to properly set things up, there was already plenty of time wasted on a pretty remedial task

Solution

To address these issues, we decided that we needed to automate the setup as much as possible from start to finish.

• To address the GitLab Project misconfigurations, we decided that we’d use a single Terraform Workspace to provision any new Terraform modules. This would ensure each GitLab Project which hosts Terraform modules would be setup identically.
• To address the module tagging issue, we decided that we’d add a GitLab CI Pipeline which would trigger whenever there is an approved merge request which would incrementally tag the code base. This tag would automatically get updated in the Terraform Cloud private registry module.
• To address the manual work, we’d implement this solution, so that the majority of setup, would be automated, which would reduce the setup time to within minutes.

TechStack

• Terraform Cloud
• GitLab Cloud

What are we going to build?

• Design a GitOps solution from end-to end
• Create a GitLab Project to manage all Terraform modules
• Develop Terraform to properly and securely provision new GitLab Projects and Terraform Registry modules
• Automate the tagging of new module release through a GitLab Pipeline, merge request approvals, and git tagging