Terraform - Deploy GKE Kubernetes Cluster incl. Workloads (Certmanager, Nginx Ingress, Testapp)

Today we deploy a complex Kubernetes cluster on the Google Cloud Platform including workloads (NGINX Ingress, Cert Manager, Testapp) in about 10 minutes.
opensight.ch – roman hüsler

We use the Terraform tool for this. Manually you need at least several hours to configure all services. The “Infrastructure as Code” approach with Terraform requires an initial effort. You can then create such complex clusters with just a few clicks.

Inhalt

• Cluster Design
• Prerequisites
• What is Terraform
• Deployment of a GKE Cluster

Cluster Design

Specifically, it is about a GKE deployment for a cluster with 1 node and workloads:

• GKE Cluster
• Nodepool
  1 Node (n1-standard-1)
• Network
  VPC (10.0.0.0/18)
  Subnet k8s-pod-range (10.48.0.0/16)
  Subnet k8s-service-range (10.52.0.0/16)
  NAT Gateway (outbound traffic)
• Workloads
• NGINX Ingress Controller (HELM Chart)
  Creates corresponding load balancer with public IP in GCP
• Cert Manager (HELM Chart)
  Automatically issues SSL certificates for Nginx Ingress resources
  e.g. for the web service in the “testapp” module
• Test-App (nginx webserver)
  Simple example web service for testing

All source code files described can be found in the public GIT repository (link on top).

The GKE cluster that we are installing is a private cluster – the clsuter nodes do not have a public IP address and are therefore not directly accessible via the Internet. Outbound traffic is routed through the NAT gateway. Incoming traffic (HTTP, HTTPS) is routed through a Google Load Balancer to an NGINX Ingress Controller on Kubernetes. We define a VPC and some subnets for address assignment. This is a test cluster – to save costs we only create one node, but make provisions in the code to also have a zonal cluster with redundant nodes spread over several zones if required.

All right – on to implementation.

Prerequisites

Today’s blog post deals with the topic “Infrastructure as Code” – in order to follow the steps described, you should already be familiar with GIT and GIT repositories and the Terraform Basics.

What is Terraform

Terraform is an “Infrastructure as Code” tool from Hashicorp. It makes it possible to define entire infrastructures as code (declarative) to describe the desired end state of the infrastructure and then deploy the whole thing in the cloud.

After the infrastructure has been deployed, Terraform saves the “state” of the infrastructure. The state contains all information about the infrastructure that you have deployed in the cloud. For example, it is possible to shut down the infrastructure completely or to detect manual changes to the infrastructure. If you don’t define anything else, Terraform will store its state in a local file (terraform.tfstate). In enterprise environments, however, the state is usually stored in the cloud (e.g. Terraform Cloud, Amazon S3 Bucket, etc.).

Installing Terraform is quick and a manual can be found here.

Most important commands

# Initialisation of all terraform modules for the project
terraform init

# compare the state of the infrastructure with the declaration (tf files)
# plan changes to the infrastructure
terraform plan

# make the necessary changes in the infrastructure to match the declaration
terraform apply

# make the necessary changes in the infrastructure to match the declaration
# but just for module test_app
terraform apply -target module.test_app

# roll back the infrastructure
terraform destroy

# roll back the infrastructure
# but just the module test_app
terraform destroy -target module.test_app

Deployment of a GKE Cluster

Here we go. First, let’s clone the GIT repository and login to Google Cloud. For this you need the “gcloud” command / the Google Cloud CLI (instructions) on your computer.

# clone git repo
git clone https://github.com/butschi84/terraform-gke.git
cd terraform-gke

# initialize terraform module
terraform init

# login on google cloud
gcloud auth login

# edit variables.tf
- in variables.tf you can add your private IP address (whitelist-ips), so that you can access the control plane / kubernetes api server of the cluster after creation
- also you should specify the name of your gcp project
- in the file clusterissuer.yaml you can set your e-mail adress in order to receive notifications about expiring certs. cert manager will take care of cert renewal automatically though

# create infrastructure (gke cluster and workloads)
terraform apply

# info - reapply just a module
terraform apply -target module.

A video of the deployment process with terraform can be found on youtube.

Video – Terraform GKE Deployment