How to Design a Cloud Infrastructure with Google Cloud Platform

Cloud computing has been around for more than two decades, and despite the technical efficiencies and cost-related benefits it delivers to DevOps teams, a large portion of the agile community continues to create and run their products without this technology.

As Dell reports, 29% of U.S. businesses keep all of their vital data and systems up and running on in-house server infrastructures. However, 94% of companies that are already using cloud servers say that it is the cloud infrastructure that makes it easier to meet DevOps challenges like cybersecurity, big data processing, mobility, and sustainability. Those businesses enjoy up to 62% faster revenue growth than their competitors.

There is no doubt that cloud infrastructures will remain a dominating trend to support virtual services and resources in custom product development, DevOps, and continuous delivery. Google cloud platform (GCP) has been recognized as a leading cloud computing solution for DevOps and product teams to build and manage infrastructure in the cloud, using virtual machines and services. Google offers one of the largest networks in the world to deliver fast, consistent, and scalable performance.

Here below we talk about the basics of deploying and managing infrastructure in the cloud using Google cloud platform.

What cloud services and tools to use

When you as a DevOps engineer need to create and manage your product in the cloud, you should find answers to at least these questions:

• What cloud platform is more useful for your product?
• What cloud  services to use for the infrastructure?
• What tools to use for continuous integration (CI)?
• What cloud technologies are more useful?
• How to manage your infrastructure in the cloud?
• How to automate processes?

And a lot of other questions are likely to come.

At Intersog, we want to help you with cloud development, DevOps and continuous delivery. Here are the tools and technologies we use:

• Google Cloud Platform
• Kubernetes Engine (GKE), Container Registry, Compute Engine as the main Google cloud computing services and components
• Jenkins server for continuous integration and automation
• Docker, Helm, Kubernetes, microservices as the main cloud computing technologies
• Terraform open-source software to manage infrastructure
• Puppet for automation processes

We break down the whole process into two phases. The first phase is to set up the infrastructure, and the second one is to build and deploy the product. Let’s see each in detail.

1. Creating infrastructure on Google cloud platform

To get this done, you will need Terraform, Puppet, Helm, Kubernetes, and Google cloud services. As the block-scheme demonstrates below, the setup process is straightforward.

Hire talented developers from LATAM, Canada, and Europe

Let’s get in detail. Here are the eight essential steps to set up infrastructure on Google cloud platform:

1. Create an account with Google Cloud Platform
2. Prepare the environment for your future infrastructure using "Organization" and "Projects" in your GCP account
3. Create your infrastructure - use Kubernetes engine service, for example.
4. Run Terraform and install Kubernetes clusters for the development, demo, production, and admin environments:

resource "google_container_cluster" "primary" {
  name     = "development"
  location = "us-central1"
  remove_default_node_pool = true
  initial_node_count       = 1

  master_auth {
    username = ""
    password = ""

    client_certificate_config {
      issue_client_certificate = false
    }
  }
}

5. When you have clusters ready and you can start installing additional software, you need to set up the admin cluster. Here’s a code example with Puppet and Helm:

# Global variables
export nodes="node1,node2,nodeN"
export repo_path="/path/to/infrastructure"
export project="admin"

####
# PostgreSQL installation
# Mount additional volume
bolt plan run pear_ai::mount \  
   --nodes=${nodes} 
   --u centos --run-as root \  
   --private-key ${repo_path}/terraform/${project}/id_rsa \  
   --modulepath ${repo_path}/puppet/modules \  
   --no-host-key-check mount_path='/var/lib/pgsql'

# Install PostgreSQL
bolt plan run pear_ai::postgres_install \  
   --nodes=${nodes} --u centos --run-as root \  
   --private-key ${repo_path}/terraform/${project}/id_rsa \  
   --modulepath ${repo_path}/puppet/modules \  
   --no-host-key-check

6. Next, install these essentials on your admin cluster:
   • VPN
   • FreeIpa
   • Jenkins
   • PostgreSQL
   • Docker
7. Install the following on your on product clusters:
   • PostgreSQL
   • Elasticsearch
   • Docker
   • Cassandra
8. Configure these pieces of software for further usage in the cloud as you need.

When you have taken all these steps to get your cloud infrastructure ready, next you proceed to the product development phase. Make sure, you create users and grant permissions for additional software.

2. Building and deploying your product

Your product development starts with Jira, an issue tracking system, where you have a story point and several related tasks. With this system, you can automate the process of building and deploying your product as much as possible. Take the following steps to get this done:

1. In Jira, create a flow for the development process. Your flow will count 8-10 transitions, for example: "In progress", "On hold", "Backlog", "On dev-testing", "On demo-testing", "On prod-testing", and more. You integrate all the transitions with GitHub and Jenkins.
2. Configure GitHub. You create the following default branches:
   1. develop - the main branch for development
   2. demo - second branch for pre-release
   3. prod - main branch for product

For each of these branches you need to do some configuration including: create webhooks to Jenkins and Jira, grant permissions for admin and development groups, create a flow for "Pull Request", and check CI. Each developer needs to create a new branch with the same name as in Jira.

3. Jenkins is the main tool for CI. In Jenkins, we configure several main jobs (build, deploy, release, etc). Each job is created like a pipeline in Groovy.

You installed and configured plugins for working with GitHub, GCP, Kubernetes, Jira, Slack and etc. As mentioned above, Jenkins needs to be installed into the Kubernetes cluster, and for Slaves you use Jenkins Kubernetes agent with running on pod into the cluster.

When you have a Ticket branch, we use "Pull Request" and webhooks for building your applications. Each application should work with Docker. You start building the application using Jenkins server, and then you send a request to the Jenkins plugin for creating a slave pod.

4. When the Jenkins slave is ready, begin the test of the "Pull Request" process. This process consists of several steps:
   1. Clone the branch
   2. Check that the branch has a correct name, author and "Pull Request" title
   3. Start the process with building a Docker image (if something goes wrong, you get a Slack notification to the CI panel)
   4. Test the Docker image after a successful build

5. After a successful test of "Pull Request", you can merge the branch to the master. This procedure is similar to the previous item; however, after a successful build the image needs to be pushed to the Kubernetes registry with a special tag.
6. Use Jenkins job and Google Cloud Platform API to deploy (or upgrade) your application to the Develop cluster.
7. Run automation tests for checking application health status
8. Merge the develop branch with the demo cluster
9. Move the Jira ticket to the next step - "Tested on dev"
10. Update your application in the demo environment using Github webhooks and Jenkins job.
11. Note that you don't update the application on the production server automatically (Senior DevOps specialist or Team lead have permissions for that.

As we mentioned above, Jenkins server works with the admin cluster. You can manage users and their privileges using FreeIpa via a secured VPN tunnel. Nobody can access FreeIpa from outside the network; however, Jenkins server is still available because you use Github webhooks, Jira CI, and Slack notifications.

In Jenkins server, you use PostgreSQL to store essential data (ticket numbers, service names, QA reporting, and more). All Github jobs and Jenkins pipelines are saved on Groovy.

podTemplate(containers: [    
   containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'), 
   containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')  
]) {
    node(build) {        
stage('Get a Maven project') {            
git 'https://github.com/jenkinsci/kubernetes-plugin.git'            
container('maven') {                
stage('Build a Maven project') {                    
sh 'mvn -B clean install'         
       }         
   }        
}
        stage('Get a Golang project') {      
git URL:  'https://github.com/hashicorp/terraform.git'            
container('golang') {                
stage('Build a Go project') {                    
sh """                    
mkdir -p /go/src/github.com/hashicorp                    
ln -s `pwd` /go/src/github.com/hashicorp/terraform                    
cd /go/src/github.com/hashicorp/terraform && make core-dev                    
"""       
         }       
     }      
  } 
}