Continuous Deployment of an application using CircleCI and Helm

Introduction

This document covers how to continuously deploy your application to the Cloud Platform using CircleCI.

Objective

By the end of the tutorial, you will have:

Created a service account for CircleCI in your namespace.
Generated a CircleCI configuration file in your application repository.
Created an automated CircleCI pipeline, that builds, pushes and deploys your application to the Cloud Platform using a Helm chart.

Requirements

It is assumed you have the following:

You have created an environment for your application (also known as a namespace).
You have an application repository with a Helm chart.
You have created an ECR repository.

Creating a service account for CircleCI

As part of the CircleCI deployment pipeline, CircleCI will need to authenticate with the Cloud Platform cluster. In order to do so, Kubernetes uses service accounts. Service accounts provide an identity for processes that run in a cluster allowing the process to access the API server.

To create a service account:

Download the cloud-platform-cli tool
Run the command cloud-platform environment serviceaccount create in your namespace as outlined in the instructions
Add a serviceaccount_name variable called circleci (This doesn’t have to be circleci, it’s just easier for reference)
Ensure github_owner and github_token variables are defined in variables.tf. This is added by default if you created the environment using the cloud-platform-cli tool
Ensure github provider block is added to versions.tf and main.tf. This is added by default if you created the environment using the cloud-platform-cli tool
Raise a PR for your service account creation. Once approved you should then see your new CircleCI service account in your namespace.

Note - A good example of a service account created in the Cloud Platform can be found in the cloud-platform-reference-app namespace.

Once your service account is created you should be able to see it by running the following commands:

  $ kubectl get serviceaccounts --namespace <YOUR-NAMESPACE>
  NAME       SECRETS   AGE
  circleci   1         3h

  $ kubectl get serviceaccounts/circleci --namespace <YOUR-NAMESPACE> -o yaml
  apiVersion: v1
  kind: ServiceAccount
  ...
  secrets:
  - name: circleci-token-prlkp

To access the credentials of the service account, a ca.crt and a token are generated and stored in a Kubernetes secret in your namespace. This will be used to authenticate to your namespace and deploy your application via CircleCI.

Retrieving the service account credentials

Once the service account is created, you can grab the ca.crt and token using the following commands:

kubectl -n <YOUR-NAMESPACE> get secrets <NAME-OF-CIRCLECI-SERVICE-ACCOUNT-SECRET> -o json | jq -r '.data."ca.crt"'

Note - CircleCI does not seem to accept the “ca.crt” @base64 decoded value as an environment variable, so run the above command to get the encoded “ca.crt” value and do the decode later at runtime as shown here.

Use the cloud-platform decode-secret CLI tool to view and Base64-decode the token:

cloud-platform decode-secret -n <YOUR-NAMESPACE> -s <NAME-OF-CIRCLECI-SERVICE-ACCOUNT-SECRET> | jq -r '.data."token"'

Both the ca.crt and token will be added as environment variables in the CircleCI console later in this document.

Setting up CircleCI

MoJ Digital have an account with CircleCI, to login you must be a member of the ministryofjustice GitHub organisation. Use the GitHub integration and enter your credentials if required.

To setup your repository, simply go to “Projects”, enter your repository name and select “Set Up Project”. You’ll be presented with a default .circleci/config.yml file, copy the contents of this file and create it in the root of your repository. An example of this can be found here

Add variables to CircleCI

There are two ways to add variables to your CircleCI pipeline: - Contexts: Shared variables across an organisation (ministryofjustice). - Environment variables: Unique variables defined in your project.

In this document, we will use environment variables to deploy to a single environment/namespace. However, if you wish to deploy to multiple namespaces (dev/staging/production) it’s recommended you explore contexts further.

To deploy to a single namespace, select the “Project Settings” and then “Environment Variables”. The following should be set:

- AWS_DEFAULT_REGION # This should be set to `eu-west-2`.
- AWS_ACCESS_KEY_ID # If you're deploying to ECR you can find this value in a namespace secret: (https://user-guide.cloud-platform.service.justice.gov.uk/documentation/getting-started/ecr-setup.html#accessing-the-credentials).
- AWS_SECRET_ACCESS_KEY # As above, this should be stored in a namespace secret and is used for ECR.

- ECR_ENDPOINT # This should be the ECR name your intend to push your images to e.g. `738437979232.dkr.ecr.eu-west-2.amazonaws.com/webops/cloud-platform-reference-app-ecr`

- K8S_CLUSTER_NAME # The full name of the "live" cluster `DF366E49809688A3B16EEC29707D8C09.gr7.eu-west-2.eks.amazonaws.com`
- K8S_NAMESPACE # The name of the namespace/environment (see https://user-guide.cloud-platform.service.justice.gov.uk/documentation/getting-started/env-create.html#creating-a-cloud-platform-environment)
- K8S_CLUSTER_CERT # The CA Certificate for the cluster, can be acquired from the `Secret` that is generated for the `serviceaccount` (as stated above)
- K8S_TOKEN # The access token generated for the `serviceaccount`.

Configuring CircleCI

This is a really simple example of adding a build and deploy section to your configuration. If you’d rather see this in action, please checkout the cloud-platform-reference-app

setup-kube-auth is deprecated

Early on in our examples we used a custom image just for CircleCI and a wrapper script named setup-kube-auth that handled the kubeconfig generation; both are obsolete so if you have a reference to either image: ${ECR_ENDPOINT}/cloud-platform/tools:circleci-internal or setup-kube-auth in your steps: just remove and configure as described below.

Building and pushing image to ECR

Below is an example how you can build and push an image to ECR:

version: 2

workflows:
jobs:
  build:
    docker:
    - image: ministryofjustice/cloud-platform-tools
    steps:
    - checkout
    - setup_remote_docker:
        docker_layer_caching: true
    - run:
        name: Build application Docker image
        command: |
          docker build -t app .
    - deploy:
        name: Push application Docker image
        command: |
          login="$(AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID_DEMO} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY_DEMO} aws ecr get-login --no-include-email)"
          ${login}
          docker tag app "${ECR_ENDPOINT}:${CIRCLE_SHA1}"
          docker push "${ECR_ENDPOINT}:${CIRCLE_SHA1}"
          if [ "${CIRCLE_BRANCH}" == "main" ]; then
            docker tag app "${ECR_ENDPOINT}:latest"
            docker push "${ECR_ENDPOINT}:latest"
          fi

Note - Update the aws ecr login command above if you are using AWS version 2.0. Click here for reference.

Deploy to Kubernetes

Below is example of how you can first authenticate and then deploy to Kubernetes:

      - run:
          name: Authenticate with cluster
          command: |
            echo -n ${K8S_CLUSTER_CERT} | base64 -d > ./ca.crt
            kubectl config set-cluster ${K8S_CLUSTER_NAME} --certificate-authority=./ca.crt --server=https://${K8S_CLUSTER_NAME}
            kubectl config set-credentials circleci --token=${K8S_TOKEN}
            kubectl config set-context ${K8S_CLUSTER_NAME} --cluster=${K8S_CLUSTER_NAME} --user=circleci --namespace=${K8S_NAMESPACE}
            kubectl config use-context ${K8S_CLUSTER_NAME}
            kubectl --namespace=${K8S_NAMESPACE} get pods

    - run:
        name: Install demo helm chart
        command: |
          if [ "${CIRCLE_BRANCH}" == "main" ]; then
            helm upgrade demo helm_deploy --install --namespace=${K8S_NAMESPACE}
          fi
          kubectl --namespace=${K8S_NAMESPACE} get pods

Triggering your pipeline

On a commit to any branch the pipeline will be triggered, which you can view via the CircleCI console.

Where to go from here

As previously mentioned, this is a simple example of how to deploy your Helm packaged application to a single namespace. If you want to deploy to dev, staging or production you should read more about CircleCI contexts.

Adding tests should also be explored; the following video shows detail on how to build, test and deploy your application: https://circleci.com/docs/2.0/test/

This page was last reviewed on 1 March 2023. It needs to be reviewed again on 1 June 2023 .

This page was set to be reviewed before 1 June 2023. This might mean the content is out of date.