Skip to main content

Switch ingress to the new nginx-ingress-controller v1.2

Introduction

An ingress-controller is a specialised load balancer for Kubernetes. The Cloud Platform use the nginx-ingress-controller to accept traffic from outside the Kubernetes platform and load balance it to pods running in Namespaces.

Currently all service teams are using 1 of 2 ingress-controllers using one of the following annotations:

  • kubernetes.io/ingress.class: nginx - This is the default ingress-controller
  • kubernetes.io/ingress.class: modsec01 - This ingress-controller has the modsec WAF enabled.

These ingress-controllers are running on version v0.47.0

Unfortunately, there is a breaking change in the upgrade path of the nginx-ingress-controller to v1.0 and above. This would require all service teams to make changes to all ingress manifests at the same time as the upgrade.

To avoid this, the Cloud Platform team have released a new set of nginx-ingress-controllers (v1.2) alongside the current nginx-ingress-controllers (v0.47.0).

There are number of benefits to using the new v1.2 ingress controllers:

  • Security updates (The current controllers are no longer getting security patches. The new versions already have patches for the 3 latest CVEs)
  • Introducing support for TLS 1.3 (new feature)
  • Retiring support for TLS 1.0 and 1.1 (mitigate security risks, address Service Team ITHC report and to align to NCSC recommendations)
  • Prerequisite for Kubernetes 1.22 upgrade

Service teams using the Cloud Platform are asked to switch to the new releases of the nginx-ingress-controller(v1.2).

Pre-requisite - Ingress API Version

This is a critical change/check all users should complete before switching to the new nginx-ingress-controller(v1.2).

This change is also in preparation for the upgrade of the Cloud-Platform to Kubernetes 1.22 which has deprecated all beta versions of the Ingress API.

Please follow the Deprecated Ingress API document on how to check and if required, change to the stable Ingress API version.

Switch to the new nginx-ingress-controller

IMPORTANT: There may be downtime of up to 10 minutes while switching to the new ingress controller due to DNS propagation. Please test in non-production environments before making the change on your production applications.

Alternatively you can follow the steps later in this document to switch to the new nginx-ingress-controller(v1.2) without downtime.

There are 2 releases of the new nginx-ingress-controller(v1.2). They can be selected using the new IngressClassName field:

  • ingressClassName: default - This is the default ingress-controller
  • ingressClassName: modsec - This ingress-controller has the modsec WAF enabled.

To switch to the new nginx-ingress-controller(v1.2):

  • Remove the kubernetes.io/ingress.class annotation
  • Add the ingressClassName field under spec (using the default or modsec value)

The changes should look like the below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

The helm ingress template manifest changes will look like this:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "app.fullname" . -}}
{{- $ingressPath := .Values.ingress.path -}}
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: helloworld
  labels:
    {{- include "app.labels" . | nindent 4 }}
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  {{- range .Values.ingress.hosts }}
  - hosts:
    - {{ .host }}
    {{ if .cert_secret }}secretName: {{ .cert_secret }}{{ end }}
  {{- end }}
  rules:
  {{- range .Values.ingress.hosts }}
    - host: {{ .host }}
      http:
        paths:
          - path: {{ $ingressPath }}
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ $fullName }}
                port: 
                  name: http
  {{- end }}
{{- end }}

Switch to the new ingress controller with zero downtime

The following steps describe how to switch ingress controllers without downtime. We strongly recommend teams test these steps on their non-production environments before attempting on production.

Step 1 - Changes to your current ingress resource

You must set external-dns.alpha.kubernetes.io/aws-weight value to “0” for the current ingress. This does not mean traffic will stop, 0 is a valid value for a single ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld
  annotations:
    kubernetes.io/ingress.class: nginx
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "0"
spec:
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

The helm manifest changes will look like:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "app.fullname" . -}}
{{- $ingressPath := .Values.ingress.path -}}
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: helloworld
  labels:
    {{- include "app.labels" . | nindent 4 }}
  annotations:
    kubernetes.io/ingress.class: nginx
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "0"
spec:
  tls:
  {{- range .Values.ingress.hosts }}
  - hosts:
    - {{ .host }}
    {{ if .cert_secret }}secretName: {{ .cert_secret }}{{ end }}
  {{- end }}
  rules:
  {{- range .Values.ingress.hosts }}
    - host: {{ .host }}
      http:
        paths:
          - path: {{ $ingressPath }}
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ $fullName }}
                port: 
                  name: http
  {{- end }}
{{- end }}

Step 2 - Create a new ingress resource

Create a new Ingress with a different ingress name to the existing one with the following changes:

  1. Create a new ingress name different to the existing one:

    name: helloworld-new

  2. Update set-identifier annotation to match with the new ingress name (-green remains the same):

    external-dns.alpha.kubernetes.io/set-identifier: helloworld-new-mynamespace-green

  3. Update aws-weight annotation to “100”:

    external-dns.alpha.kubernetes.io/aws-weight: "100"

  4. Remove the kubernetes.io/ingress.class annotation.

  5. Use the new ingress controller:

    Add ingressClassName under spec, using the value of default or modsec

The new manifest after changes will look like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloworld-new
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-new-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  - hosts:
    - helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
  rules:
  - host: helloworld-app.apps.live.cloud-platform.service.justice.gov.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: helloworld
            port:
              number: 4567

To apply the changes to the ingress, as a quick alternative to running your application’s CI/CD pipeline, you can apply them directly:

kubectl -n <namespace-name> apply -f ingress-new.yaml

For the helm deployments, create a copy of your existing ingress template file and make the above changes.

The new ingress template file after changes will look like this:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "app.fullname" . -}}
{{- $ingressPath := .Values.ingress.path -}}
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: helloworld-new
  labels:
    {{- include "app.labels" . | nindent 4 }}
  annotations:
    external-dns.alpha.kubernetes.io/set-identifier: helloworld-new-mynamespace-green
    external-dns.alpha.kubernetes.io/aws-weight: "100"
spec:
  ingressClassName: default OR modsec
  tls:
  {{- range .Values.ingress.hosts }}
  - hosts:
    - {{ .host }}
    {{ if .cert_secret }}secretName: {{ .cert_secret }}{{ end }}
  {{- end }}
  rules:
  {{- range .Values.ingress.hosts }}
    - host: {{ .host }}
      http:
        paths:
          - path: {{ $ingressPath }}
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ $fullName }}
                port: 
                  name: http
  {{- end }}
{{- end }}

To apply your changes:

helm upgrade -f <values-file.yaml> <your-release> <your-chart> --namespace <your-namespace>

Verify and wait for traffic to be sent to the new ingress

After you have applied new ingress with external-dns.alpha.kubernetes.io/aws-weight value set to “100”, wait for couple of minutes for the traffic to be sent to the new ingress, you can verify traffic is being sent to the new ingress using kibana

Get the “hostname” used in your ingress for your namespace

  kubectl get ingress <ingress-name> -n <namespace-name> -o json | jq -r '.spec.rules[].host'

Using the “hostname” from the above command, make a cURL call every 5 seconds to send traffic to your service

  while sleep 5; do curl -I https://<host-name>; done

To view your ingress logs, login to kibana, select index live_kubernetes_ingress*, and do a search using "host-name" and "new-ingress-name", you will see traffic for your new ingress. An example of this kibana search

Your new ingress is successfully switched to the new ingress controller.

Step 3 - Cleanup old ingress

Once you have successfully switched to the new ingress controller, you can tidy up by deleting the old ingress and updating your CI pipeline/job with new ingress manifest that deploys to the cluster.

For the helm deployments, delete your old ingress template file. Apply the changes to delete the old ingress in your namespace.

helm upgrade -f <values-file.yaml> <your-release> <your-chart> --namespace <your-namespace>

Getting help

If you have any questions, please contact us on #ask-cloud-platform Slack channel.

This page was last reviewed on 21 July 2022. It needs to be reviewed again on 21 October 2022 .
This page was set to be reviewed before 21 October 2022. This might mean the content is out of date.