Moving from YAML defined Service Accounts to Terraform module
This page is intended for users of the Cloud Platform who still have Kubernetes Service Accounts defined in
yaml, and how to remove these in favour of Terraform module based Service Accounts.
Background
As of Kubernetes version 1.24 (Cloud Platform cluster upgrade completed July 2023), Kubernetes ServiceAccount resources no longer automatically create an associated Secret for your service account’s token.
In light of this change, the team made the decision to deprecate the use of YAML defined Service Accounts in favour of the cloud-platform-terraform-serviceaccount module, which creates the required token Secret for you. You can refer to the original Slack notification for this change here.
At the time of writing, over 300 YAML defined Service Account resources still exist in the cloud-platform-environments repository, and whilst they will continue to work if preserved, any changes (intentional or otherwise) will result in the Service Account no longer working as expected.
This guide will help you switch your Service Account(s) to use the Terraform module, and cleanly remove all legacy YAML defined Service Accounts and their associated resources.
Moving to the Terraform module
We recommend that you create new Service Accounts via the Terraform module prior to removing the YAML defined Service Account. This will ensure that your application / pipelines continue working until the new credentials are generated and updated where required.
Creating a new Service Account
Follow the steps detailed here to create a Terraform module based Service Account.
Note the module defines default values for the name of your new Service Account, and the associated Role and Rolebinding resources. If you intend to change these values, it is recommended you first capture the values of your old YAML defined resources, to ensure there is no clash between the old and new resources at deployment time.
Service Account best practice
At Cloud Platform we encourage users to create and manage their Service Accounts (sa) and associated tokens via our terraform module. The module handles role, role binding, token creation and token rotation. We have not completely blocked creation of sa via kubectl or by helm but would encourage users to create the service account via the module and then pass the created sa to the relevant places eg. helm chart via values.yaml.
here is an example from a redis helm chart:
serviceAccount:
## @param master.serviceAccount.create Specifies whether a ServiceAccount should be created
##
create: false
## @param master.serviceAccount.name The name of the ServiceAccount to use.
## If not set and create is true, a name is generated using the common.names.fullname template
##
name: "$YOUR_PREVIOUSLY_CREATED_SERVICE_ACCOUNT_NAME_HERE"
## @param master.serviceAccount.automountServiceAccountToken Whether to auto mount the service account token
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server
##
automountServiceAccountToken: false
## @param master.serviceAccount.annotations Additional custom annotations for the ServiceAccount
##
annotations: {}
Obtain the new Service Account credentials
If you are using the Service Account token in an external application, such as CircleCI, you will need to obtain the new token and also ca.crt content for your Service Account, and update the application to use the new token.
You can obtain the ca.crt and token using the following commands:
kubectl -n <your-namespace> get secret <name-of-serviceaccount-secret> -o json | jq -r '.data."ca.crt"'
Note - CircleCI does not accept the
ca.crt base64decoded value as an environment variable, so run the above command to get the encodedca.crtvalue and handle the decode within your pipeline logic.
Use the cloud-platform decode-secret CLI tool to view and Base64-decode the token:
cloud-platform decode-secret -n <your-namespace> -s <name-of-serviceaccount-secret> | jq -r '.data."token"'
Now you have these values, you can update your application / pipeline to use the new credentials.
Removing the YAML defined Service Account
Find the names of your Service Account and associated resources
Locate the name of your YAML defined Service Account, and it’s associated Role and Rolebinding resources. You will find these in your Service Account YAML file, under the metadata.namefields for each resource.
apiVersion: v1
kind: ServiceAccount
metadata:
name: circleci
namespace: my-dev-namespace
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: circleci
namespace: my-dev-namespace
rules:
- apiGroups:
...
...
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: circleci
namespace: my-dev-namespace
subjects:
- kind: ServiceAccount
name: circleci
namespace: my-dev-namespace
roleRef:
kind: Role
name: circleci
apiGroup: rbac.authorization.k8s.io
Locate the name of the associated Secret containing the Service Account token, found under the secrets.name field of the Service Account resource.
kubectl get serviceaccount circleci -oyaml -n my-dev-namespace
Will yield similar to:
apiVersion: v1
kind: ServiceAccount
metadata:
name: circleci
namespace: my-dev-namespace
secrets:
- name: circleci-token-bwbcz
Delete the Service Account and associated resources
Delete the Service Account, and associated Role, Rolebinding and secret resources using the kubectl delete command.
kubectl delete serviceaccount circleci -n my-dev-namespace
kubectl delete role circleci -n my-dev-namespace
kubectl delete rolebinding circleci -n my-dev-namespace
kubectl delete secret circleci-token-bwbcz -n my-dev-namespace
Finally, delete the YAML file containing the old Service Account definition within your namespace folder in the cloud-platform-environments repository, raising a PR to merge the changes.