CircleCI Security Incident - Secrets Rotation Guidance
Overview
Following the announcement of the CircleCI Security Alert issued 04/01/2023, the Cloud Platform team has been focusing on working to ensure that all secrets impacted by this event have been rotated.
Is my Cloud Platform namespace affected?
As part of our investigation into this incident, we have identified a number of namespaces which require AWS secrets to be rotated. Prior to
commencing the rotation work, we will be sending out comms in the cloud-platform-update and ask-cloud-platform
Slack channels. These comms will contain a list of all affected namespaces.
Impacted Services
Cloud Platform Terraform Modules
If your Cloud Platform workload utilises any of the following Cloud Platform Terraform Modules, their AWS Access Key and associated Secret Access Keys will be rotated.
- RDS
- RDS Aurora
- DynamoDB
- S3
- SNS Topic
- SQS
- DMS
ECR Module AWS secrets have already been rotated in a previous activity, so will not be covered during this process.
Non Module defined AWS Secrets
In addition to the above modules, if you have any iam_user Terraform resources defined in your namespace, their associated access keys will also be rotated. For example,
you may have similar to the following in your namespace/resources folder:
resource "aws_iam_user" "user" {
name = "s3-bucket-user-${random_id.id.hex}"
path = "/path/to/iamuser"
}
resource "aws_iam_access_key" "user" {
user = aws_iam_user.user.name
}
resource "kubernetes_secret" "aws_iam_secret" {
metadata {
name = "s3-bucket"
namespace = var.namespace
}
data = {
destination_bucket = "s3://${var.namespace}"
user_arn = aws_iam_user.user.arn
access_key_id = aws_iam_access_key.user.id
secret_access_key = aws_iam_access_key.user.secret
}
}
The rotation process will recreate the aws_iam_access_key resource and update the access_key_id and secret_access_key fields in the kubernetes_secret
resource.
Rotation Process
NOTE: It is the responsibility of application owners to identify and carry out any remediation activites to update these values wherever they are used for their service. It is for this reason that the team has elected to follow a ‘user merge’ managed approach for triggering the rotation of these secrets.
The Cloud Platform team will raise a rotation PR per namespace in the cloud-platform-environments
repository, which will be pre-approved for merging by the namespace owners.
Once this merge has been confirmed, the rotation process will be triggered, so it is important that any pre-requesite checks are carried out before proceeding.
Steps to follow
IMPORTANT: As with any service impacting change, we strongly encourage you to carry out thorough testing of this rotation process using PRs for your non-production environments before proceeding with production namespaces. The Cloud Platform team has conducted testing of secrets rotation against all modules and in many scenarios, services relying on any of the impacted resources will immediately fail to work as expected. You must ensure all pods in your namespace are recycled following completion of rotation PR.
It is advised that before merging your secret rotation PR that you check the values of all of your namespace’s AWS secrets. This will assist in verifying the rotation has completed once applied via our Concourse pipeline. This can be done for each of your Kubernetes secrets containing AWS secrets as follows:
To get a list of all Kubernetes secrets in your namespace:
$ kubectl -n <YOUR-NAMESPACE> get secrets
To retrieve the values stored in a given secret:
$ kubectl -n <YOUR-NAMESPACE> get secret <SECRET-NAME> -o yaml
You can use the cloud-platform decode-secret CLI command to automatically base64 decode the values:
$ cloud-platform decode-secret -n <YOUR-NAMESPACE> -s <SECRET-NAME>
An example of the above command for obtaining RDS Module secrets output would be similar to the following:
$ cloud-platform decode-secret -n my-namespace -s rds-postgresql-instance-output
{
"apiVersion": "v1",
"data": {
"access_key_id": "xxxxxxx",
"database_name": "db1234567",
"database_password": "xxxxxxx",
"database_username": "dbuser",
"rds_instance_address": "cloud-platform-xxxxxxx.xxxxxxx.eu-west-2.rds.amazonaws.com",
"rds_instance_endpoint": "cloud-platform-xxxxxxx.xxxxxxx.eu-west-2.rds.amazonaws.com:5432",
"secret_access_key": "xxxxxxx",
"url": "postgres://xxxxxxx:xxxxxxx@cloud-platform-xxxxxxx.xxxxxxx.eu-west-2.rds.amazonaws.com:5432/db1234567"
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2023-01-11T13:37:38Z",
"name": "rds-postgresql-instance-output",
"namespace": "your-namespace",
"resourceVersion": "11111111",
"uid": "xxxxxxx-xxxxxxx-xxxxxxx"
},
"type": "Opaque"
}
The values being updated here are access_key_id and secret_access_key. You will need to carry out similar checks for each
of your Kubernetes secrets.
NOTE: There are teams which share AWS secrets to other relying namespaces, for example ECR & RDS credentials. This will need to be taken into consideration as applications running in these other namespaces will also be impacted. You will need to update these values yourself.
Triggering Rotation of Secrets
At this stage you should be ready to merge your rotation PR. Once you confirm your rotation PR merge, you can expect your namespace secrets to be rotated shortly afterwards.
Pre-approved rotation PRs will be named following the convention:
Rotate Secrets in Namespace: <NAMESPACE>
To locate your PR merge, head over to the cloud-platform-environments repo “Pull requests” view here,
and use the search bar to locate your namespace rotation PR, as demonstrated in the image below:
Tracking Secrets Rotation Status
The secrets rotation process will update existing Kubernetes secrets in namespaces rather than replacing them entirely; therefore it
is not possible to rely on checking the age of your secrets as this will not change.
There are two checks that you can follow to confirm rotation has completed:
Monitoring Concourse Pipeline
After you confirm your rotation PR merge, you will receive a message from cloud-platform-concourse-bot containing a URL to the Concourse build
applying your secrets rotation in the PR Converation comments panel in GitHub, which you can use to track progress.
This will look like the following:
If you fail to receive a link in GitHub following your merge, you can head over to the Concourse apply-namespace-changes-live
pipeline and search for your build. Each build run in this pipeline outputs the following information:
Found PR: [PR number]
Applying Namespace: [namespace being applied]
This information can be found by expanding the task: apply-namespace-changes pane in the build view.
Monitoring Kubernetes Secrets
Conducting a “before and after” check of your AWS secrets values using the above kubectl or cloud-platform decode-secret CLI
commands. Once you have observed these have been updated you are ready to proceed with updating your environment with the new
values.
Post rotation activities
We highly recommend that users work through the rotation process in non-production environments in order to assess the impact this may have on workloads, and to implement strategies to restore normal service. We have to assume that every application utilises secrets in unique ways, as attempting to cover every scenario in this guide is not possible.
In testing, Cloud Platform have found that where deployments are mounting Kubernetes secrets as application environment
variables, a recycling of all namespace pods is required to restore service.
NOTE: Please take extra care to ensure that you are targeting the correct cluster context and namespace when running the following, or similar, commands:
$ kubectl delete --all pods -n <YOUR-NAMESPACE>
You can then check for pod statuses:
$ kubectl get pods -n <YOUR-NAMESPACE>
Support
If you require any further support during and after the rotation process, please contact the Cloud Platform team via our
ask-cloud-platform Slack channel.