RDS Snapshots

RDS snapshots have multiple purposes: migrations, backups, etc. When RDS DB instances are created using the Cloud Platform RDS terraform module, an IAM user account is created for management purposes. This user can create, delete, copy, and restore RDS snapshots.

Examples of managing RDS DB snapshots using the AWS CLI, via the Cloud Platform Service Pod can be found in the README within the RDS terraform module

Warning Do not restore a production database until you have rehearsed the same restore in a non-production environment first, such as your development or staging database. Restoring destroys and recreates the database rather than updating it in place, so an incorrect snapshot, an undersized db_allocated_storage value or a mismatched engine version will take the database down, and not every mistake can be undone. Rehearse the full process end to end, including the verification step, and only proceed in production once it has succeeded in a non-production environment.

Considerations

The amount of manual snapshots per AWS account is limited, so it’s important to cleanup old snapshots
Daily snapshots are provided “out of the box”, and do not count towards the “Manual Snapshots” total
Managing snapshots is the teams’ responsibility (as are snapshot restores), so teams are responsible for cleaning up unneeded manual snapshots in order to avoid hitting our AWS account limits
Manual snapshots persist even after the source database is deleted. Automated snapshots are deleted when the source DB is deleted
Manual snapshot creation takes approximately 2-3 minutes
Before starting, check that you don’t have skip_final_snapshot set to true, unless you are really happy to proceed without that backup.
You cannot restore a snapshot taken from an older DB engine version to a newer DB engine version, or vice versa. The engine version of the snapshot must match the engine version of the target DB instance when restoring from a snapshot.

Restoring live services from an RDS DB snapshot

Note: If your existing RDS instance has deletion protection enabled, you must first disable it before performing a restore.

Important: Database read-write operations are not availiable during the database restore step. This can take some time. Carry out testing on a representative test database to see how long your restore will take.

There are two restore scenarios:

Scenario A: Restore to the SAME database (replace existing)
Scenario B: Restore to a NEW database (cross-DB restore)

Before you start (both scenarios)

Complete these checks before raising any PR. They apply to both Scenario A and Scenario B.

Check the allocated storage

Restoring fails if the target database has less storage than the database the snapshot came from. The apply ends with:

InvalidParameterCombination: Invalid storage size for engine name postgres and storage type gp3: <size>

Check the storage recorded against the snapshot you intend to use:

aws rds describe-db-snapshots \
  --db-snapshot-identifier <snapshot-name> \
  --query 'DBSnapshots[0].[DBSnapshotIdentifier,AllocatedStorage,EngineVersion]' \
  --output text \
  --region eu-west-2

Compare that AllocatedStorage figure against the db_allocated_storage value in your RDS module. If you have not set it, the module default is 20GB. db_allocated_storage must be at least the snapshot’s AllocatedStorage.

If you need to raise it, AWS requires any increase in allocated storage to be at least 10% of the current value, so a smaller change is rejected. See Increasing DB instance storage capacity.

Warning: You cannot use a snapshot restore to reduce storage. The size shown against a snapshot in the AWS console reflects the data in use rather than the volume provisioned, so it is not the figure to size from. Use AllocatedStorage as above. To shrink a database, dump the data logically and load it into a new, smaller instance, as described in Migrating an RDS instance.

Use a manual snapshot

Restore from a manual snapshot. Automated snapshots (those prefixed rds:) are tied to the lifecycle of the source instance and are deleted when it is deleted. Terraform destroys the existing instance before creating its replacement, so an automated snapshot can disappear part-way through a restore.

Important: Before you start either scenario, take a manual snapshot of the database as it stands now. Restoring deletes the current database along with its automated backups, so without one there is no way back to the state the database is in today. Use the create-db-snapshot command below.

List what already exists for the database. The SnapshotType column tells you whether each one is manual or automated:

aws rds describe-db-snapshots \
  --db-instance-identifier <rds_name> \
  --query 'DBSnapshots[*].[DBSnapshotIdentifier,SnapshotCreateTime,SnapshotType,Status]' \
  --output table \
  --region eu-west-2

You then need a manual snapshot to restore from. Depending on which point in time you need, either create a new one or copy an existing automated one.

Create a manual snapshot to capture the database as it stands now. The suggested name format is rds-name-namespace-purpose-date:

aws rds create-db-snapshot \
  --db-instance-identifier <rds_name> \
  --db-snapshot-identifier <snapshot-name> \
  --region eu-west-2

Copy an automated snapshot to a manual one where the point in time you need is only held in an automated snapshot:

aws rds copy-db-snapshot \
  --source-db-snapshot-identifier <automated-snapshot-id> \
  --target-db-snapshot-identifier <new-manual-snapshot-name> \
  --copy-tags \
  --region eu-west-2

Confirm the snapshot belongs to the right database

Check the snapshot’s source instance, engine version and master username before using it. db_engine_version and rds_family in your config must match the snapshot.

aws rds describe-db-snapshots \
  --db-snapshot-identifier <snapshot-name> \
  --query 'DBSnapshots[0].[DBInstanceIdentifier,InstanceCreateTime,EngineVersion,AllocatedStorage,MasterUsername]' \
  --output text

Restoring a snapshot that belongs to a different database carries that database’s credentials and encryption key across with it. See The restored database has the wrong username or password.

Check for a final snapshot name clash

When Terraform destroys a database it writes a final snapshot named <rds_name>-finalsnapshot. That name is fixed, so a snapshot left over from an earlier restore will block the apply. Check whether one already exists:

aws rds describe-db-snapshots \
  --db-snapshot-identifier <rds_name>-finalsnapshot \
  --region eu-west-2

If it does, see DBSnapshotAlreadyExists.

Check for a read replica

If the database has a read replica, it must be dealt with before restoring. See Read replicas.

Scenario A: Restore to Same Database

Use this when restoring a snapshot back to the original database instance.

Preperation

Raise a PR to temporarily set deletion_protection = false in your RDS module configuration. Attempting to restore the RDS instance while deletion protection is enabled will result in an error from AWS and Terraform.

1. Get current DB credentials

cloud-platform decode-secret -n <namespace> -s <rds-secret-name>

Note down: rds_instance_endpoint, database_name, database_username, database_password

The rds_name is the first part of the endpoint (e.g., cloud-platform-xxxxxxxx from cloud-platform-xxxxxxxx.xxxxx.eu-west-2.rds.amazonaws.com).

2. Update Terraform

Add snapshot_identifier to your RDS module:

module "rds" {
  source = "github.com/ministryofjustice/cloud-platform-terraform-rds-instance?ref=9.2.0"
  # ... existing config ...

  snapshot_identifier = "<snapshot name>"
}

Raise PR and merge.

3. Verify data

Port-forward to the restored DB:

Ensure there are no port-forward pods already running. If there are, they will need to be cleaned up.

kubectl -n <namespace> run port-forward-pod --image=ministryofjustice/port-forward \
  --port=5432 \
  --env="REMOTE_HOST=<db-endpoint>" \
  --env="LOCAL_PORT=5432" \
  --env="REMOTE_PORT=5432"

kubectl -n <namespace> port-forward port-forward-pod 5432:5432

Connect and verify:

PGPASSWORD=<password> psql -h 127.0.0.1 -p 5432 \
  -U <username> -d <dbname> \
  -c "SELECT * FROM <table> LIMIT 5;"

4. Tidy up

Important: Removing snapshot_identifier forces Terraform to destroy and recreate the database. The replacement is built without a snapshot, so the new database will be empty and the data you have just restored will be lost. Do not raise this as a routine clean-up PR.

Delete the port-forward pod straight away: kubectl -n <namespace> delete pod port-forward-pod
Leave snapshot_identifier in place. While its value is unchanged it produces no plan diff and has no effect on your running database. If you want it removed, ask in #ask-cloud-platform so it can be cleared without rebuilding the instance.
Raise a PR to set deletion_protection = true. This is your safeguard against an accidental replacement: with it enabled AWS refuses the delete, so the pipeline fails instead of destroying the database.

Scenario B: Restore to NEW Database (Cross-DB)

Use this when restoring a snapshot to a completely different database instance.

Note: These instructions are only relevant to databases being migrated within the same namespace.

Prerequisites

Complete the checks that apply to both scenarios first. The allocated storage check is the most common cause of a failed apply in this scenario, because the new database must be provisioned with at least as much storage as the one the snapshot came from.

1. Get SOURCE DB credentials

You need the source DB credentials for verification later.

cloud-platform decode-secret -n <source-namespace> -s <source-rds-secret-name>

Note down: database_name, database_username: these will be needed to connect after restore.

2. Add new RDS module

Add the RDS, including both rds_name AND snapshot_identifier:

module "rds_new" {
  source = "github.com/ministryofjustice/cloud-platform-terraform-rds-instance?ref=9.2.0"

  rds_name            = "<new-db-name>"       # What you want the new DB instance to be called
  snapshot_identifier = "<snapshot name>"     # Snapshot to restore from

  db_engine_version = "<version>"    # Must match source DB version
  rds_family        = "<family>"      # Must match source DB family

  # ... other required config ...
}

Raise PR and merge.

3. Verify data

Port-forward to the new DB:

kubectl -n <namespace> run port-forward-pod --image=ministryofjustice/port-forward \
  --port=5432 \
  --env="REMOTE_HOST=<new-db-endpoint>" \
  --env="LOCAL_PORT=5432" \
  --env="REMOTE_PORT=5432"

kubectl -n <namespace> port-forward port-forward-pod 5432:5432

Connect and verify:

PGPASSWORD=<k8s-secret-password> psql -h 127.0.0.1 -p 5432 \
  -U <username> -d <dbname> \
  -c "SELECT * FROM <table> LIMIT 5;"

Clean up:

kubectl -n <namespace> delete pod port-forward-pod

4. Enable deletion protection

After verifying the restored data, raise a PR to enable deletion_protection on the new database:

module "rds_new" {
  source = "github.com/ministryofjustice/cloud-platform-terraform-rds-instance?ref=9.2.0"

  rds_name            = "<new-db-name>"
  snapshot_identifier = "<snapshot name>"
  deletion_protection = true

  # ... other config ...
}

This prevents accidental destruction of the database. If snapshot_identifier is ever removed from the config by mistake, Terraform will attempt to destroy and recreate the database, but AWS will block the deletion with:

Cannot delete protected DB Instance, please disable deletion protection and try again.

The database and its data will remain intact.

5. ⚠️ CRITICAL: Do NOT remove snapshot_identifier

Warning: If you are restoring across databases, never remove snapshot_identifier from the Terraform config. Add a comment above it warning future engineers that removing it will destroy the database.

Removing it will:

Destroy the restored database
Create a new empty database in its place
Lose all of the restored data

Leave snapshot_identifier in your config permanently. While its value is unchanged it produces no plan diff and has no effect on the running database.

With deletion_protection enabled (step 4), an accidental removal of snapshot_identifier is caught and blocked before data is lost.

Version Downgrades

AWS does not support in-place version downgrades. If you attempt to restore a snapshot to a DB with a mismatched engine version, you will see an error like:

Error: creating RDS DB Instance (restore from snapshot): InvalidParameterCombination: 
The parameter group with DBParameterGroupFamily postgres17 can't be used for this instance. 
Use a parameter group with DBParameterGroupFamily postgres15.

To restore from a snapshot taken on an older version, use Scenario B (cross-DB restore) ensuring db_engine_version and rds_family match the source DB.

Disabling Deletion Protection

If you have a genuine need to delete a database that has deletion_protection enabled (e.g. decommissioning a service, removing a test database), you will need to disable it first. Attempting to delete a protected database will fail with:

Cannot delete protected DB Instance, please disable deletion protection and try again.

To proceed with deletion:

1. Disable deletion protection via AWS CLI

You cannot disable deletion protection through a Terraform PR alone. Because snapshot_identifier has been removed, Terraform will attempt to destroy and recreate the database rather than perform an in-place update, and the destroy will be blocked by deletion protection.

Instead, disable deletion protection directly using the AWS CLI from a Cloud Platform Service Pod:

aws rds modify-db-instance \
  --db-instance-identifier <rds_name> \
  --no-deletion-protection

Verify it has been disabled:

aws rds describe-db-instances \
  --db-instance-identifier <rds_name> \
  --query 'DBInstances[0].DeletionProtection'

This should return false.

2. Remove the database

Raise a PR to remove the RDS module (and its associated kubernetes_secret and kubernetes_config_map resources) from your Terraform config. Also remove the deletion_protection parameter in the same PR so your config is in-sync with state.

Merge and wait for the apply to complete. The database will now be successfully destroyed.

Note: This two-step process is intentional. It ensures that database deletion is always a deliberate action.

Troubleshooting

DBSnapshotAlreadyExists

Error: deleting RDS DB Instance (<rds_name>): DBSnapshotAlreadyExists: Cannot create
the snapshot because a snapshot with the identifier <rds_name>-finalsnapshot already exists.

Terraform destroys the existing instance before creating its replacement, and on the way out it writes a final snapshot named <rds_name>-finalsnapshot. That name is fixed, so a snapshot left over from an earlier restore blocks the apply.

Confirm the data you need is held elsewhere, then delete the conflicting snapshot and re-run the build:

aws rds delete-db-snapshot \
  --db-snapshot-identifier <rds_name>-finalsnapshot \
  --region eu-west-2

Warning: Check what that snapshot contains before deleting it. If your database has already been replaced once, <rds_name>-finalsnapshot may hold the only copy of the data from before that happened. Compare its InstanceCreateTime against the database you expect it to have come from, and copy it to a safely named manual snapshot if you are unsure.

Setting skip_final_snapshot does not avoid the clash

Adding skip_final_snapshot = true in the same PR as snapshot_identifier will not prevent the error above.

Changing snapshot_identifier forces Terraform to replace the instance, and the destroy half of a replacement uses the values already recorded in state, not the values in your new PR. skip_final_snapshot = true only takes effect once it has been applied and stored in state, which cannot happen in the same run.

Delete the conflicting snapshot instead.

The restored database has the wrong username or password

The master username and password are generated by the Terraform module and held in state, but a restored instance keeps the master username from the snapshot.

If you restore from a snapshot belonging to a different database, such as another environment or namespace, the restored instance comes up with that database’s credentials, which will not match what your application expects. Your pods will fail to connect even though the restore itself reported success.

A snapshot from another database also carries that database’s KMS encryption key, and the restored instance inherits that key rather than using your own. If you need to restore from a snapshot belonging to a different database, raise it in #ask-cloud-platform before you proceed.

Always check the snapshot’s source instance before using it. See Confirm the snapshot belongs to the right database.

Read replicas

If the database has a read replica, Terraform fails with:

Error: cannot elect new source database for replication

When a source database is destroyed, AWS promotes its read replica to a standalone instance, and a promoted instance can no longer be used as a replication target. Terraform therefore has no way to reattach it to the restored database, and re-running the build will not clear this. See Considerations when deleting replicas.

The replica has to be destroyed and rebuilt, in two separate PRs:

Set count = 0 on the read replica module and on the kubernetes_secret resource for the replica. Both must change together, or the plan fails with module.read_replica is empty tuple. Merge and let it apply.
Set both back to count = 1. This builds a fresh replica from the restored database.

The rebuilt replica has a new database identifier and endpoint, so restart any pods that read from it.

If you want the promoted replica’s data kept, snapshot it before step 1. The module sets skip_final_snapshot = true and db_backup_retention_period = 0 on replicas, so nothing is retained when one is deleted.

Permission denied copying a snapshot

Copying an encrypted snapshot from the service pod may fail with an access denied error. Raise this in #ask-cloud-platform with the error message, and the Cloud Platform team can carry out the copy for you.

This page was last reviewed on 16 October 2026. It needs to be reviewed again on 16 April 2027 by the page owner #cloud-platform-notify .