DEV Community

Cover image for Day 23.Data Migration Between S3 Buckets Using AWS CLI
Thu Kha Kyawe
Thu Kha Kyawe

Posted on • Edited on

Day 23.Data Migration Between S3 Buckets Using AWS CLI

Lab Information

As part of a data migration project, the team lead has tasked the team with migrating data from an existing S3 bucket to a new S3 bucket. The existing bucket contains a substantial amount of data that must be accurately transferred to the new bucket. The team is responsible for creating the new S3 bucket and ensuring that all data from the existing bucket is copied or synced to the new bucket completely and accurately. It is imperative to perform thorough verification steps to confirm that all data has been successfully transferred to the new bucket without any loss or corruption.

As a member of the Nautilus DevOps Team, your task is to perform the following:

Create a New Private S3 Bucket: Name the bucket datacenter-sync-14788.

Data Migration: Migrate the entire data from the existing datacenter-s3-12806 bucket to the new datacenter-sync-14788 bucket.

Ensure Data Consistency: Ensure that both buckets have the same data.

Use AWS CLI: Use the AWS CLI to perform the creation and data migration tasks.

Lab Solutions

✅ STEP 1 — Create the New S3 Bucket

Run this command in us-east-1:

aws s3 mb s3://datacenter-sync-14788 --region us-east-1
Enter fullscreen mode Exit fullscreen mode

S3 buckets are private by default — no extra action needed.

✅ STEP 2 — Sync (Copy) All Data From the Source Bucket

Existing bucket (source):

datacenter-s3-12806

New bucket (destination):

datacenter-sync-14788

Run:

aws s3 sync s3://datacenter-s3-12806 s3://datacenter-sync-14788
Enter fullscreen mode Exit fullscreen mode

This copies:

All objects

All folders

Maintains structure

Skips duplicates on re-run

✅ STEP 3 — Verify the Data Migration
✔ Check number of files and total size

# Source:

aws s3 ls s3://datacenter-s3-12806 --recursive --summarize


# Destination:

aws s3 ls s3://datacenter-sync-14788 --recursive --summarize
Enter fullscreen mode Exit fullscreen mode

Both should show the same number of files and the same total size.

# Dry-run comparison (should show no differences)
aws s3 sync s3://datacenter-s3-12806 s3://datacenter-sync-14788 --dryrun
Enter fullscreen mode Exit fullscreen mode

Resources & Next Steps
📦 Full Code Repository: KodeKloud Learning Labs
📖 More Deep Dives: Whispering Cloud Insights - Read other technical articles
💬 Join Discussion: DEV Community - Share your thoughts and questions
💼 Let's Connect: LinkedIn - I'd love to connect with you

Credits
• All labs are from: KodeKloud
• I sincerely appreciate your provision of these valuable resources.

Top comments (0)