≡

S3

AWS S3 Sync is Not Reliable and Slow!

This article explores reliability issues with AWS CLI’s S3 sync functionality and provides alternative solutions for better file synchronization.

While migrating from s3cmd to AWS S3 CLI, I noticed that files don’t sync properly when using AWS CLI.

I tested with different versions and they all revealed the same behavior:

Test Setup

  1. Setup AWS CLI utility and configure your credentials
  2. Create a testing S3 bucket
  3. Setup some random files
# Create 10 random files of 10MB each
for i in {1..10}; do dd if=/dev/urandom of=multi/part-$i.out bs=1MB count=10; done;
# Then copy the first 5 files over
mkdir multi-changed
cp -r multi/part-{1,2,3,4,5}.out multi-changed
# And replace the content in 5 files
for i in {6..10}; do dd if=/dev/urandom of=multi-changed/part-$i.out bs=1MB count=10; done;

Testing S3 Sync with AWS CLI

Cleanup

$ aws s3 rm s3://testbucket/multi --recursive

Initial Sync

$ aws s3 sync multi s3://testbucket/multi
upload: multi/part-1.out to s3://testbucket/multi/part-1.out         
upload: multi/part-3.out to s3://testbucket/multi/part-3.out      
upload: multi/part-2.out to s3://testbucket/multi/part-2.out      
upload: multi/part-4.out to s3://testbucket/multi/part-4.out      
upload: multi/part-10.out to s3://testbucket/multi/part-10.out    
upload: multi/part-5.out to s3://testbucket/multi/part-5.out      
upload: multi/part-6.out to s3://testbucket/multi/part-6.out      
upload: multi/part-8.out to s3://testbucket/multi/part-8.out      
upload: multi/part-7.out to s3://testbucket/multi/part-7.out      
upload: multi/part-9.out to s3://testbucket/multi/part-9.out

Update Files

Only 5 files should now be uploaded. Timestamps for all 10 files should be changed.