≡

Performance

AWS S3 Sync is Not Reliable and Slow!

This article explores reliability issues with AWS CLI’s S3 sync functionality and provides alternative solutions for better file synchronization.

While migrating from s3cmd to AWS S3 CLI, I noticed that files don’t sync properly when using AWS CLI.

I tested with different versions and they all revealed the same behavior:

Test Setup

  1. Setup AWS CLI utility and configure your credentials
  2. Create a testing S3 bucket
  3. Setup some random files
# Create 10 random files of 10MB each
for i in {1..10}; do dd if=/dev/urandom of=multi/part-$i.out bs=1MB count=10; done;
# Then copy the first 5 files over
mkdir multi-changed
cp -r multi/part-{1,2,3,4,5}.out multi-changed
# And replace the content in 5 files
for i in {6..10}; do dd if=/dev/urandom of=multi-changed/part-$i.out bs=1MB count=10; done;

Testing S3 Sync with AWS CLI

Cleanup

$ aws s3 rm s3://testbucket/multi --recursive

Initial Sync

$ aws s3 sync multi s3://testbucket/multi
upload: multi/part-1.out to s3://testbucket/multi/part-1.out         
upload: multi/part-3.out to s3://testbucket/multi/part-3.out      
upload: multi/part-2.out to s3://testbucket/multi/part-2.out      
upload: multi/part-4.out to s3://testbucket/multi/part-4.out      
upload: multi/part-10.out to s3://testbucket/multi/part-10.out    
upload: multi/part-5.out to s3://testbucket/multi/part-5.out      
upload: multi/part-6.out to s3://testbucket/multi/part-6.out      
upload: multi/part-8.out to s3://testbucket/multi/part-8.out      
upload: multi/part-7.out to s3://testbucket/multi/part-7.out      
upload: multi/part-9.out to s3://testbucket/multi/part-9.out

Update Files

Only 5 files should now be uploaded. Timestamps for all 10 files should be changed.

Away with grep! Use ripgrep!

Ripgrep is a tool that recursively searches your current directory for a regex pattern. This article provides a brief overview of ripgrep and why it is a good alternative to grep.

I used ack for some time for grepping source code, yet I moved on to ripgrep since it offers quite a bit more than grep and source-code-greps like The Silver Searcher. Plus it can mostly replace grep and is about 6 times faster (it’s written in Rust). I am still astonished when I get results instantly.