aws s3 cp hangs on download more than 500MB on content
Problem
Hello, I am just doing daily backups from s3 files. Bassically I copy from a s3 bucket files to local disk to zip them and copy after to another s3 bucket the zipped contain. The issue is the aws s3 get stuck in any moment and doesn't download, I mean, the download starts but is interrupted in an specific moment in an specific file. This file can be accessed via web or any s3 explorer. the files are not big. My aws cli is: aws-cli/1.9.18 Python/2.7.10 Linux/4.1.13-19.31.amzn1.x86_64 botocore/1.3.18 OK I will change the issue, aws s3 cp is hanging when is downloading a lot of files. I checked with 2 s3 buckets with no long content and it downloads well but not the same when the bucket has a lot of files: S3 bucket content between 200 and 500MB on storage are downloaded without problems but not the same when content is beyond 500MB. Regards
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: aws s3 cp hangs on download more than 500MB on content
I'm having the same problem with both cp --recursive and sync, with this version: `aws-cli/1.10.8 Python/2.7.6 Linux/3.16.0-41-generic botocore/1.3.30` I also tried the ubuntu package version, and the same problem : `aws-cli/1.2.9 Python/3.4.3 Linux/3.16.0-41-generic` It always hangs towards the end, not at the beginning, which makes it more annoying. For example this is the last message I see :
Trust Score
3 verifications
- 1
I'm having the same problem with both cp --recursive and sync, with this version
`aws-cli/1.10.8 Python/2.7.6 Linux/3.16.0-41-generic botocore/1.3.30` I also tried the ubuntu package version, and the same problem : `aws-cli/1.2.9 Python/3.4.3 Linux/3.16.0-41-generic`
- 2
It always hangs towards the end, not at the beginning, which makes it more annoy
`Completed 6343 of 6349 part(s) with 6 file(s) remaining`
- 3
I run a very basic command :
`aws s3 sync s3://bucketname/dir /tmp/localdestination` or `aws s3 cp --recursive s3://bucketname/dir /tmp/localdestination`
- 4
This happens too when the content has as few as 100 files, each of size 2MB.
When I try to interrupt the script with CTRL+C, the "Cleaning up. Please wait..." phase also takes a long time, so I hit CTRL+C a second time to exit.
Validation
Resolved in aws/aws-cli GitHub issue #1775. Community reactions: 4 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep