Uploading files to S3 from EC2 instances fails on some instance types
Problem
We have a very strange problem. We are uploading large (10M) files from an EC2 instance to an S3 bucket in the same region. We use `aws s3 cp` for the upload and instance roles for authentication. If we use an `m1.small` EC2 instance the upload works fine. But if we increase the instance size (we have tried `m1.large` and `m3.large`) the upload fails. Here is the error we get: [code block] This is completely reproducible -- we have never had such a upload succeed after tens of attempts. We have never seen the problem on an `m1.small` instance in hundreds of attempts. We ran into this problem with 10M files. We have found that it is reproducible down to about 1M. Much less than this and it works fine every time. Any ideas about what is going on would be much appreciated.
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Adjust EC2 Instance Network Configuration for S3 Uploads
The issue arises from the network configuration and performance characteristics of different EC2 instance types. Larger instances may have different network interfaces or configurations that impact how they handle large file uploads, especially when using the AWS CLI. The m1.small instance may have a simpler network stack that handles uploads more reliably under certain conditions, while larger instances could be experiencing timeouts or throttling due to their network settings or the way they manage TCP connections.
Awaiting Verification
Be the first to verify this fix
- 1
Increase TCP Timeout Settings
Modify the TCP timeout settings on the EC2 instances to allow longer upload times for larger files. This can help prevent timeouts that may be causing the upload failures.
bashecho 'net.ipv4.tcp_fin_timeout = 60' | sudo tee -a /etc/sysctl.conf && sudo sysctl -p - 2
Enable Enhanced Networking
Ensure that Enhanced Networking is enabled for the larger instance types. This can improve network performance and reduce latency during uploads.
bashaws ec2 modify-instance-attribute --instance-id <instance-id> --ena-support - 3
Adjust S3 Transfer Acceleration
Consider enabling S3 Transfer Acceleration on the bucket to improve upload speeds and reliability, especially for larger files.
bashaws s3api put-bucket-accelerate-configuration --bucket <bucket-name> --accelerate-configuration Status=Enabled - 4
Test with Different MTU Settings
Experiment with different Maximum Transmission Unit (MTU) settings on the EC2 instances to find an optimal configuration that reduces packet fragmentation during uploads.
bashsudo ifconfig eth0 mtu 9001
Validation
To confirm the fix worked, perform multiple uploads of large files (1M and 10M) from the modified EC2 instances to the S3 bucket. Monitor the success rate of these uploads and check for any error messages. Additionally, verify the network performance metrics using CloudWatch.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep