Intermittent "Connection is closed" errors
Problem
We are currently working on a Lambda function, which connects to a Redis 3.2.10 cluster on [AWS Elasticache][aws-ec]. This Lambda function will connect to the Redis cluster, run `KEYS` on each master node, collect the responses from each node and return an array of keys. We then publish an SNS message for each key in this array, then close the cluster connection, before the Lambda ends. AWS Lambda freezes and thaws the container in which programs run. So, ideally we would create a connection once then re-use it on every invocation. However, we have found that for the Lambda to end, we must explicitly end the client connection to the cluster as Lambda waits for the Node event loop to empty before the Lambda ends. This is why we create the connection at the start of the function (representing a Lambda invocation) run our queries and then when this completes we attempt to gracefully `.quit()` the `Redis.Cluster` connection. I can't share the actual code that we're working on, but I've been able to extract the logic and create a simple example of the issue we're facing: `test.js` [code block] Example output: [code block] Why would we be getting the `Connection is closed` rejection error? This feels like a bug, as I think we are going about this in the correct way, but I'm happy to be proved wrong! [aws-ec]: https://aws.amazon.com/elasticache/
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Implement Connection Pooling for Redis in Lambda
The 'Connection is closed' error occurs because the Redis connection is being closed before all pending operations are completed. AWS Lambda's execution environment can freeze and thaw, leading to unexpected behavior if connections are not managed properly. When the Lambda function attempts to quit the Redis connection while there are still pending commands, it results in a rejection error.
Awaiting Verification
Be the first to verify this fix
- 1
Use Connection Pooling
Instead of creating a new Redis connection for each invocation, implement a connection pool that allows multiple invocations to share the same connection. This reduces the overhead of establishing new connections and minimizes the risk of closing a connection prematurely.
javascriptconst Redis = require('ioredis'); const redis = new Redis.Cluster(['redis-cluster-endpoint']); exports.handler = async (event) => { const keys = await redis.keys('*'); // Process keys and publish SNS messages return keys; }; - 2
Gracefully Handle Connection Closure
Ensure that the Redis connection is only closed after all operations are complete. Use a try-catch block to handle any errors during the process and close the connection in a finally block to guarantee it runs after the operations.
javascripttry { const keys = await redis.keys('*'); // Publish SNS messages } catch (error) { console.error('Error fetching keys:', error); } finally { await redis.quit(); } - 3
Increase Timeout Settings
Adjust the timeout settings for the Redis connection to allow more time for operations to complete, especially under heavy load. This can help prevent premature closure of connections.
javascriptconst redis = new Redis.Cluster(['redis-cluster-endpoint'], { connectTimeout: 10000, maxRetriesPerRequest: 3 }); - 4
Monitor Connection State
Implement logging to monitor the state of the Redis connection. This will help identify if connections are being closed unexpectedly or if there are other issues affecting connectivity.
javascriptredis.on('error', (err) => { console.error('Redis error:', err); }); redis.on('connect', () => { console.log('Redis connected'); });
Validation
To confirm the fix worked, deploy the updated Lambda function and monitor the logs for any 'Connection is closed' errors. Additionally, verify that the Redis connection remains open during the execution of the function and that all keys are fetched and processed without errors.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep