All of a sudden, I'm receiving this error
Problem
[code block] I have a 6 nodes setup, I saw this in the log of a node: [code block] I think ioredis is not recovering well alongside the redis cluster(3.2)
Error Output
Error: Too many Cluster redirections. Last error: ReplyError: MOVED 6686 10.240.100.31:7000]
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Increase Cluster Redirection Limit in ioredis Configuration
The error 'Too many Cluster redirections' indicates that the ioredis client is encountering excessive redirection responses from the Redis cluster. This typically occurs when the client is trying to access a key that has been moved to a different node in the cluster, but is not handling the redirection properly. The default redirection limit may be too low for your workload, especially if there are frequent key migrations or if the cluster is under heavy load.
Awaiting Verification
Be the first to verify this fix
- 1
Update ioredis Configuration
Modify the ioredis client configuration to increase the maximum number of allowed redirections. This can help the client recover from transient issues more effectively.
javascriptconst Redis = require('ioredis'); const redis = new Redis({ clusterRetryStrategy: (times) => { return Math.min(times * 100, 3000); }, maxRedirections: 20 // Increase this value as needed }); - 2
Monitor Redis Cluster Health
Check the health of your Redis cluster to ensure that all nodes are operational and there are no network issues causing excessive redirections. Use the 'CLUSTER INFO' command to gather insights.
javascriptredis.cluster('info').then(console.log).catch(console.error); - 3
Review Key Distribution
Analyze the key distribution across your Redis nodes to ensure that keys are evenly distributed. Uneven distribution can lead to hotspots and increased redirection. Use the 'CLUSTER KEYSLOT' command to check key slots.
javascriptconst slot = redis.cluster('keyslot', 'your-key'); console.log(slot); - 4
Test Application Load
Simulate application load to verify that the changes to the ioredis configuration effectively reduce the number of redirection errors. Monitor logs for any remaining errors.
javascriptfor (let i = 0; i < 1000; i++) { redis.get('your-key').then(console.log).catch(console.error); }
Validation
Confirm that the number of redirection errors in the logs has decreased significantly after implementing the changes. Additionally, monitor application performance to ensure that key retrievals are successful without excessive delays.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep