Cluster mode connections growing over time
Problem
We've recently been seeing huge slowdowns on our servers. We can't totally blame on redis yet, but we've noticed a huge amount of connections on our redis instance It seems to go down only when we restart the servers. For context, we have roughly 16 servers running all creating 4 connections each to the cluster. The connections are long-lived (aka we never create new ones during the process). So I can only assume something in the reconnection behavior maybe isn't cleaning up the previous connections. We are currently using ioredis version 4.19.2. I'll look at updating to latest see if it helps, but I haven't seen anything in the changelog that would suggest a fix. I also stumbled on #1295 which was closed for lack of more info. I can provide as much info as I can. Not sure I can run `DEBUG=ioredis:*` on our production servers, but here's what we have from cloudwatch for the past 3 months where we sometimes peak over 700 connections I also ran `client list` on our redis instance a few hours after restarting many of our servers if it can help. We can see that a single addr can have multiple connections to the same topic(name) <details> [code block] </details>
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: Cluster mode connections growing over time
To give an idea, since I created this issue yesterday (about 17hours ago) the number of connections went from ~180 to ~280 Might be useful to know that out of our 16 servers, 8 are for production and 8 are for beta. Whenever we redeploy to the beta servers, the connections barely changes, which seems to hint that our production servers are using pretty much all the connections. So I'm guessing th
Trust Score
1 verification
- 1
To give an idea, since I created this issue yesterday (about 17hours ago) the nu
To give an idea, since I created this issue yesterday (about 17hours ago) the number of connections went from ~180 to ~280
- 2
Might be useful to know that out of our 16 servers, 8 are for production and 8 a
Might be useful to know that out of our 16 servers, 8 are for production and 8 are for beta. Whenever we redeploy to the beta servers, the connections barely changes, which seems to hint that our production servers are using pretty much all the connections. So I'm guessing that work load probably have an impact
Validation
Resolved in redis/ioredis GitHub issue #1449. Community reactions: 0 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep