Memory leak on high frequency data.
Problem
Hi All I am using oiredis to save some data in bulk, around 12k rows every second. the main issue that it happening is that node is not able to recover memory from the ioredis operation growing forever. I am saving the data in two stages, first one I received several thousand messages that are stored in memory reaching upto 140 mb then once a second I grab part of that data and I pipeline it into redis where grows to 1gb within 4 hours until it crash the process due the memory limitations. The easy fix it is to use "old --max-old-space-size" trick but everyone know that this does not fix the problem but the symptoms. Node version 12.4.0 NPM 6.9.0 ioredis 4.9.0 Windows 10 Any idea or recommendation, what is better ? reuse/create new connection, use/avoid await/async ? AA
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Solution: Memory leak on high frequency data.
Nevermind to reply.... Removed ioredis from my code. Nonetheless It's certain that ioredis has a serious memory leak issues. Would revisit DataHandler code ( created a fork of Tedis project with code which may help you to identify the issue and fix it... so suggest you guys look at it - maybe helps). BTW, I am sure it's ok to use ioredis to eventually send a command() SETs GETs, etc... but would
Trust Score
4 verifications
- 1
Nevermind to reply.... Removed ioredis from my code.
Nevermind to reply.... Removed ioredis from my code.
- 2
Nonetheless It's certain that ioredis has a serious memory leak issues. Would re
Nonetheless It's certain that ioredis has a serious memory leak issues. Would revisit DataHandler code ( created a fork of Tedis project with code which may help you to identify the issue and fix it... so suggest you guys look at it - maybe helps). BTW, I am sure it's ok to use ioredis to eventually send a command() SETs GETs, etc... but would not use it for real production in the backend.
Validation
Resolved in redis/ioredis GitHub issue #891. Community reactions: 5 upvotes.
Verification Summary
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep