Sockets stay / remain established / connected on operation system but disconnected in socket.io
Problem
Hello, my problem : on .connection-event i count new connections, on disconnect-event i reduce the counter. internal it looks good - but on operation system the sockets stay / remain established and the amount is growing. (see my log downside) Please look at my 2nd posting - i guess i found out what's going on. Some ideas how to monitor it more detailed ? Can it be a problem between Node and Socket.io an one hand and the interaction on OS level ? At the moment i have to restart the node.js process every few hours ... Some ideas what to check ? (by the way we use {'sync disconnect on unload':true} on client side) Server: Debian 7.7 full updated, 4 GB, 4 CPU, KVM Server, express@4.10.6, Socket.io 1.2.1, Node.js v0.10.31 - could not finde bugs or issues related to this kind of problems. Yesterday i pachted Host Server up to 7.7 and i changed Node.js to v.0.10.33 - still growing amount of sockets on OS level Networksettings - i used Debian default and echo "1024 64000" > /proc/sys/net/ipv4/ip_local_port_range echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse echo "15" > /proc/sys/net/ipv4/tcp_fin_timeout Counted on Server/ OS level --- Fr 12. Dez 18:51:05 CET 2014 8 CLOSING 3 FIN_WAIT1 3 FIN_WAIT2 9 LAST_ACK 1 LISTEN 4 SYN_RECV 1308 TIME_WAIT 8009 VERBUNDEN/Established 9343 Counted internal in Socket.io/ Node.js --- root@aaaaaa:~tail -f /var/log/push.log Fri Dec 12 2014 18:46:33 GMT+0100 (CET) 2475 Fri Dec 12 2014 18:47:03 GMT+0100
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Implement Socket Cleanup and Monitoring for Persistent Connections
The issue arises due to lingering TCP connections in the TIME_WAIT state, which can occur when clients disconnect improperly or when the server does not handle disconnections effectively. The use of 'sync disconnect on unload' may not be sufficient to close sockets on the client side, leading to an accumulation of established connections on the server. Additionally, the outdated versions of Node.js and Socket.io may not handle socket disconnections optimally.
Awaiting Verification
Be the first to verify this fix
- 1
Upgrade Node.js and Socket.io
Upgrade Node.js to a more recent LTS version (e.g., Node.js 14 or 16) and Socket.io to the latest stable version to benefit from improved socket management and bug fixes.
bashnvm install 16 && nvm use 16 npm install socket.io@latest - 2
Implement Socket Disconnection Handling
Ensure that the server properly handles disconnections by implementing a timeout for inactive sockets. This can help clean up lingering connections that are not properly closed.
typescriptio.on('connection', (socket) => { socket.on('disconnect', () => { // Handle disconnection logic }); socket.setTimeout(30000); // Set timeout to 30 seconds }); - 3
Monitor Active Connections
Implement a monitoring solution to log active connections and their states. This can help identify patterns of disconnections and potential leaks.
typescriptsetInterval(() => { const activeSockets = io.sockets.sockets.size; console.log(`Active sockets: ${activeSockets}`); }, 60000); // Log every minute - 4
Adjust TCP Settings
Review and adjust TCP settings to optimize connection handling. Consider increasing the tcp_fin_timeout and lowering the tcp_tw_reuse settings if necessary.
bashecho '30' > /proc/sys/net/ipv4/tcp_fin_timeout - 5
Test and Validate Changes
After implementing the above changes, conduct load testing to simulate multiple connections and disconnections. Monitor the server's socket states to ensure that the number of established connections does not grow uncontrollably.
bashab -c 100 -n 1000 http://yourserver:port/ // Use Apache Benchmark for load testing
Validation
Confirm the fix by monitoring the number of established sockets on the server after implementing the changes. Ensure that the count stabilizes and does not grow excessively over time. Use tools like 'netstat' to check socket states.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep