FG
🗄️ Databases

Cluster: problem when using Slaves for READ operations

Freshabout 21 hours ago
Mar 14, 20260 views
Confidence Score51%
51%

Problem

I'm using Redis in Cluster mode with ioredis. I discovered the `{ readOnly: true }` option which looks amazing. However, I've got some new MOVED errors since using this option. It is only happening on write operations (`del`, `hincrby`, etc.). I looked at the code and this piece of code (in cluster.js) attracted my attention: [code block] My guess: when the option readOnly is activated, it selects randomly one of the nodes that is reponsible of the targetSlot. In a setup where you've got a Slave for each Master, the selection is made between two nodes (one Master and its Slave). It means that the Read operations are randomly distributed between these two nodes, which is the objective (even though I originally thought it would only go on Slave nodes). However, it seems that the same process is applied for Write operations. It creates an issue because if the Slave node is selected for this operation, it will return a MOVED error. Eventually, with a few retries there is a good chance that the Master node will be picked by the random selector. So the operation has a good chance of succeeding. However, it looks very inefficient and with a lot of operations you'd get some operations actually failing. Please let me know if it's a bug or if I'm not using the library properly

Unverified for your environment

Select your OS to check compatibility.

1 Fix

Canonical Fix
Unverified Fix
New Fix – Awaiting Verification

Correctly Configure ioredis for Cluster Read/Write Operations

Medium Risk

The issue arises because the `{ readOnly: true }` option in ioredis causes the client to randomly select between Master and Slave nodes for read operations. When a write operation is attempted on a Slave node, it results in a MOVED error. This happens because the library does not differentiate between read and write operations when selecting nodes, leading to inefficient retries and potential operation failures.

Awaiting Verification

Be the first to verify this fix

  1. 1

    Remove readOnly Option for Write Operations

    Ensure that write operations are not using the `{ readOnly: true }` option. This will prevent the client from attempting to write to Slave nodes, thus avoiding MOVED errors.

    javascript
    const client = new ioredis.Cluster(nodes, { readOnly: false });
  2. 2

    Use Separate Clients for Read and Write

    Create two separate ioredis clients: one for read operations (with `{ readOnly: true }`) and another for write operations (without this option). This ensures that reads are directed to Slaves and writes to Masters.

    javascript
    const readClient = new ioredis.Cluster(nodes, { readOnly: true });
const writeClient = new ioredis.Cluster(nodes, { readOnly: false });
  3. 3

    Implement Error Handling for MOVED Errors

    Add error handling logic to retry write operations if a MOVED error occurs. This will ensure that the operation is retried on the correct Master node.

    javascript
    writeClient.del('key').catch(err => {
  if (err.message.includes('MOVED')) {
    // Logic to retry on the correct Master node
  }
});
  4. 4

    Test Configuration

    After making the changes, run a series of tests to ensure that read operations are successfully directed to Slave nodes and write operations are handled correctly without MOVED errors.

    javascript
    // Example test
await readClient.get('key');
await writeClient.del('key');

Validation

Confirm that write operations no longer return MOVED errors and that read operations are successfully distributed across Slave nodes. Monitor logs for any further error occurrences.

Sign in to verify this fix

Environment

Submitted by

AC

Alex Chen

2450 rep

Tags

redisiorediscachediscussion