[SERVER-58964] Cannot add an arbiter to the replicaSet Created: 30/Jul/21  Updated: 27/Oct/23  Resolved: 24/Aug/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.0, 5.0.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Juan Jose Martos Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: arbiter
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Steps To Reproduce:

Try to add an arbiter node to a replicaSet

Participants:
Case:

 Description   

The rs.addArb command doesn't work in MongoDB 5.x.

rs.addArb('mongodb-arbiter:27017')

We have a replicaSet with 1 primary node, 1 secondary node and are trying to add the arbiter to the replicaSet. However, when running the rs.addArb command, we get this error message

Error: assert.soon failed: function() {
        var cfg = hostport;
 
        var local = db.getSiblingDB("local");
        assert(local.system.replset.count() <= 1,
               "error: local.system.replset has unexpected contents");
        var c = local.system.replset.findOne();
        assert(c, "no config object retrievable from local.system.replset");
 
        const attemptedVersion = c.version++;
 
        var max = 0;
        for (var i in c.members) {
            // Omit 'newlyAdded' field if it exists in the config.
            delete c.members[i].newlyAdded;
            if (c.members[i]._id > max)
                max = c.members[i]._id;
        }
        if (isString(hostport)) {
            cfg = {_id: max + 1, host: hostport};
            if (arb)
                cfg.arbiterOnly = true;
        } else if (arb == true) {
            throw Error(
                "Expected first parameter to be a host-and-port string of arbiter, but got " +
                tojson(hostport));
        }
 
        if (cfg._id == null) {
            cfg._id = max + 1;
        }
        c.members.push(cfg);
 
        res = self._runCmd({replSetReconfig: c});
        if (res === "") {
            // _runCmd caught an exception.
            return true;
        }
        if (res.ok) {
            return true;
        }
        if (res.code === ErrorCodes.ConfigurationInProgress ||
            res.code === ErrorCodes.CurrentConfigNotCommittedYet) {
            return false;  // keep retrying
        }
        if (res.code === ErrorCodes.NewReplicaSetConfigurationIncompatible) {
            // We will retry only if this error was due to our config version being too low.
            const cfgState = local.system.replset.findOne();
            if (cfgState.version >= attemptedVersion) {
                return false;  // keep retrying
            }
        }
        // Take no action on other errors.
        return true;
    } : {
        "ok" : 0,
        "errmsg" : "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.",
        "code" : 103,
        "codeName" : "NewReplicaSetConfigurationIncompatible",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1627634005, 1),
                "signature" : {
                        "hash" : BinData(0,"0HvodHFBd9MboNymy6f1RCY2YZs="),
                        "keyId" : NumberLong("6990612431667986435")
                }
        },
        "operationTime" : Timestamp(1627634005, 1)
} The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). :
doassert@src/mongo/shell/assert.js:20:14
assert.soon@src/mongo/shell/assert.js:382:17
rs.add@src/mongo/shell/utils.js:1624:5
rs.addArb@src/mongo/shell/utils.js:1696:12
@(shell):1:1

The version of MongoDB we are currently using is 5.0.1 and the scripts we use to configure the replicaSet work properly with MongoDB 4.x. We also tried to use rs.add('mongodb-secondary:27017', true) but got the same error. I can confirm that there is connectivity between the primary/secondary node and the arbiter one.



 Comments   
Comment by Mike M [ 02/Jun/22 ]

Dear Mongo community, we just set up a new database following the instructions and hit the same problem. Is it safe to change this default setting in MongoDB? What is the implication? I am also curious as to why it isn’t documented in any tutorial we can find.

Comment by Henk De Groot [ 15/Oct/21 ]

Hi @Dmitry,

I am trying to deploy a PSA replica set using MongoDB 5.0.3 and I see the same issue as reported by @Juan.

In previous deployments (with v3) I used to follow this process (which I believe is what the documentation suggest being correct):

I install MongoDB on the servers I want to use and get them all up and running.

Next I use the server I want to use as primary and issue the rs.initiate() command. I specify the host, the priority for the host and replica set name in this.

Now I am adding my secondary server using the command: rs.add('SECONDARYHOSTNAME')

So far working fine. Running rs.status() at this time shows all good and primary and secondary are set as expected.

Now I want to add my arbiter server to the replica set and use the command: rs.addArb('ARBITERHOSTNAME') 

The shell freezes....and after a long wait the message as indicated by Juan is returned to me in the shell.

Going back to the first post/reply from you, the issue is caused by the Default Write Concern option.

When I run the getDefaultRWConcern command, I notice the defaultReadConcern level is set to local. The defaultWriteConcenrt w is set to majority. The defaultWriteConcenrSource and defaultReadConcernSource are both set to implicit.

I think I maybe able to "fix" my install using the information provide, but I wonder why this is not part of the documentation for setting up a replica set and I wonder what the best option/command would be to do this.

As a side note question, I also wonder why I can just add secondary server to the replica set without running into this issue. So for using a PSS replica set, the Default Write Concern does not seem to be causing "issues".

Hope you have some more information to share on this.

Comment by Dmitry Agranat [ 24/Aug/21 ]

Hi jotamartos@gmail.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Juan Jose Martos [ 09/Aug/21 ]

Thank you for the information Dmitry! We are going to review that material and will update this thread if we have any questions.

Regards,
Jota

Comment by Dmitry Agranat [ 03/Aug/21 ]

Hi jotamartos@gmail.com, the issue you are experiencing is due to a change in 5.0 in regard to the Default Write Concern. So when manually adding an arbiter prior to setting the cluster-wide write concern, mongo correctly responds with:

Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.

Specifically, this function is responsible for this logic. Here is the documentation on how to change the Default Write Concern which you need to enforce before adding an Arbiter (or any other changes that will cause the implicit default WC to change).

Regards,
Dima

Generated at Thu Feb 08 05:45:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.