[SERVER-19366] Not able to add a node to cluster Created: 11/Jul/15  Updated: 13/Jul/15  Resolved: 13/Jul/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Varun Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

When I am trying to add a node to the cluster we are getting the below error. Please let me know for any further information

qa_1:PRIMARY> rs.add("n7vgq1mhddbmgd02.general.disney.private:10001");
assert failed : error: local.system.replset has unexpected contents
Error: Printing Stack Trace
at printStackTrace (src/mongo/shell/utils.js:37:15)
at doassert (src/mongo/shell/assert.js:6:5)
at assert (src/mongo/shell/assert.js:14:5)
at Function.rs.add (src/mongo/shell/utils.js:1019:5)
at (shell):1:4
Sat Jul 11 08:46:56.898 assert failed : error: local.system.replset has unexpected contents at src/mongo/shell/assert.js:7



 Comments   
Comment by Sam Kleinman (Inactive) [ 13/Jul/15 ]

Have you been able to reproduce this error? The error you encountered was in the mongo shell, so you may be able to get around it by using a different client (i.e. driver) or a more recent version of the shell.

MongoDB 3.0 includes more strict enforcement of replica set configurations notes, although its unclear from the current state of the cluster if the replica set configuration was previously invalid.

You may have better luck attempting to resync the secondary nodes using this procedure which might help you cleanly add them to the set.

I hope this helps. For now, I'm going to go ahead and close this ticket: the SERVER project is for bugs and feature requests with the core MongoDB software. For questions and support with your MongoDB deployment consider contacting the mongodb-users group
or Stack Overflow with the mongodb tag. If you are able to reproduce the failure on a new deployment, especially on a more recent version of MongoDB or uncover more data about the cause of the error, feel free to reopen this ticket.

Regards,
sam

Comment by Varun [ 13/Jul/15 ]

Sam,
Replica set members
1. n7vgq1mhddbmgd01.general
2. n7vgq1mhddbmgd02.general
3. n7vgq1mhddbmgd03.general

MongoDB version for all replica sets: 2.4.12
MongoDB shell version for all replica sets: 2.4.12

n7vgq1mhddbmgd03.general is primary and below are the outputs of rs.conf() & rs.status()

qa_1:PRIMARY> rs.conf()
{
	"_id" : "qa_1",
	"version" : 10,
	"members" : [
		{
			"_id" : 2,
			"host" : "n7vgq1mhddbmgd03.general.disney.private:10001"
		}
	]
}
qa_1:PRIMARY> rs.status()
{
	"set" : "qa_1",
	"date" : ISODate("2015-07-13T11:56:35Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 2,
			"name" : "n7vgq1mhddbmgd03.general.disney.private:10001",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 2213577,
			"optime" : Timestamp(1436788568, 1),
			"optimeDate" : ISODate("2015-07-13T11:56:08Z"),
			"self" : true
		}
	],
	"ok" : 1
}

Previously we had 02 & 01 nodes shown in the cluster having its "stateStr" : "RECOVERY" so we tried to remove the members and add back to the cluster and the mentioned error is seen while adding members.

Comment by Sam Kleinman (Inactive) [ 11/Jul/15 ]

Hello,

Sorry that you have run into this issue. Could you please answer the following questions about your deployment:

  1. What version of MongoDB are all of the existing replica set members?
  2. Could you provide the output of the rs.conf() and rs.status() operations?
  3. What version of the mongo shell were you using when you got this error?

Regards,
sam

Generated at Thu Feb 08 03:50:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.