[SERVER-28505] Not able to add a node to repica set Created: 27/Mar/17  Updated: 27/Oct/23  Resolved: 11/Apr/17

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alok Rawat Assignee: Mark Agarunov
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-26839 Improve readWriteDatabase role coverage Closed
Operating System: ALL
Participants:

 Description   

When I am trying to add a node to the cluster we are getting the below error. -

Hotels:PRIMARY> rs.add("hotelsdrmongo01:27017")
assert failed : error: local.system.replset has unexpected contents
Error: assert failed : error: local.system.replset has unexpected contents
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at assert (src/mongo/shell/assert.js:20:5)
    at Function.rs.add (src/mongo/shell/utils.js:980:5)
    at (shell):1:4
2017-03-27T17:23:11.546+0530 E QUERY    Error: assert failed : error: local.system.replset has unexpected contents
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at assert (src/mongo/shell/assert.js:20:5)
    at Function.rs.add (src/mongo/shell/utils.js:980:5)
    at (shell):1:4 at src/mongo/shell/assert.js:13

mongo version— 3.0.8

Hotels:PRIMARY> rs.status()
{
	"set" : "Hotels",
	"date" : ISODate("2017-03-27T12:22:48.199Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 3,
			"name" : "p.ip.ip.ip:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 2345841,
			"optime" : Timestamp(1490617345, 1),
			"optimeDate" : ISODate("2017-03-27T12:22:25Z"),
			"electionTime" : Timestamp(1488271545, 1),
			"electionDate" : ISODate("2017-02-28T08:45:45Z"),
			"configVersion" : 27,
			"self" : true
		},
		{
			"_id" : 4,
			"name" : "ip.ip.ip.ip:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 2345716,
			"optime" : Timestamp(1490617345, 1),
			"optimeDate" : ISODate("2017-03-27T12:22:25Z"),
			"lastHeartbeat" : ISODate("2017-03-27T12:22:47.609Z"),
			"lastHeartbeatRecv" : ISODate("2017-03-27T12:22:47.607Z"),
			"pingMs" : 0,
			"syncingTo" : "ip.ip.ip.ip:27017",
			"configVersion" : 27
		},
		{
			"_id" : 5,
			"name" : "celery02.prod.goibibo.com:27017",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 144,
			"lastHeartbeat" : ISODate("2017-03-27T12:22:47.539Z"),
			"lastHeartbeatRecv" : ISODate("2017-03-27T12:22:47.545Z"),
			"pingMs" : 0,
			"configVersion" : 27
		}
	],
	"ok" : 1
 
Hotels:PRIMARY> rs.conf()
{
	"_id" : "Hotels",
	"version" : 27,
	"members" : [
		{
			"_id" : 3,
			"host" : "ip.ip.ip.ip:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 2,
			"tags" : {
 
			},
			"slaveDelay" : 0,
			"votes" : 1
		},
		{
			"_id" : 4,
			"host" : "ip.ip.ip.ip:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
 
			},
			"slaveDelay" : 0,
			"votes" : 1
		},
		{
			"_id" : 5,
			"host" : "ip.ip.ip.ip:27017",
			"arbiterOnly" : true,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
 
			},
			"slaveDelay" : 0,
			"votes" : 1
		}
	],
	"settings" : {
		"chainingAllowed" : true,
		"heartbeatTimeoutSecs" : 10,
		"getLastErrorModes" : {
 
		},
		"getLastErrorDefaults" : {
			"w" : 1,
			"wtimeout" : 0
		}
	}
}
Hotels:PRIMARY>



 Comments   
Comment by Ramon Fernandez Marina [ 09/Aug/18 ]

dbapower, can you please elaborate on what you're trying to do? If you think you've found a bug, please open a new SERVER ticket, otherwise please post your question on the mongodb-user group group.

Thanks,
Ramón.

Comment by He Lei [ 09/Aug/18 ]

Hi mark.agarunov,the local database system.replset collection can not be update or delete,i found the information on the doc:

https://docs.mongodb.com/manual/reference/built-in-roles/index.html

the system.replset collection only have these privileges.

so,is the https://jira.mongodb.org/browse/DOCS-4798 is not correct?

Comment by Mark Agarunov [ 11/Apr/17 ]

Hello aloksinghrawat@gmail.com,

To execute operations on the local database, a you must explicitly grant a permission of readWrite on the local database. This intentional and is due to a separation of privileges. The root role is a super-set of permissions affecting user data specifically, not system data, therefore the permissions must be explicitly granted to perform operations on local.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thanks,
Mark

Comment by Alok Rawat [ 29/Mar/17 ]

Hi Thomas,

Would you suggest to assign __system role to a user on local DB and then execute this delete ?

Regards,
Alok

Comment by Alok Rawat [ 28/Mar/17 ]

Thanks Thomas,

In an another replica set we encountered the same issue where authentication is also enabled. i am getting authentication fail even by the user which has 'root' role allocated -

goibiboCMS1:PRIMARY> db.system.replset.remove({_id:ObjectId("5873b330d5237e47e518f1ce")})
WriteResult({
"writeError" : {
"code" : 13,
"errmsg" : "not authorized on local to execute command { delete: \"system.replset\", deletes: [ { q:

{ _id: ObjectId('5873b330d5237e47e518f1ce') }

, limit: 0.0 } ], ordered: true }"
}
})

Regards,
Alok

Comment by Kelsey Schubert [ 28/Mar/17 ]

Hi aloksinghrawat@gmail.com,

I'm glad you were able to resolve the issue. From your responses, I do not see anything to indicate a bug in the MongoDB server. However, if you encounter this behavior again and can provide steps to reproduce we will gladly continue to investigate.

Kind regards,
Thomas

Comment by Alok Rawat [ 28/Mar/17 ]

db.system.replset.remove({_id:ObjectId("58cfb609db8dece90d1819b2")})

Worked for me. able to add node after remove these un wanted entries.

Regards,
Alok

Comment by Alok Rawat [ 27/Mar/17 ]

Hello Thomas,

Here is output-

ingoibibo:PRIMARY> db.getSiblingDB("local").system.replset.find()
{ "_id" : "ingoibibo", "version" : 24, "members" : [ { "_id" : 3, "host" : "ipadd:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 2, "tags" : {  }, "slaveDelay" : 0, "votes" : 1 }, { "_id" : 4, "host" : "ipadd:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : {  }, "slaveDelay" : 0, "votes" : 1 }, { "_id" : 5, "host" : "ipadd:27017", "arbiterOnly" : true, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : {  }, "slaveDelay" : 0, "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatTimeoutSecs" : 10, "getLastErrorModes" : {  }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 } } }
{ "_id" : ObjectId("58739d84b5a2a8019a31bda5") }
{ "_id" : ObjectId("58739d89b5a2a8019a31bda8") }
{ "_id" : ObjectId("58739dabb5a2a8019a31bdab") }
{ "_id" : ObjectId("5880a3dfb5a2a8019a31c75b") }
{ "_id" : ObjectId("5880a413b5a2a8019a31c75e") }
{ "_id" : ObjectId("5880a422b5a2a8019a31c761") }
{ "_id" : ObjectId("58cfb5c3db8dece90d1819ac") }
{ "_id" : ObjectId("58cfb5eedb8dece90d1819af") }
{ "_id" : ObjectId("58cfb609db8dece90d1819b2") }

Seems there is some unexpected data in the collection. i tried to remove these IDs but could not do so.

Thanks,
Alok

Comment by Kelsey Schubert [ 27/Mar/17 ]

Hi aloksinghrawat@gmail.com,

Would you please provide the output of db.getSiblingDB("local").system.replset.find()?

Thank you,
Thomas

Generated at Thu Feb 08 04:18:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.