[SERVER-2015] Assertion failure false db/repl/rs.cpp 315 Created: 27/Oct/10  Updated: 02/Sep/11  Resolved: 02/Sep/11

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 1.6.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alvin Richards (Inactive) Assignee: Dwight Merriman
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC 2 / Ubuntu 10.04
1.6.3


Operating System: ALL
Participants:

 Description   

Problem:
Saw the following in the log of the arbiter node of a three node replset

Assertion failure false db/repl/rs.cpp 315

0x534771 0x54134f 0x66aa3a 0x66ed0e 0x66f936 0x694bb3 0x549183 0x547531 0x546c93
0x52acf8 0x837460 0x7f987aca09ca 0x7f987a25069d
/var/opt/mongodb/current/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb1) [0x534771]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x10f) [0x54134f
]
/var/opt/mongodb/current/bin/mongod(ZN5mongo11ReplSetImpl14initFromConfigERNS
13ReplSetConfigEb+0x17a) [0x66aa3a]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo7ReplSet13haveNewConfigERNS_13Repl
SetConfigEb+0xfe) [0x66ed0e]
/var/opt/mongodb/current/bin/mongod(ZN5mongo7Manager20msgReceivedNewConfigENS
7BSONObjE+0x256) [0x66f936]
/var/opt/mongodb/current/bin/mongod(_ZN5boost6detail8function26void_function_ob
j_invoker0INS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo7ManagerENS7_7BSONObjEEENS3_5lis
t2INS3_5valueIPS8_EENSC_IS9_EEEEEEvE6invokeERNS1_15function_bufferE+0x63) [0x694
bb3]
/var/opt/mongodb/current/bin/mongod(_ZNK5boost9function0IvEclEv+0x243) [0x549183]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo4task6Server6doWorkEv+0x141) [0x547531]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo4task4Task3runEv+0x33) [0x546c93]

This appear to start when the replset was taken out of the config, the following assert can be seen
Mon Oct 25 23:55:31 [ReplSetHealthPollTask] replSet 10.202.215.63:27000 SECONDARYTue Oct 26 02:08:25 [rs Manager] replset msgReceivedNewConfig version: version:
2
Tue Oct 26 02:08:25 [rs Manager] replSet info saving a newer config version to local.system.replset
Tue Oct 26 02:08:25 [rs Manager] replSet error can't find self in the repl set configuration:
Tue Oct 26 02:08:25 [rs Manager] { _id: "leadpoint0", version: 2, members: [

{ _id: 1, host: "10.202.215.63:27000", votes: 5 }

,

{ _id: 2, host: "10.243.7.47:270 00", votes: 7 }

] }Tue Oct 26 02:08:25 [rs Manager] Assertion failure false db/repl/rs.cpp 336
0x534771 0x54134f 0x66b7ac 0x66ed0e 0x66f936 0x694bb3 0x549183 0x547531 0x546c93 0x52acf8 0x837460 0x7f987aca09ca 0x7f987a25069d
/var/opt/mongodb/current/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb1) [0x534771]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x10f) [0x54134f] /var/opt/mongodb/current/bin/mongod(_ZN5mongo11ReplSetImpl14initFromConfigERNS_13ReplSetConfigEb+0xeec) [0x66b7ac]
/var/opt/mongodb/current/bin/mongod(_ZN5mongo7ReplSet13haveNewConfigERNS_13ReplSetConfigEb+0xfe) [0x66ed0e]

Reproduce:
This is what I did

  • create a three node set, one node being the arbiter
  • remove the arbiter by running replsetinitate with a new config minus the arbiter (and bumping the version)
  • add the same arbiter back

Business case:

  • reliability


 Comments   
Comment by Eliot Horowitz (Inactive) [ 02/Sep/11 ]

This was the concurrency issue fixed in 1.8.3

Comment by Alvin Richards (Inactive) [ 27/Oct/10 ]

My mistake

1) replsetinitate
2) replSetConfig

No .js script, this was not testing this was just using.

Comment by Dwight Merriman [ 27/Oct/10 ]

you should get better logging/messaging on a repeat replsetinitiate though of course, but first want to confirm you really initiated twice.

Comment by Dwight Merriman [ 27/Oct/10 ]

you should not run replsetinitiate more than once.

you should instead use replSetConfig.

do you have a .js file that reproduces? if so i can repro and add to test suite.

Generated at Thu Feb 08 02:58:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.