[SERVER-5647] adding secondary caused existing few secondaries to crash with db/repl/rs.cpp 155 Created: 19/Apr/12  Updated: 15/Aug/12  Resolved: 19/Apr/12

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Harpreet Kaur Assignee: Kristina Chodorow (Inactive)
Resolution: Duplicate Votes: 0
Labels: replicaset
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Production, EC2


Issue Links:
Duplicate
duplicates SERVER-3032 mongod crashed in ReplSetImpl summari... Closed
Participants:

 Description   

Have following configuration in Production db01-db06, added additional secondary db07 and db08, these new secondaries crash after a few minutes with the error message db/repl/rs.cpp 155. whats wrong with the configurations? i want to add additional db09 and db10 to the replicaset once this stabalizes.

db01- vote 1,Priority 1,
db02 - vote 1,Priority 1,
db03 - vote 1,Priority 1,
db04 - vote 1,Priority 0, state hidden
db05 - vote 1,Priority 0, state hidden
db06 - vote 0,Priority 0, state hidden
db07 - vote 1,Priority 0,
db08 - vote 1,Priority 0,



 Comments   
Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

Yes, same bug.

Comment by Harpreet Kaur [ 19/Apr/12 ]

i added node using following command

rs.add(

{"_id": 19, "host": "mongo-db09.prod1.cloud.xxx.com:27017", "priority": 0,"hidden":true,"votes":0}

)

mongo on db07 and db08 crashed with the following error .. is this a known bug?

mongod(thread_proxy+0x80) [0x8b6a40]
/lib64/libpthread.so.0 [0x2ac0a96ec73d]
/lib64/libc.so.6(clone+0x6d) [0x2ac0aa166f6d]
Thu Apr 19 11:45:14 [initandlisten] connection accepted from 127.0.0.1:45671 #1548
Thu Apr 19 11:45:14 Invalid access at address: 0x90

Thu Apr 19 11:45:14 Got signal: 11 (Segmentation fault).

Thu Apr 19 11:45:14 Backtrace:
0x8a2e59 0x8a3430 0x2ac0a96f4b10 0x6d6052 0x6d0694 0x7dae1e 0x7dc741 0x642c25 0x6477c6 0x752225 0x757938 0x8a3b3e 0x8b6a40 0x2ac0a96ec73d 0x2ac0aa166f6d
mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8a2e59]
mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0x8a3430]
/lib64/libpthread.so.0 [0x2ac0a96f4b10]
mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x72) [0x6d6052]
mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x114) [0x6d0694]
mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x5ae) [0x7dae1e]
mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7dc741]
mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x642c25]
mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x2e86) [0x6477c6]
mongod [0x752225]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x757938]
mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a3b3e]
mongod(thread_proxy+0x80) [0x8b6a40]
/lib64/libpthread.so.0 [0x2ac0a96ec73d]
/lib64/libc.so.6(clone+0x6d) [0x2ac0aa166f6d]

Thu Apr 19 11:45:14 dbexit

Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

Oh, setting votes:0 is okay, then.

Comment by Harpreet Kaur [ 19/Apr/12 ]

i can update user api to not connect to these hosts which they are being synced. however how will voting work, since i already have 7 votes in repl-set.

Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

Don't change # of votes.

Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

Add hidden:true if you have client calling these commands. If you're on MMS, you might want to temporarily disable the agent, it automatically calls these commands.

Comment by Harpreet Kaur [ 19/Apr/12 ]

So to add db09 and db10 will the following configurations work?
priority 0,vote 0 - so they are just available for read requests. will this avoid "clients calling isMaster or replSetGetStatus on these nodes"

Comment by Harpreet Kaur [ 19/Apr/12 ]

clients connect though API, how can this be accomplished? change config for vote on these nodes ?

Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

The only workaround is to not have clients calling isMaster or replSetGetStatus on these nodes while reconfiguring.

Comment by Harpreet Kaur [ 19/Apr/12 ]

this is production and we are adding nodes to accomodate traffic which will take effect later today. We cann't update to 2.0, is there a work around to this issue?

I restarted both db07 and db08 one at a time and they are both available, no crash. What chnages to config , should i do to add db09 and db10, and not see this issue.

Comment by Harpreet Kaur [ 19/Apr/12 ]

this is production and we are adding nodes to accomodate traffic which
will take effect later today. We cann't update to 2.0, is there a work
around to this issue?

On Thu, Apr 19, 2012 at 7:52 AM, Kristina Chodorow (JIRA)

Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ]

This is a known bug that was fixed for 2.0.

You should be able to restart these servers (using appropriate recovery procedures) and they will pick up the new config.

Comment by Harpreet Kaur [ 19/Apr/12 ]

Exact error message ...

Thu Apr 19 06:00:17 [conn29] Assertion failure m db/repl/rs.cpp 155
0x54e8de 0x55f971 0x6bce15 0x6bd20a 0x69f772 0x6b3c5a 0x7dae1e 0x7dc741 0x642c25 0x6477c6 0x752225 0x757938 0x8a3b3e 0x8b6a40 0x2acf09ac573d 0x2acf0a53ff6d
mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54e8de]
mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x55f971]
mongod(ZN5mongo11ReplSetImpl17_fillIsMasterHostEPKNS_6MemberERSt6vectorISsSaISsEES7_S7+0x2c5) [0x6bce15]
mongod(_ZN5mongo11ReplSetImpl13_fillIsMasterERNS_14BSONObjBuilderE+0x27a) [0x6bd20a]
mongod(_ZN5mongo21appendReplicationInfoERNS_14BSONObjBuilderEbi+0x42) [0x69f772]
mongod(_ZN5mongo11CmdIsMaster3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x8a) [0x6b3c5a]
mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x5ae) [0x7dae1e]
mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7dc741]
mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x642c25]
mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x2e86) [0x6477c6]
mongod [0x752225]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x757938]
mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a3b3e]
mongod(thread_proxy+0x80) [0x8b6a40]
/lib64/libpthread.so.0 [0x2acf09ac573d]
/lib64/libc.so.6(clone+0x6d) [0x2acf0a53ff6d]

Generated at Thu Feb 08 03:09:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.