[SERVER-5647] adding secondary caused existing few secondaries to crash with db/repl/rs.cpp 155 Created: 19/Apr/12 Updated: 15/Aug/12 Resolved: 19/Apr/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 1.8.1 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Harpreet Kaur | Assignee: | Kristina Chodorow (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | replicaset | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Production, EC2 |
||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
Have following configuration in Production db01-db06, added additional secondary db07 and db08, these new secondaries crash after a few minutes with the error message db/repl/rs.cpp 155. whats wrong with the configurations? i want to add additional db09 and db10 to the replicaset once this stabalizes. db01- vote 1,Priority 1, |
| Comments |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
Yes, same bug. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
i added node using following command rs.add( {"_id": 19, "host": "mongo-db09.prod1.cloud.xxx.com:27017", "priority": 0,"hidden":true,"votes":0}) mongo on db07 and db08 crashed with the following error .. is this a known bug? mongod(thread_proxy+0x80) [0x8b6a40] Thu Apr 19 11:45:14 Got signal: 11 (Segmentation fault). Thu Apr 19 11:45:14 Backtrace: Thu Apr 19 11:45:14 dbexit |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
Oh, setting votes:0 is okay, then. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
i can update user api to not connect to these hosts which they are being synced. however how will voting work, since i already have 7 votes in repl-set. |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
Don't change # of votes. |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
Add hidden:true if you have client calling these commands. If you're on MMS, you might want to temporarily disable the agent, it automatically calls these commands. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
So to add db09 and db10 will the following configurations work? |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
clients connect though API, how can this be accomplished? change config for vote on these nodes ? |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
The only workaround is to not have clients calling isMaster or replSetGetStatus on these nodes while reconfiguring. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
this is production and we are adding nodes to accomodate traffic which will take effect later today. We cann't update to 2.0, is there a work around to this issue? I restarted both db07 and db08 one at a time and they are both available, no crash. What chnages to config , should i do to add db09 and db10, and not see this issue. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
this is production and we are adding nodes to accomodate traffic which On Thu, Apr 19, 2012 at 7:52 AM, Kristina Chodorow (JIRA) |
| Comment by Kristina Chodorow (Inactive) [ 19/Apr/12 ] |
|
This is a known bug that was fixed for 2.0. You should be able to restart these servers (using appropriate recovery procedures) and they will pick up the new config. |
| Comment by Harpreet Kaur [ 19/Apr/12 ] |
|
Exact error message ... Thu Apr 19 06:00:17 [conn29] Assertion failure m db/repl/rs.cpp 155 |