[SERVER-6959] 2.0.6 server crashed when movechunk failed because a config server was down Created: 06/Sep/12 Updated: 15/Feb/13 Resolved: 10/Sep/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Internal Code |
| Affects Version/s: | 2.0.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Mark N | Assignee: | Spencer Brody (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 10.4, mongo 2.0.6. 8 single replica set servers, 3 config servers, multiple mongos |
||
| Operating System: | ALL |
| Participants: |
| Description |
|
We have 8 servers as single replica sets. This is because we can lose the data at any time and its okay. We just start over. Its a caching system. 3 config servers We were moving a config server to another location. Right around the crash, we had lots of these because of the config server that was offline. Wed Sep 5 13:52:31 [conn31392] waiting till out of critical section Then Wed Sep 5 13:52:37 [conn31375] waiting till out of critical section
Wed Sep 5 14:07:45 [initandlisten] MongoDB starting : pid=17294 port=27017 dbpath=/var/lib/mongodb 64-bit hos Wed Sep 5 14:07:45 [initandlisten] journal dir=/var/lib/mongodb/journal and the recovery took place and it was fine. |
| Comments |
| Comment by Spencer Brody (Inactive) [ 10/Sep/12 ] |
|
Yes, this is expected behavior when a config server fails at a certain point in the middle of the migration. This means that the shard had updated its state to think the migration had been completed, but because the chunk data was never updated on the config server it detects an inconsistent state and shuts down. When the shard comes back online after a restart it reloads the chunk data from the config server and the migration is effectively reverted. To avoid errors like this in the future we recommend disabling the balancer before doing any maintenance on the config servers. |