[SERVER-19101] ERROR: writer worker caught exception: :: caused by :: 10287 insertDocument :: caused by :: 10287 btree: key+recloc already in index on Created: 24/Jun/15 Updated: 25/Jun/15 Resolved: 24/Jun/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 2.6.1 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | rujun | Assignee: | Sam Kleinman (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
two members of a replica set can't restart after unclean shut down(power loss), we use version 2.6.1 at first, the primary power loss, then restart, we find logs :
after restart 10 times, we find logs :
after 30 minutes, a secondary power loss, then restart, we find logs :
finally, we clean up the dbpath of two members, and copy data from the other, this issue resolved. |
| Comments |
| Comment by Sam Kleinman (Inactive) [ 25/Jun/15 ] |
|
I just want to make it clear that I'm very sorry that you've encountered this issue, and I definitely understand it's severity and impact. Without more information its difficult to determine the exact source of the issue. If you can provide a reproduction script or instructions against the current stale release (i.e. 3.0.x) we can investigate. Issues like this; however, may also be a result of errors at the storage or networking layer are are difficult to isolate without the ability to reproduce the issue and on older versions. Using additional replica set members, and re-syncing when you encounter a data validity bug is the correct way to manage issues like this in the future. If you are able to reliably reproduce this error, we'd be more eager to revisit this issue and investigate that further. Sorry again that you encountered this issue. Regards, |
| Comment by rujun [ 25/Jun/15 ] |
|
Thank you Sam |
| Comment by Sam Kleinman (Inactive) [ 24/Jun/15 ] |
|
It looks as if this issue is caused by an invalid document in the oplog, which could be caused by the power failure or another issue. In general, the most expedient solution in this case is to resync the member as soon as you encounter a data validity issue like this. You may also consider increasing the number of replica set members to increase the amount of redundancy and allow the system to remain available during disruptions and maintenance.. Additionally, we recommend always running the latest release in your release stable (2.6.10 for 2.6). I hope this helps. Regards, |