[SERVER-22618] RangeDeleter asserts and causes primary to be unusable Created: 15/Feb/16 Updated: 03/Nov/16 Resolved: 08/Apr/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Yoni Douek | Assignee: | Andy Schwerin |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Sprint: | Sharding 11 (03/11/16), Sharding 12 (04/01/16), Sharding 13 (04/22/16) |
| Participants: |
| Description |
|
#8. Note: this looks like Our setup: primary+secondary+arbiter+another secondary which is >1 hour behind and syncing. in SECONDARY. we added it as votes:0 and priority:0 so it will not crash (as a workaround The other secondary is perfectly sync'd (0 sec lag). Range deleter shows this:
Note: this DOES NOT crash the node, it keeps living - but - after exactly 1 hour, the primary becomes unusable and very very slow. 1. This is a bug. 2. Why does the range deleter wait for the secondary if it has votes:0 and priority:0? 3. Any workaround for this? This will happen again. |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 03/Nov/16 ] | |||||||||||||||||
|
As Andy mentioned in the comment above, the assertion and stack trace are not the interesting issue here. But it does indicate that the w:2 write during the range deletion timed out. Which likely means you have replication lag in your system, and this log message is a symptom of a different problem, not the problem in itself. Thanks | |||||||||||||||||
| Comment by Viktor Gema [ 03/Nov/16 ] | |||||||||||||||||
|
Look like that issue happens in our cluster can we reopen that issue ?
| |||||||||||||||||
| Comment by Andy Schwerin [ 08/Apr/16 ] | |||||||||||||||||
|
I've done a bit of digging, and the stack trace is almost certainly not the cause of the primary becoming very very slow. In fact, there's really no reason to print the stack trace. It's acceptable for secondary throttle waits to time out, and while they cancel the current range deletion task, they do not by themselves destabilize or endanger the system. The two interesting questions are
The data available in the ticket are insufficient to answer either question, and we have not been able to reproduce the performance behavior in house. | |||||||||||||||||
| Comment by Yoni Douek [ 17/Mar/16 ] | |||||||||||||||||
|
Hey, This happened in our production servers and due to this bug (and others) - we are preventing lagging secondaries. We cannot dedicate the efforts to try to reproduce this on our test environment. I wouldn't close this issue if I were you. This is an assert, which needs to be rethinked, even if not reproduced. The description above should be enough to show the potential of what it does. | |||||||||||||||||
| Comment by Misha Tyulenev [ 16/Mar/16 ] | |||||||||||||||||
|
yoni@appsee.com , could you please clarify if the issue is still happening and if so could you please provide more info to help reproducing the problem? | |||||||||||||||||
| Comment by Misha Tyulenev [ 19/Feb/16 ] | |||||||||||||||||
|
Thanks for submitting the issue description. To the point 2 in the report: votes:0, priority:0 should not affect the replication sync behavior, because in this case moveChunk which is issued by balancer in migration thread should use writeConcern w:2 that will be satisfied if P and S1 are healthy. To confirm and identify the issue we need more info:
Thanks! |