[SERVER-35124] Stepdown suites with MMAP V1 often fail due to `flushing mmaps` taking long time Created: 21/May/18 Updated: 29/Oct/23 Resolved: 23/Jun/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.6.6, 4.0.1, 4.1.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Kaloian Manassiev | Assignee: | Max Hirschhorn |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v4.0, v3.6
|
||||||||
| Sprint: | TIG 2018-07-02 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 15 | ||||||||
| Description |
|
On some occasions, right around the time when the ContinousStepdownThread issued a replSetStepdown command, the MMAP V1 storage engine started flushing the file mappings, which took 13 seconds. Since this happens under a lock, the stepdown thread was not able to acquire the global S lock for 10 seconds and failed. Here are the relevant logs:
|
| Comments |
| Comment by Githook User [ 03/Jul/18 ] |
|
Author: {'username': 'visemet', 'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com'}Message: It is possible for a database operation to prevent the global X lock (cherry picked from commit d7ed31017007fd5963390247e6ae68714cb6a61c) |
| Comment by Githook User [ 03/Jul/18 ] |
|
Author: {'username': 'visemet', 'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com'}Message: It is possible for a database operation to prevent the global X lock (cherry picked from commit d7ed31017007fd5963390247e6ae68714cb6a61c) |
| Comment by Githook User [ 23/Jun/18 ] |
|
Author: {'username': 'visemet', 'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com'}Message: It is possible for a database operation to prevent the global X lock |
| Comment by Judah Schvimer [ 11/Jun/18 ] |
|
If we know when we want to be electable again, we could use the replSetFreeze: 0 command to make it electable again. |
| Comment by Eric Milkie [ 11/Jun/18 ] |
|
You're right, I forgot I already filed a ticket to fix this: |
| Comment by Max Hirschhorn [ 11/Jun/18 ] |
milkie, I don't think so. The linked BF ticket failed due to not being able to acquire the global X lock, which is controlled by the stepDownForSecs value associated with the command name itself. |
| Comment by Eric Milkie [ 11/Jun/18 ] |
|
I thought the replSetStepDown command took two parameters. The parameter for the command name itself is only the freeze time after stepdown. The parameter named "secondaryCatchUpPeriodSecs" is optional and if not provided, defaults to 10 seconds. |
| Comment by Max Hirschhorn [ 09/Jun/18 ] |
The downside of changing to run the {replSetStepDown: 30} command instead of the {replSetStepDown: 10} command is that while the primary will wait up to 30 seconds to acquire the global X lock, it will also be unelectable for 30 seconds after stepping down. While the Python version of the StepdownThread uses the "replSetStepUp" command to restore write available more quickly, it is best-effort and may not succeed. I'd rather change the StepdownThread to tolerate the ExceededTimeLimit error response from the "replSetStepDown" command by swallowing the pymongo.errors.ExecutionTimeout exception. CC judah.schvimer, spencer |