[SERVER-18314] Stall during fdatasync phase of checkpoints under WiredTiger and EXT4 Created: 04/May/15 Updated: 09/Nov/23 Resolved: 19/Jun/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Ramon Fernandez Marina |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | WTplaybook, wttt | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||
| Description |
During each checkpoint two calls to fdatasync are made. Because this scenario is i/o constrained the fdatasyncs take a substantial amount of time, and during both fdatasync calls throughput falls to exactly 0 for the duration of the fdatasync. This is seen in A-B, C-D, E-F, G-H, I-J, K-L below. In many, but not all, such cases WT bumps the "eviction server unable to reach goal" counter.
Similar test with a larger cache (the default 64GB) does not show this issue. Note: this is the same test as reported in |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 19/Jun/15 ] | ||||||||||||||||||||
|
Resolving this issue as a duplicate of | ||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 19/Jun/15 ] | ||||||||||||||||||||
|
Wrote a standalone C program that does writes on some number of threads while doing an fdatasync, and demonstrated that the cause of the stalls in mongod was that writes to the file can block for the duration of the fdatasync. Also observed that the writes would not necessarily stall immediately after the start of the fdatasync, but might only begin to stall after a few seconds. Using perf to probe context switch events was able to determine that this is the point where the writes block:
So it appears that the stall is related to ext4 journaling - apparently journaling is blocked while the fdatasync is in progress. The delay in stalling mentioned above is explained by the periodic journal commits: the writer thread is able to continue writing during fdatasync until the next journal commit; the default commit interval is 5 seconds, and indeed observed that the writes might be able to continue up to 5 seconds after the start of the fdatasync. Since the stalls occur in ext4-specific code, and are related to periodic journal commits, we can avoid the stalls, both in the test program and in mongod, by either
| ||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 13/May/15 ] | ||||||||||||||||||||
|
michael.cahill, tried master as of yesterday's nightly build.
3.0.2
MASTER, EXT4
MASTER, EXT4, BUT DISABLE EXT4 TIMED JOURNAL COMMITS (-o commit=... option)
MASTER, XFS
|