[SERVER-56424] improve index build invariant message for system error ENOSPC "28: No space left on device" Created: 28/Apr/21 Updated: 29/Oct/23 Resolved: 27/May/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.5 |
| Fix Version/s: | 5.0.0-rc3, 4.4.8, 5.1.0-rc0 |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Dmitry Agranat | Assignee: | Benety Goh |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | Atlas_Failure_Analysis | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Backport Requested: |
v5.0, v4.4
|
||||||||||||||||||||
| Sprint: | Execution Team 2021-05-31 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Currently we print this error in the log:
But if we know that the cause was "No space left on device", can we replace the "Unnexpected error code during index build cleanup: Locat" with "No space left on device"? |
| Comments |
| Comment by Vivian Ge (Inactive) [ 06/Oct/21 ] | |||
|
Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you! | |||
| Comment by Githook User [ 20/Jul/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 044ada4e8958efb1c8e045bb5a6e0702bb0686cf) | |||
| Comment by Githook User [ 20/Jul/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 46cd6f9251a1595e640fcdb1788329e520acf695) | |||
| Comment by Githook User [ 18/Jun/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 044ada4e8958efb1c8e045bb5a6e0702bb0686cf) | |||
| Comment by Githook User [ 18/Jun/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 46cd6f9251a1595e640fcdb1788329e520acf695) | |||
| Comment by Githook User [ 26/May/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: | |||
| Comment by Githook User [ 26/May/21 ] | |||
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: | |||
| Comment by Benety Goh [ 25/May/21 ] | |||
|
The new logs in 4.4 should look something like this if we decide to detect and trigger a fatal assertion on a out of disk space error:
| |||
| Comment by Benety Goh [ 24/May/21 ] | |||
|
The error code 16821 was added in | |||
| Comment by Benety Goh [ 24/May/21 ] | |||
|
We can probably try to map std::errc::no_space_on_device (ENOSPC) (ENOSPC) to ErrorCodes::OutOfDiskSpace. | |||
| Comment by Benety Goh [ 24/May/21 ] | |||
|
The index build error originated from a std::exception in the Sorter as it was persisting the in-memory state. This error would have an error code of 16821. Inserting into a WiredTiger table used for a MongoDB index may also fail because insufficient disk space errors - in this case the underlying OS error code and shadowed by a generic ErrorCodes::UnknownError. We actually have an existing named error OutOfDiskSpace that was introduced for repairDatabase in To improve the error message here in the logs, we could either:
Checking for the 28 errno code would probably not work on all platforms (Windows, for example). |