[SERVER-65330] Index builds should fail gracefully when running out of disk space Created: 07/Apr/22  Updated: 27/Oct/23  Resolved: 03/Jul/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Louis Williams Assignee: Yujin Kang Park
Resolution: Gone away Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Storage Execution
Sprint: Execution EMEA Team 2023-07-24
Participants:

 Description   

When an index build fails due to running out of disk space, it will cause the node to crash. We should attempt to handle this more gracefully.

If we get an ENOSPC from WiredTiger, we should drop any storage engine tables and external sort files immediately and then write an abortIndexBuild oplog entry to cancel the index build.

Alternatively, we could:

  • Proactively cancel the index build if we expect that we won't have any disk space to complete the index build
  • Expose metrics about expected disk space requirements and allow administrators to cancel index builds that are expected to exceed this limit.


 Comments   
Comment by Yujin Kang Park [ 03/Jul/23 ]

Closing, after PM-2994 index builds are proactively aborted when below gIndexBuildMinAvailableDiskSpaceMB.

Comment by Yujin Kang Park [ 03/Jul/23 ]

I would say yes, we now monitor the available disk space and proactively cancel index builds if it runs low. 

Comment by Gregory Noma [ 03/Jul/23 ]

yujin.kang@mongodb.com can we close this out now?

Generated at Thu Feb 08 06:02:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.