[SERVER-25986] Failed chunk moves should not leave behind files on disk Created: 07/Sep/16 Updated: 06/Dec/22 Resolved: 17/Dec/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dai Shi | Assignee: | [DO NOT USE] Backlog - Sharding EMEA |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Sharding EMEA
|
| Participants: |
| Description |
|
When a chunk move fails, mongo leaves behind files named "preCleanup.$timestamp.bson" inside the $DBPATH/moveChunk. Often times, the reason why the chunk move fails is not a transient condition, causing the move to fail again if attempted, until the root cause is fixed. When the balancer is enabled, it will choose to move the same chunk to the same destination over and over, failing each time, causing these preCleanup files to be placed on disk and never getting reaped. Over a short period of time (say, a day), this can easily use up all of the available inodes on that filesystem. We had this happen over the weekend, and once all the inodes are used, the mongoD will exit and will fail to restart until there are available inodes again. This seems like non ideal behavior, and I think it would be much better if the preCleanup files would also get cleaned up after a failed chunk move instead of allowing them to accumulate on disk. |
| Comments |
| Comment by Kaloian Manassiev [ 17/Dec/21 ] |
|
We will not invest time in improving the moveParanoia so closing as Won't Do. |
| Comment by Kelsey Schubert [ 09/Sep/16 ] |
|
Thanks for confirming that you want to to keep moveParanoia enabled. I'm marking this ticket to be considered by our Sharding team. Please continue to watch for updates. Kind regards, |
| Comment by Dai Shi [ 09/Sep/16 ] |
|
Thomas, So, I realize that turning off moveParanoia will stop these files from being created. However, we do not want to do that, as the post-cleanup files are still useful in recovering data in cases where we suspect data loss has occurred (most recently due to this bug: The behavior that we would like to see is that if the chunk move failed, it should clean up any of the preCleanup files created. If the chunk move succeeded, then it should preserve the post-cleanup files. |
| Comment by Kelsey Schubert [ 09/Sep/16 ] |
|
Thanks for reporting this behavior. As you likely know, these files not written by default in MongoDB 3.2. I would recommend turning off moveParanoia to address this issue. Best regards, |