[SERVER-21460] WiredTiger primaries hold on to deleted collection data and index files Created: 13/Nov/15 Updated: 09/Dec/15 Resolved: 24/Nov/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Dai Shi | Assignee: | Kelsey Schubert |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
We are running v3.0.6 mongoD binaries. In one of our clusters, we dropped a large collection last week. I noticed today that all the primaries in the cluster had significantly higher disk utilization than the secondaries, so I tried stepping down a primary. Upon stepping down, the disk util immediately dropped to the level of the secondaries. The output of du also shows that after stepping down, a collection file along with several index files had been deleted. Note the mongoD process was not restarted at all. Before:
After:
Is this expected behavior? Do we need to step down all primaries after dropping collections or indexes to reclaim disk space? |
| Comments |
| Comment by Ramon Fernandez Marina [ 24/Nov/15 ] |
|
Understood – thanks dai@foursquare.com. We'll close this ticket for now. If the issue reappears please post here so we can reopen or feel free to open a new ticket. Regards, |
| Comment by Dai Shi [ 23/Nov/15 ] |
|
Apologies for the delay, I had to get our staging cluster on the same version with the right data. Unfortunately I was unable to reproduce the behavior in that environment. The collection and index files were deleted upon running the db.collection.drop(), and the disk space became free. I'm not sure why the primaries did not drop the files in production, but since neither of us can reproduce it we can close this for now. I will try and remember to pay attention to this the next time we drop anything in production. |
| Comment by Kelsey Schubert [ 23/Nov/15 ] |
|
Thank you for the additional details about this behavior. It would be great if you could reproduce on your staging cluster, preferably with log level 1. Please let us know whether are able to successfully reproduce so we can continue to investigate. |
| Comment by Dai Shi [ 18/Nov/15 ] |
|
Hi Thomas, Unfortunately the logs have already been rotated out so I no longer have them. I can try reproducing it on our staging cluster. For what it's worth, I saw this behavior on every single primary in this cluster (8 of them). Immediately after stepping them down, the collection and index files associated with the dropped collection were deleted, and disk space became free. |
| Comment by Kelsey Schubert [ 18/Nov/15 ] |
|
I haven't been able to reproduce this behavior. This issue may be a duplicate of Thank you, |