[SERVER-22117] WiredTiger journal files not deleted/ Way too many journal files Created: 11/Jan/16 Updated: 06/Dec/22 Resolved: 30/Mar/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.5, 3.3.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Maciej Galkowski | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | WTplaybook, code-and-test | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Storage Execution
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Completed: | |||||||||
| Participants: | |||||||||
| Description |
|
Hi, We did notice that the WiredTiger journal files are not deleted and just left in the /journal dir when we are pushing the data. From my understanding of the documentation the journal files would be flushed frequently, not left on the disk indefinitely. We are using MongoDB 3.2 on FreeBSD 10.1. The files are on zfs, with atime disabled. We are using zlib on both journal files and data files, but we did see the same issue with snappy. Currently we have ~520 journal files, and growing every second. They are taking 16GB of space at the moment. Is is normal? I believe it is a bug, there should be a way to limit the number of journal files. |
| Comments |
| Comment by Rob Offer [ 01/Apr/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you, I have downgraded for now. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 01/Apr/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you for the report. Generally it's preferable to open a new ticket – that way we can keep investigations of different issues from getting muddled. In this case, I believe you're encountering Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Rob Offer [ 01/Apr/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Apologies if posting on this ticket is not the right place, but we are seeing this problem in production. We recently upgraded to 3.6.3 We are running a replica set and one server has already filled up, the other is starting to run out of space. We are looking at approaching 100 GB. We are running on WIndows in Azure, if there is any info I can provide then please let me know. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 02/May/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
HI juanroy, This bug was resolved during the development of MongoDB 3.4 and all versions of MongoDB 3.4.x contain this fix. The issue you describe is likely related to
In addition, we may need to inspect some WiredTiger journal files. Thank you, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Juan Antonio Roy Couto [ 02/May/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello, @michael, @thomas. I have had this issue. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 30/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fixed by latest merge of WiredTiger, see | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 11/Jan/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi mgalkowski, Thank you for uploading the diagnostic data. We have identified this as a known issue in the WiredTiger storage engine: Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Maciej Galkowski [ 11/Jan/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Attaching diagnostics file from today. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 11/Jan/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi mgalkowski, Can you please archive the $dbpath/diagnostic.data directory and attach it to this ticket? These files contain periodically collected serverStatus data, which will help us to identify what is happening here. Thank you, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Maciej Galkowski [ 11/Jan/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
UPDATE : | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Maciej Galkowski [ 11/Jan/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I just checked the procstat result :
Only one journal time is open at a time, apparently. |