[SERVER-29102] WiredTiger does not rotate journal log files Created: 08/May/17 Updated: 21/Jul/17 Resolved: 18/May/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Juan Antonio Roy Couto | Assignee: | Susan LoVerso |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
SO: Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-75-generic x86_64) |
||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Storage 2017-05-29 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
I have problems with the "/var/lib/mongodb/journal$" directory due to WiredTiger creates more than three files. More or less, it creates a new file per 30 minutes. The disk is going to be full and mongod will not be able to write any operations. Thank you very much for your help! |
| Comments |
| Comment by Juan Antonio Roy Couto [ 18/May/17 ] | |
|
I have eliminated these messages and everything looks fine. | |
| Comment by Juan Antonio Roy Couto [ 17/May/17 ] | |
|
Hi Bruce Lucas | |
| Comment by Bruce Lucas (Inactive) [ 17/May/17 ] | |
|
Hi Juan, The syslog that you uploaded shows frequent messages like this:
A google search for that message turned up this page with someone who had a similar problem and what they did to fix it. They were running under Windows Hyper-V, but even if that is not your case maybe this will provide a clue to fix your problem. Once you have eliminated the "Time has been changed" messages from syslog, please let us know if that resolves your other problems. | |
| Comment by Juan Antonio Roy Couto [ 17/May/17 ] | |
|
Hi, Sue LoVerso | |
| Comment by Susan LoVerso [ 17/May/17 ] | |
|
Can you describe the system you're running on? Is it a virtual machine? Or an AWS instance or some other cloud instance? | |
| Comment by Susan LoVerso [ 16/May/17 ] | |
|
juanroy Looking at this, the error paths I referred to earlier are correct. However one thing that could explain the values we see is if the system time went backward. Are you running some kind of time daemon process like ntpd or something similar? Could time be getting adjusted backward? | |
| Comment by Juan Antonio Roy Couto [ 16/May/17 ] | |
|
Sue LoVerso I do not have that system log but I have one of another day I issued the same problem. Perhaps It can help us. In it I can see: Apr 29 23:55:14 susanpre03 systemd[1]: mongod.service: Main process exited, code=killed, status=6/ABRT I've attached it! Thanks | |
| Comment by Susan LoVerso [ 16/May/17 ] | |
|
Looking more closely at this, I suspect that the wildly large values indicate disk errors (and a bug in the error path handling that doesn't stop some timers). For example I see that if the logging code performing an fsync gets an error it doesn't clear the timer. juanroy if you have access to system logs from this timeframe can you check if there are any errors reported? | |
| Comment by Susan LoVerso [ 08/May/17 ] | |
|
There are definitely some wild values in the metrics. Before I get to that, here's what I found:
| |
| Comment by Juan Antonio Roy Couto [ 08/May/17 ] | |
|
Hi! |