[SERVER-33291] Mongo crashing every 20 mins - WiredTiger Error Created: 13/Feb/18 Updated: 21/Mar/18 Resolved: 20/Feb/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.19 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Anthony | Assignee: | Mark Agarunov |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
Hello, I figured I'd start by posting this as a question, perhaps later it can be upgraded to a bug. First some background information. I am running the v3.2.19 version of Mongo via docker as a DB for rocketchat which is also containerized. I have the mongodb data directory bind mounted into the container via docker. I have a node running with 8 gbs of ram on ubuntu 16.04. The database is using WiredTiger as it's backend. I have the rocketchat instance up and running, the db is writeable and everything seems to function fine. However roughly every 20 mins or so the mongo container crashes with the following output.
It seems obvious that this may be a permission issue, however I have verified that within the container the ownership both group and user of the data dir are mongodb and I can easily touch and delete files within as that user. I have noticed that the file WiredTiger.turtle.set does not exist in the directory. Not sure what else I should be poking at or trying to resolve this. Any pointers would be greatly appreciated. |
| Comments |
| Comment by Kelsey Schubert [ 20/Feb/18 ] |
|
Thanks for confirming that the issue is outside of mongodb, I hope you're able to track down the permissions issue. Kind regards, |
| Comment by Anthony [ 14/Feb/18 ] |
|
Okay, so I guess the permissions were just messed up. Outside of the container I have chowned all the files to docker:docker with a mode of 755. So far the database has been up for 17 hours with no crashes. I believe the issue was the ownership but I have made so many changes at this point, I cannot pin point exactly if it was the mode or ownership on the db files which fixed this. Thanks for your quick response Mark. Much appreciated. I'm off to test a bit more. Cheers |
| Comment by Anthony [ 13/Feb/18 ] |
|
Hello Mark, So the machine is a fresh box, we have no crons present on it at the moment. As for the 20 mins I should have mentioned that is an approximation, sometimes it stays up for 18 other times almost 30. The "Bind" mount in question is actually just a docker volume mount, we are mounting /opt/rocketchat/db to /data/db in the mongo container. Permission both inside and out are the same, I have tested writing files to /data/db within the container as the mongodb user with no issues. The underlying file system on the machine is ext4, I know it's not the recommended but I believe it is supported. I am continuing to test for permission issues. I fear it may be something obvious I am missing. I am currently testing 777 on all files under the db dir outside of the container, these permission will translate within as well. Waiting to see if it crashes again. |
| Comment by Mark Agarunov [ 13/Feb/18 ] |
|
Hello aeb123, Thank you for the report. To get a better idea of what may be causing this error, I'd like to request some additional information.
This should give some insight into why you're seeing the permission denied error. Thanks, |