[SERVER-21176] Mongo not starting after upgrading to 3.0.7: extraneous permissions in journal file or group ownership problem Created: 27/Oct/15 Updated: 28/Oct/15 Resolved: 28/Oct/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.7 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Teemu Sirkiä | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
||
| Participants: |
| Description |
|
I'm using MongoDB 3.0.6 with WiredTiger. I tried to update the mongodb-org-server package in Ubuntu 14.04 to version 3.0.7. Installation completes but after adding engine: wiredShark to the config file, the server won't start as a daemon. It complains:
Starting MongoDB in terminal with command "sudo mongod --config /etc/mongod.conf" everything works. All file permissions should be OK, files and directories belong to mongodb:mongodb. As I wasn't able to run MongoDB as a daemon anymore, I had to rollback to 3.0.6 and now everything works again. What might be the problem here? |
| Comments |
| Comment by Teemu Sirkiä [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you for your help! Yes, it remains as a mystery what happened directly after apt-get install, why the daemon failed to start after editing the mongod.conf. Of course it tried to start before that but that attempt failed because there were those WiredTiger files in the db directory.
Maybe that attempt made something that caused problems after the correct engine was set. I think this problem was originally caused because the 3.0.6 used INI file configuration and 3.0.7 YAML. I selected to ovewrite the existing config file which caused this attempt to start with the wrong database engine. But if anyone else gets this same problem, it is very good that this ticket is here and it contains a hint how to resolve the situation. Thanks again! BR, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for running the experiment ttsirkia, much appreciated. The results confirm the group ownership is not a real issue for users upgrading to 3.0.7. When mongod was run as root the WiredTigerLog.0000000704 journal file was used for recovery and later removed, so when 3.0.6 was started that file was no longer there. Given that your 3.0.7 installation is up and running, and since we don't know what permissions that file was created with or how this happened we're going to close this ticket. If this happens again I'd suggest running ls -laR /var/lib/mongodb before attempting anything else. Thanks again for the help with the troubleshooting. Regards, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Teemu Sirkiä [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, it still works after changing the group to nogroup. I cannot anymore investigate how the permissions were yesterday. As I wrote, mongod was running a short time as root because that was the only way to get it running. But before I tested that, it was running (or tried to run) as a daemon and it didn't work. Instantly after downgrading to 3.0.6, the daemon started to work without changing any permissions. If there was I file with wrong permissions, why 3.0.6 started after downgrading? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for the quick feedback ttsirkia, and glad to hear you got 3.0.7 running. Group membership should not be an issue here, so I'm wondering if the /var/lib/mongodb/journal/WiredTigerLog.0000000704 file was created by a mongod running as root at some point. I'd like to ask you to try the following:
and see if mongod starts. If it doesn't then I would argue you've found a bug regarding group ownership and we should investigate it. Could you please try the experiment above so we can determine if this is a bug in the server? Thanks, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Teemu Sirkiä [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for the tip. This seems to work! Maybe there might be mention about this in the release notes? This is quite nasty thing that it won't start after the upgrade process without changing permissions. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 28/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for uploading the requested information ttsirkia. This may be related to Could you please try to chgrp -R mongodb /var/lib/mongodb and fire up 3.0.7 again? Thanks, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Teemu Sirkiä [ 27/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The config file was that I tried with the version 3.0.7. The directory listing is the current listing in use with the version 3.0.6. BR, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Teemu Sirkiä [ 27/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 27/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi ttsirkia, can you please send us the content of your /etc/mongod.conf file? The fact that you can run as root seems to indicate a permission problem, so we should rule that out first. Can you also send the output of ls -laR /var/lib/mongodb? Thanks, |