[SERVER-10623] Mongod fails to start as unprivileged user when {{/sys/devices/system/node}} is not readable and executable. Created: 26/Aug/13 Updated: 11/Jul/16 Resolved: 27/Sep/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability |
| Affects Version/s: | 2.4.6, 2.5.2 |
| Fix Version/s: | 2.5.3 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Hugues Lismonde | Assignee: | Kyle Erf |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
ArchLinux, kernel 3.10.9-xxxx-grs-ipv6-64 (OVH) x86_64, Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz |
||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | Linux | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
The function boost::filesystem::exists(boost::filesystem::path) throws an exception if some component of the path is not accessible by the executing user. This can cause the ProcessInfo::checkNumaEnabled() to throw an uncaught exception at start-up, when trying to determine if NUMA interleaving is enabled or disabled. Possible resolutions are to detect this in a different way that cannot fail, or to recover more gracefully when this fails. Original report follows.
|
| Comments |
| Comment by Matt Kangas [ 27/Jan/14 ] | ||
|
See follow-on ticket | ||
| Comment by Daniel Pasette (Inactive) [ 26/Jan/14 ] | ||
|
kyle.erf, as pointed out by statianzo, it looks like there is still an issue reading /sys/devices/system/node/node1: | ||
| Comment by Jason Staten [ 23/Dec/13 ] | ||
|
I'm still having a related issue to this in 2.5.4. Also running on an OVH kernel that /sys/devices/system/node is not readable/executable. On startup, the fatal message is logged:
While the fix solved processinfo_linux2.cpp's reading of /sys/devices/system/node/node1, there's another attempt to read it inside of startup_warnings.cpp that isn't wrapped with a try/catch. | ||
| Comment by auto [ 27/Sep/13 ] | ||
|
Author: {u'username': u'3rf', u'name': u'Kyle Erf', u'email': u'erf@mongodb.com'}Message: Signed-off-by: Matt Kangas <matt.kangas@mongodb.com> | ||
| Comment by Hugues Lismonde [ 27/Aug/13 ] | ||
|
Thanks for the workaround. It seems a lot of needed folders where unreadable under /sys. Mongod kept failing because it couldn't read /sys/dev/block/9:2/queue/read_ahead_kb. The error was this time much less "explosive":
was in the mongodb.log. Currently I've chosen the nuke approach and +rx'ed all folders under /sys. I'll have a look at the permissions on another server with the same mongodb version where it starts correctly. | ||
| Comment by Andy Schwerin [ 27/Aug/13 ] | ||
|
The problem is definitely that /sys/devices/system/node is not +rx for the user in question. Better behavior would be for mongod to issue a warning that it cannot detect if NUMA interleaving is enabled or disabled in this case, or to detect this by a means that is guaranteed to be readable by non-root users, if one exists. I will update the description of this ticket accordingly. As a workaround, is it reasonable to make said directory +rx in your deployment. | ||
| Comment by Hugues Lismonde [ 27/Aug/13 ] | ||
|
700 also, system and above are world readable/browseable. | ||
| Comment by Andy Schwerin [ 26/Aug/13 ] | ||
|
What are the permissions on /sys/devices/system/node? | ||
| Comment by Hugues Lismonde [ 26/Aug/13 ] | ||
|
Hi, mongod --version: ldd /usr/bin/mongod: /proc/self/numa_maps is root:root, 444 so this one should be readable but I don't have a node1 and /sys/devices/system/node/node0 is root:root, 700 | ||
| Comment by Andy Schwerin [ 26/Aug/13 ] | ||
|
What build of mongod and what version are you using? The most likely culprit is that one of /sys/devices/system/node/node1 and /proc/self/numa_maps, or one of the parent directories thereof, are not readable (or executable in the case of directories) by the user launching MongoDB. I'm a touch surprised by this specific behavior, though. Are you running a version of mongodb downloaded from mongodb.org, or one from the Arch packages? If the latter, it would be good to know the results of running "ldd" on the binary, and also to know the version of boost installed in /usr. If you downloaded the package from mongodb.org, the result of running mongod --version would be helpful. To confirm the hypothesis that it's an unreadable file, could you check the file permissions for the whole path to numa_maps and node1, described above? |