[SERVER-9032] mongod fails when launched with misconfigured locale Created: 20/Mar/13  Updated: 14/Feb/17  Resolved: 17/Jun/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.4.0
Fix Version/s: 2.6.5, 2.7.3

Type: Bug Priority: Minor - P4
Reporter: Nick Demyanchuk Assignee: Mark Benvenuto
Resolution: Done Votes: 2
Labels: cap-ticket-needed, community-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: ubuntu 12.04 64 bit
python 2.7.3


Issue Links:
Depends
Duplicate
is duplicated by SERVER-11909 Stacktrace at start of mongod with re... Closed
is duplicated by SERVER-9219 init script is not working after upgr... Closed
is duplicated by SERVER-10563 Missing/invalid LOCALE on Solaris res... Closed
is duplicated by SERVER-15011 Segfault in 'mongos --upgrade' if loc... Closed
is duplicated by SERVER-15497 Missing locale causing exit code 14 w... Closed
Related
is related to SERVER-15030 Backport SERVER-9032 to the 2.6 branch Closed
Tested
Operating System: Linux
Backport Completed:
Steps To Reproduce:

in python:

import subprocess
 
subprocess.Popen(('mongod', '--fork', '--config', '/etc/mongodb.conf'))

Sprint: Server 2.7.3
Participants:

 Description   
Issue Status as of Sep 02, 2014

ISSUE SUMMARY
On linux systems, if the user running mongod does not have a locale set or the locale is misconfigured, mongod fails to start printing a stack trace.

USER IMPACT
mongod can't be started with a missing or misconfigured locale.

WORKAROUNDS
Configure a locale for the user running mongod.

AFFECTED VERSIONS
MongoDB production releases up to 2.6.4 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.5 production release.

RESOLUTION DETAILS
Validate that the user running mongod has a locale set. If a locale is not set, print a useful error message and exit.

Original description

Hi guys!

I'm trying to start mongod process from python script, using subprocess module, and this is what mongod crashes with:

about to fork child process, waiting until server is ready for connections.
forked process: 2462
Wed Mar 20 13:43:27.436 terminate() called, printing stack (if implemented for platform):
0xdcae01 0x6cd72e 0x7faab9b23846 0x7faab9b23873 0x7faab9b2396e 0x7faab9ad0a07 0x7faab9ae0284 0x7faab9ad4a74 0x7faab9ad565f 0xde86df 0xde902b 0xde5cd0 0x9e63ff 0x6dba43 0x6dd7e9 0x7faab91be76d 0x6cd519 
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01]
 mongod(_ZN5mongo11myterminateEv+0x3e) [0x6cd72e]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7faab9b23846]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7faab9b23873]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7faab9b2396e]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt21__throw_runtime_errorPKc+0x57) [0x7faab9ad0a07]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x72284) [0x7faab9ae0284]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt6locale5_ImplC1EPKcm+0x44) [0x7faab9ad4a74]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt6localeC1EPKc+0x22f) [0x7faab9ad565f]
 mongod(_ZN5boost11filesystem34path21wchar_t_codecvt_facetEv+0x4f) [0xde86df]
 mongod(_ZNK5boost11filesystem34path14root_directoryEv+0xbb) [0xde902b]
 mongod(_ZN5boost11filesystem38absoluteERKNS0_4pathES3_+0x40) [0xde5cd0]
 mongod(_ZN5mongo27initializeServerGlobalStateEb+0x15f) [0x9e63ff]
 mongod() [0x6dba43]
 mongod(main+0x9) [0x6dd7e9]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7faab91be76d]
 mongod(__gxx_personality_v0+0x499) [0x6cd519]
Wed Mar 20 13:43:27.531 Got signal: 6 (Aborted).
 
Wed Mar 20 13:43:27.534 Backtrace:
0xdcae01 0x6ce879 0x7faab91d34a0 0x7faab91d3425 0x7faab91d6b8b 0x6cd733 0x7faab9b23846 0x7faab9b23873 0x7faab9b2396e 0x7faab9ad0a07 0x7faab9ae0284 0x7faab9ad4a74 0x7faab9ad565f 0xde86df 0xde902b 0xde5cd0 0x9e63ff 0x6dba43 0x6dd7e9 0x7faab91be76d 
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01]
 mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6ce879]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7faab91d34a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7faab91d3425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7faab91d6b8b]
 mongod(_ZN5mongo11myterminateEv+0x43) [0x6cd733]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7faab9b23846]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7faab9b23873]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7faab9b2396e]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt21__throw_runtime_errorPKc+0x57) [0x7faab9ad0a07]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x72284) [0x7faab9ae0284]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt6locale5_ImplC1EPKcm+0x44) [0x7faab9ad4a74]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt6localeC1EPKc+0x22f) [0x7faab9ad565f]
 mongod(_ZN5boost11filesystem34path21wchar_t_codecvt_facetEv+0x4f) [0xde86df]
 mongod(_ZNK5boost11filesystem34path14root_directoryEv+0xbb) [0xde902b]
 mongod(_ZN5boost11filesystem38absoluteERKNS0_4pathES3_+0x40) [0xde5cd0]
 mongod(_ZN5mongo27initializeServerGlobalStateEb+0x15f) [0x9e63ff]
 mongod() [0x6dba43]
 mongod(main+0x9) [0x6dd7e9]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7faab91be76d]
 
ERROR: child process failed, exited with error number 14 <err>:  <args>: ('/usr/bin/sudo', '-u', 'mongodb', 'mongod', '--fork', '--config=/etc/mongodb.shardsrv.conf', '-vv')

I've tried all combinations of arguments and subprocess flags, and i still getting the same error. If i run mongod with the same args from shell - it starts, and i don't get no errors.

It worked with previous versions of mongo (2.0, 2.2), but it doesn't in 2.4.



 Comments   
Comment by Akira Kurogane [ 07/Feb/17 ]

Update Feb 2017:- it seems setting an invalid or empty locale no longer causes the problem as of 3.4.

Versions 3.0 and 3.2 use boost 1.56 (i.e. newer than the boost bug #7388 fix listed earlier) but the error is still present. That is the following code from here still throws an exception:

boost::filesystem::path("/").has_root_directory();

3.0 - Fails

akira:tmp$ mongod --version
db version v3.0.12
git version: 33934938e0e95d534cebbaff656cde916b9c3573
akira:tmp$ LANG=invalidrubbish mongod --fork --dbpath data --logpath data/mongod.log
2017-02-08T09:06:36.522+1100 F CONTROL  Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.

3.2 - Fails

akira:~$ mongod --version
db version v3.2.12-15-gc38a3a127d
...
akira:~$ cd /tmp/
akira:tmp$ mkdir /tmp/data
akira:tmp$ LANG=invalidrubbish mongod --fork --dbpath data --logpath data/mongod.log
2017-02-15T10:42:43.282+1100 F CONTROL  [main] Failed global initialization: BadValue: Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.

MongoDB 3.4 is OK. This version uses boost 1.60.

3.4 - Succeeds

akira:tmp$ mongod --version
db version v3.4.1-126-gdd491eeb54
...
akira:tmp$ LANG=invalidrubbish mongod --fork --dbpath data --logpath data/mongod.log
about to fork child process, waiting until server is ready for connections.
forked process: 19399
child process started successfully, parent exiting

Comment by Githook User [ 15/Sep/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-9032: Validate locale on startup

On some Linux machines, users may have not have a locale set which will
cause boost to fail. Validate that the user has a locale set, and if not,
give a useful error message, and exit.

(cherry picked from commit ff70eb640552c049538a03d174edbc2c0311c8bd)
Branch: v2.6
https://github.com/mongodb/mongo/commit/f46ea38b7f63987e7041a4c07ce8eea3b7b46f71

Comment by Githook User [ 17/Jun/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-9032: Validate locale on startup

On some Linux machines, users may have not have a locale set which will
cause boost to fail. Validate that the user has a locale set, and if not,
give a useful error message, and exit.
Branch: master
https://github.com/mongodb/mongo/commit/ff70eb640552c049538a03d174edbc2c0311c8bd

Comment by Mark Benvenuto [ 28/May/14 ]

Using SuSE Linux Enterprise Server 11 sp3 (PV) - ami-e8084981 (64-bit) (SuSE Linux Enterprise Server 11 Service Pack 3 (PV), EBS-backed with Amazon EC2 AMI Tools preinstalled; Apache 2.2, MySQL 5.5, PHP 5.3, and Ruby 1.8.7 available), I see

LANG=POSIX
LC_CTYPE=en_US.UTF-8

This was using GNU bash, version 3.2.51(1)-release (x86_64-suse-linux-gnu). I see that LC_CTYPE is set in /etc/profile.d/lang.sh via /etc/profile via /etc/bashrc. I also verified MongoDB 2.4.10 starts with just this single variable set.

Comment by Luis Barrios [ 19/May/14 ]

This bug also appears with MongoDB 2.4.10 in SuSE Linux Enterprise 11 SP3. The locale is not configured by default.

In order to avoid the issue, you must define the following variables in mongodb user's .bashrc file:
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Comment by Michael Grundy [ 10/Apr/13 ]

Easy repro steps for Ubuntu (from Ernie):

LANG=crash mongod --logpath $(pwd)/mongod.log

Comment by Michael Grundy [ 26/Mar/13 ]

Hi Nick -

Thanks for the confirmation! I'm not sure why it doesn't fall back. Like you mentioned in 2.2 this wasn't a failure. This is specific to the logpath option. dbpath is handled differently and will print a message if there is a locale issue (and logpath wasn't specified, so it got past this issue). We should handle misconfigurations that affect logpath similarly.

Thanks!
Mike

Comment by Nick Demyanchuk [ 26/Mar/13 ]

Michael, you are absolutely right, i was trying to start mongod with nonexistent locale.

BTW, why mongo just don't fall back to C locale in this case?

Comment by Michael Grundy [ 22/Mar/13 ]

Hi Nick -

Does your python script set a locale in it? I've seen this in an environment where the locale settings have uninstalled entries. Could you post the output of locale, locale -a, and any locale.setlocale() calls that are in your script (or possibly a module you are using).

Thanks!
Mike

Comment by Eliot Horowitz (Inactive) [ 20/Mar/13 ]

What directory are you running this from?
Try / if you're not already

Generated at Thu Feb 08 03:19:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.