[SERVER-35785] mongod crash on startup when passing -f on directory instead of file Created: 25/Jun/18  Updated: 29/Oct/23  Resolved: 21/Sep/18

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: 4.1.4

Type: Bug Priority: Major - P3
Reporter: Norberto Fernando Rocha Leite (Inactive) Assignee: Annie Black
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Platforms 2018-07-30, Platforms 2018-08-13, Platforms 2018-08-27, Platforms 2018-09-10, Platforms 2018-09-24
Participants:

 Description   

I accidentally was trying to launch a mongod (4.0.0-rc7) passing a directory instead of file and got this stack trace in return.

 
mongod -f db/
tcmalloc: large alloc 9223372036854775808 bytes == (nil) @
2018-06-25T19:16:10.655+0000 F - [main] out of memory.
0x7f341bef10a1 0x7f341bef06d4 0x7f341bfdfb6b 0x7f341c082564 0x7f341bdb8183 0x7f341bdaca8f 0x7f341bdb5b3c 0x7f341a542111 0x7f341a53c933 0x7f341bea42a1 0x7f341bea4b12 0x7f341a53aad7 0x7f341a4ccc69 0x7f341685bec5 0x7f341a539d4f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F3419AB2000","o":"243F0A1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F3419AB2000","o":"243E6D4","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"7F3419AB2000","o":"252DB6B"},{"b":"7F3419AB2000","o":"25D0564","s":"_Znam"},{"b":"7F3419AB2000","o":"2306183","s":"_ZNSt6vectorIcSaIcEE17_M_default_appendEm"},{"b":"7F3419AB2000","o":"22FAA8F","s":"_ZN5mongo17optionenvironment13OptionsParser14readConfigFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPS7_"},{"b":"7F3419AB2000","o":"2303B3C","s":"_ZN5mongo17optionenvironment13OptionsParser3runERKNS0_13OptionSectionERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISB_EERKSt3mapISB_SB_St4lessISB_ESaISt4pairIKSB_SB_EEEPNS0_11EnvironmentE"},{"b":"7F3419AB2000","o":"A90111","s":"_ZN5mongo17optionenvironment46_mongoInitializerFunction_StartupOptions_ParseEPNS_18InitializerContextE"},{"b":"7F3419AB2000","o":"A8A933","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_18InitializerContextEEPS4_E9_M_invokeERKSt9_Any_dataOS3_"},{"b":"7F3419AB2000","o":"23F22A1","s":"_ZN5mongo11Initializer19executeInitializersERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EERKSt3mapIS7_S7_St4lessIS7_ESaISt4pairIKS7_S7_EEE"},{"b":"7F3419AB2000","o":"23F2B12","s":"_ZN5mongo21runGlobalInitializersEiPKPKcS3_"},{"b":"7F3419AB2000","o":"A88AD7","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"7F3419AB2000","o":"A1AC69","s":"main"},{"b":"7F341683A000","o":"21EC5","s":"__libc_start_main"},{"b":"7F3419AB2000","o":"A87D4F"}]}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f341bef10a1]
mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x7f341bef06d4]
mongod(+0x252DB6B) [0x7f341bfdfb6b]
mongod(_Znam+0x224) [0x7f341c082564]
mongod(_ZNSt6vectorIcSaIcEE17_M_default_appendEm+0x83) [0x7f341bdb8183]
mongod(_ZN5mongo17optionenvironment13OptionsParser14readConfigFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPS7_+0x25F) [0x7f341bdaca8f]
mongod(_ZN5mongo17optionenvironment13OptionsParser3runERKNS0_13OptionSectionERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISB_EERKSt3mapISB_SB_St4lessISB_ESaISt4pairIKSB_SB_EEEPNS0_11EnvironmentE+0x163C) [0x7f341bdb5b3c]
mongod(_ZN5mongo17optionenvironment46_mongoInitializerFunction_StartupOptions_ParseEPNS_18InitializerContextE+0x51) [0x7f341a542111]
mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_18InitializerContextEEPS4_E9_M_invokeERKSt9_Any_dataOS3_+0x23) [0x7f341a53c933]
mongod(_ZN5mongo11Initializer19executeInitializersERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EERKSt3mapIS7_S7_St4lessIS7_ESaISt4pairIKS7_S7_EEE+0x371) [0x7f341bea42a1]
mongod(_ZN5mongo21runGlobalInitializersEiPKPKcS3_+0x352) [0x7f341bea4b12]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xA7) [0x7f341a53aad7]
mongod(main+0x9) [0x7f341a4ccc69]
libc.so.6(__libc_start_main+0xF5) [0x7f341685bec5]
mongod(+0xA87D4F) [0x7f341a539d4f]
----- END BACKTRACE -----

This was caused by pilot error, nevertheless `mongod` probably should not error out this way.



 Comments   
Comment by Githook User [ 21/Sep/18 ]

Author:

{'name': 'ablack12', 'email': 'annie.black@10gen.com', 'username': 'ablack12'}

Message: SERVER-35785 fix readConfigFile to not crash with -f
Branch: master
https://github.com/mongodb/mongo/commit/44290425aebb3e6db398a55169c6c5864a48829c

Comment by ADAM Martin (Inactive) [ 25/Jul/18 ]

Addendum: BSD based systems (including OSX) have slightly different semantics from SYSV based systems (or those inspired by SYSV, like Linux), when it comes to file-style IO on a directory.  IIRC, BSD based systems are incredibly forgiving, and permit direct reading of the raw directory file's dirent encodings.

POSIX forbids `fopen` in write mode on a directory, issuing an errno: EISDIR.  However it is silent on read-only access.  Further, `open` in POSIX, which underlies `fopen`, is not forbidden on directories, when opening read-only

Comment by ADAM Martin (Inactive) [ 25/Jul/18 ]

The error text `"Error reading in config file:"` comes from later in the configuration parsing logic, after the memory allocation: https://github.com/mongodb/mongo/blob/ab0c426e60c4bdcc49b5a48a93f84828414d5ba6/src/mongo/util/options_parser/options_parser.cpp#L897

However, the nearly identical error text: `"Error reading config file:"` comes earlier in this logic: https://github.com/mongodb/mongo/blob/ab0c426e60c4bdcc49b5a48a93f84828414d5ba6/src/mongo/util/options_parser/options_parser.cpp#L859

Therefore, it appears that regardless of platform, a large amount of memory is requested.  Probably depending upon system over-provisioning options and other variables, the allocation may succeed or fail.  If the allocation succeeds (allocating -1 bytes, which becomes "absurdly large"), then the results of `ftell` are used and the -1 is detected because an absurdly large amount of data to read is sent to `fread`, thus causing it to fail here: https://github.com/mongodb/mongo/blob/ab0c426e60c4bdcc49b5a48a93f84828414d5ba6/src/mongo/util/options_parser/options_parser.cpp#L892

The check for a negative return from `ftell` should probably be moved earlier.  Also, considering the age of this file and a few other potential problems I see in the configuration file logic we may want to consider investing some time into improving this code.

Comment by Mark Benvenuto [ 25/Jun/18 ]

This does not repro for me on Fedora 28. It does repro on Ubuntu 16.04.

In this case, ftell() is returning a -1 indicating an error. This -1 is then used to allocate memory.

On Fedora 28 (Glibc 2.27), ftell returns 178.

Comment by Ramon Fernandez Marina [ 25/Jun/18 ]

I can reproduce this on Ubuntu 16.04 with 3.4.15 and 3.6.5, while in macOS I get:

$ mongod -f db/
Error reading in config file: Is a directory
try 'mongod --help' for more information

Generated at Thu Feb 08 04:40:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.