[SERVER-23793] core dump on mongod Created: 19/Apr/16  Updated: 29/Apr/16  Resolved: 29/Apr/16

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 3.2.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Malie Assignee: Jonathan Reams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File core.rar    
Issue Links:
Related
related to SERVER-23902 Failing to create a thread should fai... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

1.I connect cfg servers through server A like this
server A:mongo admin -usiteRootAdmin -proot -host res_cfg/10.45.6.237:37017,10.45.6.237:37018,10.45.6.237:37019
2.and I kill -2 all the cfg servers then restart cfg servers

3.step 1 once more

4.cores appered(and I also found it in 3.0 version)

5.but I open another window to connect cfg servers it is normal

so I get confused whether it would happen in product env
and what happend to eht mongodb

(maybe his core can't reproduce,so I uploading the core file)

Sprint: Platforms 13 (04/22/16), Platforms 14 (05/13/16)
Participants:

 Description   

-bash-4.1$ mongo admin -usiteRootAdmin -proot -host res_cfg/10.45.6.237:37017,10.45.6.237:37018,10.45.6.237:37019

MongoDB shell version: 3.2.4
2016-04-19T15:54:46.950+0800 F -        [main] terminate() called. An exception is active; attempting to gather more information
2016-04-19T15:54:46.953+0800 F -        [main] std::exception::what(): 
Actual exception type: std::system_error
 
 0x986842 0x986172 0x340a2bcbe6 0x340a2bcc13 0x340a2bcd0e 0x340a26192c 0x104f0d8 0x8ab7df 0x8ab9e5 0x71dc10 0x722ebb 0x340461ecdd 0x71c285
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"586842","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"586172"},{"b":"340A200000","o":"BCBE6"},{"b":"340A200000","o":"BCC13"},{"b":"340A200000","o":"BCD0E"},{"b":"340A200000","o":"6192C","s":"_ZSt20__throw_system_errori"},{"b":"400000","o":"C4F0D8","s":"_ZNSt6thread15_M_start_threadESt10shared_ptrINS_10_Impl_baseEE"},{"b":"400000","o":"4AB7DF","s":"_ZN5mongo5mozjs17MozJSScriptEngineC1Ev"},{"b":"400000","o":"4AB9E5","s":"_ZN5mongo12ScriptEngine5setupEv"},{"b":"400000","o":"31DC10","s":"_Z5_mainiPPcS0_"},{"b":"400000","o":"322EBB","s":"main"},{"b":"3404600000","o":"1ECDD","s":"__libc_start_main"},{"b":"400000","o":"31C285"}],"processInfo":{ "mongodbVersion" : "3.2.4", "gitVersion" : "e2ee9ffcf9f5a94fad76802e28cc978718bb7a30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-358.el6.x86_64", "version" : "#1 SMP Tue Jan 29 11:47:41 EST 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EA522DA46A9516EC87C5DEC915A94F7F586CA2C8" }, { "b" : "7FFFD2CFF000", "elfType" : 3, "buildId" : "B5D86FBCF0CCB03331E6C7C73897B96845E0A4EB" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "5D2E35CD6D2B0BE8DC9281A5ED5500ED9CAD0B79" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "49A1DD932D95CD1D6FAD062F53B7521F6396DD56" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "5E9DDD9EE40AD0D4DDD032CE1086E402B7FA955A" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "15B0822C819020F18BBF0E0C0286373155E03BE2" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5864321409B6F3ED0CA97C4FD19A5FA1199F2D00" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "9984B1C6B5562DE3812A7AC7D9D8C204A1564540" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2E6E550A824EBC44AE5487B290A00923DB37761" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "14853815DD64F2B830B8DCCB3A958A3804E13EFC" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "397263907B14752BCF1258010B4527FF2EA2359E" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D6BD776B36DAC438642CF84B282956738727901D" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "AF7DB57AA4CA5C35AEEEFDB94CC1B97827C710FA" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "07BD094ED077DA56CDD76B8F562586745BA01326" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6A22EDFF4D4F04A57573E3D1536B6B4963159CD5" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "54FF9CD35F9E7E253F66C458DC902307190E0F80" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "129AD60521E5EF66722CA7C3DA6FC854DA5A8CDB" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "D454C19BE22DA338190029D28C1193600ED044EB" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "2D0F26E648D9661ABD83ED8B4BBE8F2CFA50393B" } ] }}
 mongo(_ZN5mongo15printStackTraceERSo+0x32) [0x986842]
 mongo(+0x586172) [0x986172]
 libstdc++.so.6(+0xBCBE6) [0x340a2bcbe6]
 libstdc++.so.6(+0xBCC13) [0x340a2bcc13]
 libstdc++.so.6(+0xBCD0E) [0x340a2bcd0e]
 libstdc++.so.6(_ZSt20__throw_system_errori+0x8C) [0x340a26192c]
 mongo(_ZNSt6thread15_M_start_threadESt10shared_ptrINS_10_Impl_baseEE+0x248) [0x104f0d8]
 mongo(_ZN5mongo5mozjs17MozJSScriptEngineC1Ev+0x21F) [0x8ab7df]
 mongo(_ZN5mongo12ScriptEngine5setupEv+0x35) [0x8ab9e5]
 mongo(_Z5_mainiPPcS0_+0x370) [0x71dc10]
 mongo(main+0x2B) [0x722ebb]
 libc.so.6(__libc_start_main+0xFD) [0x340461ecdd]
 mongo(+0x31C285) [0x71c285]
-----  END BACKTRACE  -----
Aborted (core dumped)



 Comments   
Comment by Jonathan Reams [ 29/Apr/16 ]

I agree that the error message wasn't clear at all, and I've opened SERVER-23902 to make sure there are better error messages for this problem. If you have any other issues you can open another ticket.

Comment by Malie [ 29/Apr/16 ]

I just think these error outputs can not let me know what happend I don't know how to analyze it ,it makes me confused。
I havec onnected to the mongodb tks

Comment by Jonathan Reams [ 22/Apr/16 ]

I've confirmed that we can repro this by setting ulimit -u to a very low number and trying to start the mongo shell. I'll open a ticket to handle std::thread creation failure better.

Comment by Jonathan Reams [ 22/Apr/16 ]

402309718@qq.com, This is failing because mongo is unable to start a new thread while initializing the javascript engine. The shell hasn't even gotten to the point where it could connect to a config server, so I don't think killing/restarting the config servers has anything to do with this.

What version/distro of Linux are you running? What is the output of ulimit -u on your system? Can you try raising the number of threads your user is allowed to start?

Comment by Jonathan Reams [ 22/Apr/16 ]

symbolized backtrace:

 /data/mci/src/src/mongo/util/stacktrace_posix.cpp:172  mongo::printStackTrace(std::ostream&)
 /data/mci/src/src/mongo/util/signal_handlers_synchronous.cpp:230  mongo::(anonymous namespace)::myTerminate()
 ??:0  _ZSt9has_facetISt8messagesIcEEbRKSt6locale@@GLIBCXX_3.4
 ??:0  _ZSt9has_facetISt8messagesIcEEbRKSt6locale@@GLIBCXX_3.4
 ??:0  _ZSt9has_facetISt8messagesIcEEbRKSt6locale@@GLIBCXX_3.4
 ??:0  _ZSt9has_facetISt8messagesIcEEbRKSt6locale@@GLIBCXX_3.4
 /data/mci/toolchain-builder/gccbuild/x86_64-mongodb-linux/libstdc++-v3/src/nonshared11/../../../../../gcc-4.8.2/libstdc++-v3/src/nonshared11/../c++11/thread.cc:146  std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>)
 /opt/mongodbtoolchain/include/c++/4.8.2/bits/shared_ptr_base.h:545  ~__shared_count
 /opt/mongodbtoolchain/include/c++/4.8.2/bits/shared_ptr_base.h:781  ~__shared_ptr
 /opt/mongodbtoolchain/include/c++/4.8.2/bits/shared_ptr.h:93  ~shared_ptr
 /opt/mongodbtoolchain/include/c++/4.8.2/thread:135  thread<void (mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>::*)(), mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>* const>
 /data/mci/src/src/mongo/scripting/deadline_monitor.h:74  DeadlineMonitor
 /data/mci/src/src/mongo/scripting/mozjs/engine.cpp:70  mongo::mozjs::MozJSScriptEngine::MozJSScriptEngine()
 /data/mci/src/src/mongo/scripting/mozjs/engine.cpp:56  mongo::ScriptEngine::setup()
 /data/mci/src/src/mongo/shell/dbshell.cpp:684  _main(int, char**, char**)
 /data/mci/src/src/mongo/shell/dbshell.cpp:938  main
 ??:0  _ZSt9has_facetISt8messagesIcEEbRKSt6locale@@GLIBCXX_3.4
 ??:0  _start

Generated at Thu Feb 08 04:04:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.