[SERVER-25718] 3.2.9 - mongos crash Created: 20/Aug/16  Updated: 26/Aug/16  Resolved: 26/Aug/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Deepak Shivamurthy Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

2016-08-19T18:20:48.197+0530 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-5-0] Failed to connect to mongo2:27018 - HostUnreachable: HostUnreachable
2016-08-19T18:20:48.197+0530 I NETWORK  [conn1732] end connection 10.140.179.44:54518 (706 connections now open)
2016-08-19T18:20:48.204+0530 E -        [conn2639] cannot open /dev/urandom Too many open files
2016-08-19T18:20:48.204+0530 I -        [conn2639] Fatal Assertion 28839
2016-08-19T18:20:48.204+0530 I -        [conn2639]
 
***aborting after fassert() failure
 
2016-08-19T18:20:48.225+0530 F -        [conn2639] Got signal: 6 (Aborted).
 
 0xc67e12 0xc66f69 0xc67772 0x7fb6a8210330 0x7fb6a7e71c37 0x7fb6a7e75028 0xbec4f2 0xa41d63 0x7c6d03 0x7c801f 0x7a04db 0x7bb6a7 0x7bd608 0xb9d2c4 0xb9df8d 0xbac53d 0xb9c702 0x6b3af5 0xc13a55 0x7fb6a8208184 0x7fb6a7f3537d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"867E12","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"866F69"},{"b":"400000","o":"867772"},{"b":"7FB6A8200000","o":"10330"},{"b":"7FB6A7E3B000","o":"36C37","s":"gsignal"},{"b":"7FB6A7E3B000","o":"3A028","s":"abort"},{"b":"400000","o":"7EC4F2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"641D63","s":"_ZN5mongo12SecureRandom6createEv"},{"b":"400000","o":"3C6D03","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs"},{"b":"400000","o":"3C801F","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs"},{"b":"400000","o":"3A04DB","s":"_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs"},{"b":"400000","o":"3BB6A7"},{"b":"400000","o":"3BD608"},{"b":"400000","o":"79D2C4","s":"_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"400000","o":"79DF8D","s":"_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi"},{"b":"400000","o":"7AC53D","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"400000","o":"79C702","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"400000","o":"2B3AF5","s":"_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"813A55","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7FB6A8200000","o":"8184"},{"b":"7FB6A7E3B000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-93-generic", "version" : "#140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "8EC7078A4D025A1355697A1C8A38BB9DD26D9E2F" }, { "b" : "7FFD936B7000", "elfType" : 3, "buildId" : "B4264B818A44D0E9AB7318174A3636E7D8F65D5D" }, { "b" : "7FB6A9122000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7FB6A8D46000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7FB6A8B3E000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7FB6A893A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7FB6A8634000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7FB6A841E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7FB6A8200000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7FB6A7E3B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7FB6A9381000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc67e12]
 mongos(+0x866F69) [0xc66f69]
 mongos(+0x867772) [0xc67772]
 libpthread.so.0(+0x10330) [0x7fb6a8210330]
 libc.so.6(gsignal+0x37) [0x7fb6a7e71c37]
 libc.so.6(abort+0x148) [0x7fb6a7e75028]
 mongos(_ZN5mongo13fassertFailedEi+0x82) [0xbec4f2]
 mongos(_ZN5mongo12SecureRandom6createEv+0x223) [0xa41d63]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs+0x17F3) [0x7c6d03]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs+0x34F) [0x7c801f]
 mongos(_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs+0x2B) [0x7a04db]
 mongos(+0x3BB6A7) [0x7bb6a7]
 mongos(+0x3BD608) [0x7bd608]
 mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x6E4) [0xb9d2c4]
 mongos(_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x2ED) [0xb9df8d]
 mongos(_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE+0x19D) [0xbac53d]
 mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x672) [0xb9c702]
 mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x65) [0x6b3af5]
 mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0xc13a55]
 libpthread.so.0(+0x8184) [0x7fb6a8208184]
 libc.so.6(clone+0x6D) [0x7fb6a7f3537d]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 26/Aug/16 ]

Hi shivamurthy.deepak@gmail.com,

Thanks for the information. Please set your ulimit according to our recommendations.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Deepak Shivamurthy [ 23/Aug/16 ]

Just to add that, crashed only once. After that its working fine. Only CPU usage is high on mongod instances.

Comment by Deepak Shivamurthy [ 23/Aug/16 ]

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128219
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 128219
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Comment by Kelsey Schubert [ 23/Aug/16 ]

Hi shivamurthy.deepak@gmail.com,

Can you please provide output of ulimit -a as the user that runs MongoDB?

Thank you,
Thomas

Comment by Deepak Shivamurthy [ 20/Aug/16 ]

mongos crashed out and above log is from mongod instance.

Generated at Thu Feb 08 04:10:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.