[SERVER-26995] mongos crashed with cannot open /dev/urandom Too many open files Created: 11/Nov/16  Updated: 04/Dec/16  Resolved: 04/Dec/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: birdylee_cn Assignee: Kelsey Schubert
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-25659 InputStreamSecureRandom should open t... Closed
is duplicated by SERVER-26996 mongos crashed with cannot open /dev/... Closed
Operating System: Linux
Participants:

 Description   

system info

[root@mongodb1 logs]# uname -a
Linux mongodb1 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

system limit info

[root@mongodb1 logs]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 61292
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 61292
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

crashed info

2016-11-11T20:20:26.324+0800 E -        [conn39211] cannot open /dev/urandom Too many open files
2016-11-11T20:20:26.324+0800 I -        [conn39211] Fatal Assertion 28839
2016-11-11T20:20:26.324+0800 I -        [conn39211]
 
***aborting after fassert() failure
 
 
2016-11-11T20:20:26.423+0800 F -        [conn39211] Got signal: 6 (Aborted).
 
 0xc71ed2 0xc70e09 0xc71612 0x7fe0535f3790 0x7fe053282625 0x7fe053283e05 0xbf74b2 0xa43703 0x7b14b3 0x7b27cf 0x7896db 0x7a55a7 0x7a7508 0xba67f4 0xba74bd 0xbb5c3d 0xba5bf2 0x697f55 0xc1bf95 0x7fe0535eba51 0x7fe05333893d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"871ED2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"870E09"},{"b":"400000","o":"871612"},{"b":"7FE0535E4000","o":"F790"},{"b":"7FE053250000","o":"32625","s":"gsignal"},{"b":"7FE053250000","o":"33E05","s":"abort"},{"b":"400000","o":"7F74B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"643703","s":"_ZN5mongo12SecureRandom6createEv"},{"b":"400000","o":"3B14B3","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs"},{"b":"400000","o":"3B27CF","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs"},{"b":"400000","o":"3896DB","s":"_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs"},{"b":"400000","o":"3A55A7"},{"b":"400000","o":"3A7508"},{"b":"400000","o":"7A67F4","s":"_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"400000","o":"7A74BD","s":"_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi"},{"b":"400000","o":"7B5C3D","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"400000","o":"7A5BF2","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"400000","o":"297F55","s":"_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"81BF95","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7FE0535E4000","o":"7A51"},{"b":"7FE053250000","o":"E893D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.el6.x86_64", "version" : "#1 SMP Wed Oct 15 04:27:16 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "F21AE95924560F6FA79AF3D2316609D992A6E7FB" }, { "b" : "7FFF40EAB000", "elfType" : 3, "buildId" : "08E42C6C3D2CD1E5D68A43B717C9EB3D310F2DF0" }, { "b" : "7FE05448B000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "b" : "7FE0540A7000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E83FD82B1AE56360B270189F636423FB3016594" }, { "b" : "7FE053E9F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "58C5A5FF5C82D7BE3113BE36DD87C7004E3C4DB1" }, { "b" : "7FE053C9B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B5AE05CEDC0CE917F50A3A468CFA2ACD8592E8F6" }, { "b" : "7FE053A17000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "b" : "7FE053801000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "b" : "7FE0535E4000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D467973C46E563CDCF64B5F12B2D6A50C7A25BA1" }, { "b" : "7FE053250000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "87A19A61DC011579F3E25DE3CA9778C6FD9E4547" }, { "b" : "7FE0546F7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "04202A4A8BE624D2193E812A25589E2DD02D5B5C" }, { "b" : "7FE05300C000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "b" : "7FE052D25000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "b" : "7FE052B21000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "b" : "7FE0528F5000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "b" : "7FE0526DF000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FE0524D4000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "b" : "7FE0522D1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7FE0520B7000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F704FA7D21D05EF31E90FB4890FCA7F3D91DA138" }, { "b" : "7FE051E98000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc71ed2]
 mongos(+0x870E09) [0xc70e09]
 mongos(+0x871612) [0xc71612]
 libpthread.so.0(+0xF790) [0x7fe0535f3790]
 libc.so.6(gsignal+0x35) [0x7fe053282625]
 libc.so.6(abort+0x175) [0x7fe053283e05]
 mongos(_ZN5mongo13fassertFailedEi+0x82) [0xbf74b2]
 mongos(_ZN5mongo12SecureRandom6createEv+0x223) [0xa43703]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorISsSaISsEEPSs+0x17F3) [0x7b14b3]
 mongos(_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPSs+0x34F) [0x7b27cf]
 mongos(_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPSs+0x2B) [0x7896db]
 mongos(+0x3A55A7) [0x7a55a7]
 mongos(+0x3A7508) [0x7a7508]
 mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x6E4) [0xba67f4]
 mongos(_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x2ED) [0xba74bd]
 mongos(_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE+0x19D) [0xbb5c3d]
 mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x672) [0xba5bf2]
 mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x65) [0x697f55]
 mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0xc1bf95]
 libpthread.so.0(+0x7A51) [0x7fe0535eba51]
 libc.so.6(clone+0x6D) [0x7fe05333893d]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 04/Dec/16 ]

Hi birdylee_cn,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Thomas

Comment by Kelsey Schubert [ 14/Nov/16 ]

Hi birdylee_cn,

Thanks for reporting this issue. As you've seen, I've linked this ticket to SERVER-25659, which reduce the number of times the urandom file is opened, and would resolve this issue. Please continue to watch SERVER-25659 for updates.

In the meantime, to better understand what is going on here, would you please provide the diagnostic.data for the affected mongod and clarify the number databases and collections are present?

Thank you,
Thomas

Generated at Thu Feb 08 04:13:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.