[SERVER-50477] Mongodb crash, backtrace Created: 22/Aug/20  Updated: 27/Oct/23  Resolved: 24/Aug/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sergio d Assignee: Dmitry Agranat
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive mongod-log.zip    
Operating System: ALL
Participants:

 Description   

----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"556FD22A5000","o":"221C4E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556FD22A5000","o":"221B6F9"},{"b":"556FD22A5000","o":"221BBDD"},{"b":"7F046EEB1000","o":"110C0"},{"b":"7F046EB12000","o":"32FCF","s":"gsignal"},{"b":"7F046EB12000","o":"343FA","s":"abort"},{"b":"556FD22A5000","o":"98AE3F","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556FD22A5000","o":"221070D","s":"_ZN5mongo12SecureRandom6createEv"},{"b":"556FD22A5000","o":"F26168","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEPS7_"},{"b":"556FD22A5000","o":"F27045","s":"_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"556FD22A5000","o":"F2115A","s":"_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"556FD22A5000","o":"F1A8ED"},{"b":"556FD22A5000","o":"F1CBA9"},{"b":"556FD22A5000","o":"1B643E6","s":"_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"556FD22A5000","o":"1B5F2FF","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"556FD22A5000","o":"C1431E"},{"b":"556FD22A5000","o":"C163D4"},{"b":"556FD22A5000","o":"C17127","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"556FD22A5000","o":"C2375A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"556FD22A5000","o":"C1F2F7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"556FD22A5000","o":"C22541"},{"b":"556FD22A5000","o":"1B1FD92","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"556FD22A5000","o":"C1E160","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"556FD22A5000","o":"C206A2","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"556FD22A5000","o":"C20F9B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"556FD22A5000","o":"C1F37D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"556FD22A5000","o":"C22541"},{"b":"556FD22A5000","o":"1B202F5"},{"b":"556FD22A5000","o":"20D7994"},{"b":"7F046EEB1000","o":"7494"},{"b":"7F046EB12000","o":"E8AFF","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.0-4-amd64", "version" : "#1 SMP Debian 4.9.51-1 (2017-09-28)", "machine" : "x86_64" }, "somap" : [ { "b" : "556FD22A5000", "elfType" : 3, "buildId" : "0E4FE5BAA7832090BCC1D8B83A80341F7227AA6D" }, { "b" : "7FFD6ADEA000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "49A8CECB342FEC1F6FF31B2D8D6FD7CE11CD3E57" }, { "b" : "7F047004D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "1F8BBD45EFD498F52135C7F5B4F856577D5A4997" }, { "b" : "7F046FDEE000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7F046F9F5000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "5C11A26E0996D9EDCF3F7F509F751BBD081ABED1" }, { "b" : "7F046F7ED000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "FE41526A83999F2FE9D0F8AADCD61D03A92CBB70" }, { "b" : "7F046F5E9000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "6A5D98612129B8186F21E800AFDFAAA627082F46" }, { "b" : "7F046F2E5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "90DA054E12EA1A53EE0CBB5BB5E65F7069AEEE44" }, { "b" : "7F046F0CE000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "036DAE71A7197C847FC5B720634642B922C74398" }, { "b" : "7F046EEB1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "968DF33F83963B559243653D74D27D89605BED02" }, { "b" : "7F046EB12000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "79450F6E36287865D093EA209B85A222209925FF" }, { "b" : "7F0470264000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F150F33B150D6A81E26A425DD47D713D00F2D29" }, { "b" : "7F046E8F8000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "908B5A955D0A73FB8D31E0F927D0CDBA810CB300" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x556fd44c14e1]
 mongod(+0x221B6F9) [0x556fd44c06f9]
 mongod(+0x221BBDD) [0x556fd44c0bdd]
 libpthread.so.0(+0x110C0) [0x7f046eec20c0]
 libc.so.6(gsignal+0xCF) [0x7f046eb44fcf]
 libc.so.6(abort+0x16A) [0x7f046eb463fa]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556fd2c2fe3f]
 mongod(_ZN5mongo12SecureRandom6createEv+0x29D) [0x556fd44b570d]
 mongod(_ZN5mongo31SaslSCRAMSHA1ServerConversation10_firstStepERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEPS7_+0x19F8) [0x556fd31cb168]
 mongod(_ZN5mongo31SaslSCRAMSHA1ServerConversation4stepENS_10StringDataEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1C5) [0x556fd31cc045]
 mongod(_ZN5mongo31NativeSaslAuthenticationSession4stepENS_10StringDataEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3A) [0x556fd31c615a]
 mongod(+0xF1A8ED) [0x556fd31bf8ed]
 mongod(+0xF1CBA9) [0x556fd31c1ba9]
 mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x556fd3e093e6]
 mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x556fd3e042ff]
 mongod(+0xC1431E) [0x556fd2eb931e]
 mongod(+0xC163D4) [0x556fd2ebb3d4]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B7) [0x556fd2ebc127]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x556fd2ec875a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x556fd2ec42f7]
 mongod(+0xC22541) [0x556fd2ec7541]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x1A2) [0x556fd3dc4d92]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x150) [0x556fd2ec3160]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF2) [0x556fd2ec56a2]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x556fd2ec5f9b]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x556fd2ec437d]
 mongod(+0xC22541) [0x556fd2ec7541]
 mongod(+0x1B202F5) [0x556fd3dc52f5]
 mongod(+0x20D7994) [0x556fd437c994]
 libpthread.so.0(+0x7494) [0x7f046eeb8494]
 libc.so.6(clone+0x3F) [0x7f046ebfaaff]
-----  END BACKTRACE  -----



 Comments   
Comment by sergio d [ 24/Aug/20 ]

thank you

Comment by Dmitry Agranat [ 24/Aug/20 ]

Hi sergioduarte.dba@gmail.com,

When the server handles SCRAM authentication, it needs to open a SecureRandom object in order to generate a nonce. SecureRandom opens a file handle on /dev/urandom in order to obtain the necessary random bytes. If it can't open /dev/urandom, SecureRandom will fassert.

The reason it could not open a file handle can be found in the attached log:

2020-08-21T14:16:19.325-0300 I NETWORK  [listener] Error accepting new connection on 10.28.60.30:27017: Too many open files

I recommend either investigating and addressing the reason why you ran out of open files on this server or increasing the value of open files to something above the current 64000.

As this is an expected behavior, I will go ahead and close this ticket.

Regards,
Dima

Comment by sergio d [ 23/Aug/20 ]

upload done!

Comment by sergio d [ 23/Aug/20 ]

usr-mongo@repl01mongo02adm:~$ cat /proc/$(pidof mongod)/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 64000 64000 processes
Max open files 64000 64000 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128622 128622 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Comment by Dmitry Agranat [ 23/Aug/20 ]

Hi sergioduarte.dba@gmail.com,

Would you please archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Also, please post the output from this command:

cat /proc/$(pidof mongod)/limits

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Thanks,
Dima

Generated at Thu Feb 08 05:22:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.