[SERVER-58084] mongoDB crash with "Got signal: 11 (Segmentation fault)." Created: 25/Jun/21  Updated: 21/Jul/21  Resolved: 21/Jul/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bulavin Denys Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File BACKTRACE.txt    
Operating System: ALL
Participants:

 Description   

Good day!

MongoDB node crashed with "Got signal: 11 (Segmentation fault)."

Thank you!

2021-06-25T03:11:48.598+0300 F - [conn169640959] Invalid access at address: 0
2021-06-25T03:11:48.688+0300 F - [conn169640959] Got signal: 11 (Segmentation fault).
 0x55d97f9f7c81 0x55d97f9f747e 0x55d97f9f765c 0x7fc7f4c42370 0x55d97f8c75df 0x55d97f8cc3b9 0x55d97f8cfdd5 0x55d97f8d0452 0x55d97f8c4d11 0x55d97f8c4def 0x55d97f8c4e22 0x55d97f8c4edb 0x55d97f23eba1 0x55d97f23f457 
0x55d97f238655 0x55d97f23913a 0x55d97e62ce26 0x55d97e37238c 0x55d97e373f53 0x55d97e374e0e 0x55d97e3756a0 0x55d97e3636dc 0x55d97e36f2cc 0x55d97e36ac1f 0x55d97e36de9c 0x55d97f14f742 0x55d97e36863d 0x55d97e36b8d3 0
x55d97e369d07 0x55d97e36ab7b 0x55d97e36de9c 0x55d97f14fbab 0x55d97f789c94 0x7fc7f4c3adc5 0x7fc7f496976d
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"55D97D278000","o":"277FC81","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"55D97D278000","o":"277F47E"},\{"b":"55D97D278000","o":"277F65C"},\{"b":"7FC7F4C33000","o":"F370"},{"b":"55D97D278000","o"
:"264F5DF","s":"_ZN5mongo11LockManager9Partition4findENS_10ResourceIdE"},\{"b":"55D97D278000","o":"26543B9","s":"_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE"},{"b":"55D97D278000","
o":"2657DD5","s":"_ZN5mongo10LockerImpl9lockBeginEPNS_16OperationContextENS_10ResourceIdENS_8LockModeE"},{"b":"55D97D278000","o":"2658452","s":"_ZN5mongo10LockerImpl4lockEPNS_16OperationContextENS_10ResourceIdEN
S_8LockModeENS_6Date_tE"},\{"b":"55D97D278000","o":"264CD11","s":"_ZN5mongo4Lock10GlobalLock8_enqueueENS_8LockModeENS_6Date_tE"},{"b":"55D97D278000","o":"264CDEF","s":"_ZN5mongo4Lock10GlobalLockC2EPNS_16Operation
ContextENS_8LockModeENS_6Date_tENS0_17InterruptBehaviorENS1_11EnqueueOnlyE"},{"b":"55D97D278000","o":"264CE22","s":"_ZN5mongo4Lock10GlobalLockC1EPNS_16OperationContextENS_8LockModeENS_6Date_tENS0_17InterruptBeha
viorE"},\{"b":"55D97D278000","o":"264CEDB","s":"_ZN5mongo4Lock6DBLockC1EPNS_16OperationContextENS_10StringDataENS_8LockModeENS_6Date_tE"},{"b":"55D97D278000","o":"1FC6BA1","s":"_ZN5mongo9AutoGetDbC2EPNS_16Operati
onContextENS_10StringDataENS_8LockModeENS_6Date_tE"},{"b":"55D97D278000","o":"1FC7457","s":"_ZN5mongo17AutoGetCollectionC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_8LockModeES6_NS0_8ViewModeENS_6Da
te_tE"},\{"b":"55D97D278000","o":"1FC0655","s":"_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_17AutoGetCollection8ViewModeENS_6Date_tE"},{"b":"55D97D278000","o":"1FC
113A","s":"_ZN5mongo31AutoGetCollectionForReadCommandC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_17AutoGetCollection8ViewModeENS_6Date_tENS_16AutoStatsTracker7LogModeE"},{"b":"55D97D278000","o":"13
B4E26"},\{"b":"55D97D278000","o":"10FA38C"},\{"b":"55D97D278000","o":"10FBF53"},\{"b":"55D97D278000","o":"10FCE0E"},{"b":"55D97D278000","o":"10FD6A0","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16Ope
rationContextERKNS_7MessageERKNS0_5HooksE"},\{"b":"55D97D278000","o":"10EB6DC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55D97D278000","o":"10F72CC","s":
"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},\{"b":"55D97D278000","o":"10F2C1F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55D97D278000","o":"10F5E9C"
},\{"b":"55D97D278000","o":"1ED7742","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55D97D278000","o":"1
0F063D","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55D97D278000","o":"10F38D
3","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},\{"b":"55D97D278000","o":"10F1D07","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55D97D278000","o":"10F2B7B",
"s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"55D97D278000","o":"10F5E9C"},\{"b":"55D97D278000","o":"1ED7BAB"},\{"b":"55D97D278000","o":"2511C94"},\{"b":"7FC7F4C33000","o":"7DC5"},
{"b":"7FC7F4872000","o":"F776D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.0", "gitVersion" : "a4b751dcf51dd249c5865812b390cfd1c0129c30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "rele
ase" : "4.12.10-1.el7.elrepo.x86_64", "version" : "#1 SMP Wed Aug 30 13:00:07 EDT 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55D97D278000", "elfType" : 3, "buildId" : "E8D75D13E92279CB6AF8104353A95729FD
262FAB" }, \{ "b" : "7FFF32E42000", "elfType" : 3, "buildId" : "C9ACAE9C5B3E7976615033969B79FC8FB2BEEAEE" }, { "b" : "7FC7F6060000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "9114859D3C4BEC47A03
CA321EE367DCA799638CD" }, \{ "b" : "7FC7F5E46000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FE7AE845A123A3DFC0FDC2408BCBC2BA8B61B158" }, { "b" : "7FC7F59E5000", "path" : "/lib64/libcrypto.so.
10", "elfType" : 3, "buildId" : "BC0AE9CA0705BEC1F0C0375AAD839843BB219CB1" }, \{ "b" : "7FC7F5773000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "ED0AC7DEB91A242C194B3DEF27A215F41CE43116" }, { "b
" : "7FC7F556F000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7FC7F5367000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77A
DE22BC9FFF8D3458BD37331E7EDF174C28" }, \{ "b" : "7FC7F5065000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7FC7F4E4F000", "path" : "/lib64/libgc
c_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7FC7F4C33000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2
" }, \{ "b" : "7FC7F4872000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "626F78F2F88860B8844A5FAC68823D9605F802B2" }, { "b" : "7FC7F62C9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "
buildId" : "0874508AA13D28E3F48637C1D5BF067BA8D9FD3A" }, \{ "b" : "7FC7F463F000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7FC7F4415000", "
path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "C4EDF92922B4FE091F9A855C50343288AAD3243D" }, { "b" : "7FC7F41C3000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "DE2FC96025F4F43AA6E48523D
354581D2C5E0669" }, \{ "b" : "7FC7F3F9C000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "9757EB6D93E31024C436919A89BEF33CA4F1B984" }, { "b" : "7FC7F3C6F000", "path" : "/lib64/libnss3.so", "elfType
" : 3, "buildId" : "289D245DC413161931C170A1DFB9676582C84DBF" }, \{ "b" : "7FC7F3A40000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "B7F45CBBFB31A1390DA3CD65494FB4824046B22C" }, { "b" : "7FC7F3
83C000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "084D2194302908913F68B9DCD27DE46FA5B50522" }, { "b" : "7FC7F3637000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "799B28AD9A5460D7
8376E2C11260F2E858B95DE3" }, \{ "b" : "7FC7F33F9000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "DE762A28174110911B273E175D54F222B313CFE0" }, { "b" : "7FC7F31AB000", "path" : "/lib64/libgssapi_krb
5.so.2", "elfType" : 3, "buildId" : "A2499C359AA179EE23324ED949C0E508E4434F10" }, \{ "b" : "7FC7F2EC4000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "E09A34D9083DC6FEAF7018C09D55631DEEE2836D" }, 
{ "b" : "7FC7F2C92000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "BF8F00D7CB849ADB0B7A4703BC7B8D66AEE6A49C" }, { "b" : "7FC7F2A8E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "bui
ldId" : "BF54B7C8932E450769FBBB8B18864D1DD70BBC67" }, \{ "b" : "7FC7F287F000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "24F3F7F4D81B64DF3A3A1628791D128AFC30ADC9" }, { "b" : "7FC7F262C000", 
"path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "21A9C885023418F71BAC07695EC9800B0A1F3F40" }, { "b" : "7FC7F2416000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E2689
0470112D97A1F1E0B65" }, \{ "b" : "7FC7F2207000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "1E7A92FDD6FB3871DA97F4BCA2E147E72B6B6E1F" }, { "b" : "7FC7F2003000", "path" : "/lib64/libkeyutil
s.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, \{ "b" : "7FC7F1DE6000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "7F93B495437EEABFFC7FB12E87902C52D4930E69" },
 \{ "b" : "7FC7F1BBF000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7FC7F1988000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "build
Id" : "5124D1B5458527B46DA60C02874A5B7053A3DECD" }, \{ "b" : "7FC7F1727000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "AE64AA461A26E01F60408013D361749D56DD0AE1" }, { "b" : "7FC7F1524000", "path"
 : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55d97f9f7c81]
 mongod(+0x277F47E) [0x55d97f9f747e]
 mongod(+0x277F65C) [0x55d97f9f765c]
 libpthread.so.0(+0xF370) [0x7fc7f4c42370]
 mongod(_ZN5mongo11LockManager9Partition4findENS_10ResourceIdE+0x8F) [0x55d97f8c75df]
 mongod(_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE+0x89) [0x55d97f8cc3b9]
 mongod(_ZN5mongo10LockerImpl9lockBeginEPNS_16OperationContextENS_10ResourceIdENS_8LockModeE+0x325) [0x55d97f8cfdd5]
 mongod(_ZN5mongo10LockerImpl4lockEPNS_16OperationContextENS_10ResourceIdENS_8LockModeENS_6Date_tE+0x32) [0x55d97f8d0452]
 mongod(_ZN5mongo4Lock10GlobalLock8_enqueueENS_8LockModeENS_6Date_tE+0x51) [0x55d97f8c4d11]
 mongod(_ZN5mongo4Lock10GlobalLockC2EPNS_16OperationContextENS_8LockModeENS_6Date_tENS0_17InterruptBehaviorENS1_11EnqueueOnlyE+0x5F) [0x55d97f8c4def]
 mongod(_ZN5mongo4Lock10GlobalLockC1EPNS_16OperationContextENS_8LockModeENS_6Date_tENS0_17InterruptBehaviorE+0x12) [0x55d97f8c4e22]
 mongod(_ZN5mongo4Lock6DBLockC1EPNS_16OperationContextENS_10StringDataENS_8LockModeENS_6Date_tE+0x9B) [0x55d97f8c4edb]
 mongod(_ZN5mongo9AutoGetDbC2EPNS_16OperationContextENS_10StringDataENS_8LockModeENS_6Date_tE+0x31) [0x55d97f23eba1]
 mongod(_ZN5mongo17AutoGetCollectionC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_8LockModeES6_NS0_8ViewModeENS_6Date_tE+0x67) [0x55d97f23f457]
 mongod(_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_17AutoGetCollection8ViewModeENS_6Date_tE+0xD5) [0x55d97f238655]
 mongod(_ZN5mongo31AutoGetCollectionForReadCommandC2EPNS_16OperationContextERKNS_21NamespaceStringOrUUIDENS_17AutoGetCollection8ViewModeENS_6Date_tENS_16AutoStatsTracker7LogModeE+0x2A) [0x55d97f23913a]
 mongod(+0x13B4E26) [0x55d97e62ce26]
 mongod(+0x10FA38C) [0x55d97e37238c]
 mongod(+0x10FBF53) [0x55d97e373f53]
 mongod(+0x10FCE0E) [0x55d97e374e0e]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x540) [0x55d97e3756a0]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x55d97e3636dc]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x55d97e36f2cc]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x55d97e36ac1f]
 mongod(+0x10F5E9C) [0x55d97e36de9c]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x55d97f14f742]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x55d97e36863d]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x843) [0x55d97e36b8d3]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x2E7) [0x55d97e369d07]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x55d97e36ab7b]
 mongod(+0x10F5E9C) [0x55d97e36de9c]
 mongod(+0x1ED7BAB) [0x55d97f14fbab]
 mongod(+0x2511C94) [0x55d97f789c94]
 libpthread.so.0(+0x7DC5) [0x7fc7f4c3adc5]
 libc.so.6(clone+0x6D) [0x7fc7f496976d]
----- END BACKTRACE -----



 Comments   
Comment by Dmitry Agranat [ 21/Jul/21 ]

Hi d.bulavin81@yandex.ru,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 13/Jul/21 ]

Hi d.bulavin81@yandex.ru,

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information?

Thanks,
Dima

Comment by Dmitry Agranat [ 29/Jun/21 ]

Hi d.bulavin81@yandex.ru, for completeness, please upload the following data covering the reported event.

  • full mongod log covering a few hours before the segmentation fault
  • $dbpath/diagnostic.data archive
  • messeges, dmesg and syslog (depends on the system, you might not have all these available)

Is core dump enabled on this server? If yes, can you identify the core dump? (for now, we are just asking if a core dump was created during this event)

I've created a secure portal for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Thanks,
Dima

Generated at Thu Feb 08 05:43:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.