[SERVER-63965] Got signal: 11 Created: 24/Feb/22  Updated: 29/Mar/22  Resolved: 29/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: yeonsoon Choi Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod_20220210_crash.log    
Issue Links:
Duplicate
is duplicated by SERVER-63964 Got signal: 11 Closed
Operating System: ALL
Participants:

 Description   

Mongod in replica node crashes with a bellow msg.

  • [thread8740268] Invalid access at address: 0
  • [thread8740268] Got signal: 11 (Segmentation fault).

Here are the stacktraces and errors:

 
2022-02-10T01:04:02.094+0900 I ACCESS [conn8756101] Successfully authenticated as principal telegraf on admin from client 127.0.0.1:57516 2022-02-10T01:04:02.094+0900 I ACCESS [conn8756103] Successfully authenticated as principal telegraf on admin from client 127.0.0.1:57520 2022-02-10T01:04:02.094+0900 I ACCESS [conn8756102] Successfully authenticated as principal telegraf on admin from client 127.0.0.1:57518 2022-02-10T01:04:02.095+0900 I ACCESS [conn8756104] Successfully authenticated as principal telegraf on admin from client 127.0.0.1:57522 2022-02-10T01:04:02.095+0900 I ACCESS [conn8756105] Successfully authenticated as principal telegraf on admin from client 127.0.0.1:57524 2022-02-10T01:04:02.101+0900 I NETWORK [conn8756103] end connection 127.0.0.1:57520 (1048 connections now open) 2022-02-10T01:04:02.102+0900 I NETWORK [conn8756102] end connection 127.0.0.1:57518 (1047 connections now open) 2022-02-10T01:04:02.102+0900 I NETWORK [conn8756104] end connection 127.0.0.1:57522 (1046 connections now open) 2022-02-10T01:04:02.102+0900 I NETWORK [conn8756101] end connection 127.0.0.1:57516 (1045 connections now open) 2022-02-10T01:04:02.103+0900 I NETWORK [conn8756105] end connection 127.0.0.1:57524 (1044 connections now open) 2022-02-10T01:04:02.106+0900 F - [thread8740268] Invalid access at address: 0 2022-02-10T01:04:02.126+0900 F - [thread8740268] Got signal: 11 (Segmentation fault). 0x564f4e503421 0x564f4e502a3c 0x564f4e502c20 0x7fb9cda94630 0x564f4e606bdb 0x564f4e606e15 0x564f4cafdb3d 0x564f4cafe13e 0x564f4cb572fe 0x564f4cb5a6d6 0x564f4cb5e7e3 0x564f4caafa54 0x564f4cab0efa 0x564f4caa90ca 0x564f4caa9975 0x564f4caaba39 0x564f4cb0ac99 0x7fb9cda8cea5 0x7fb9cd7b58dd ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"564F4BC43000","o":"28C0421","s":"_ZN5mongo15printStackTraceERSo"},{"b":"564F4BC43000","o":"28BFA3C"},{"b":"564F4BC43000","o":"28BFC20"},{"b":"7FB9CDA85000","o":"F630"},{"b":"564F4BC43000","o":"29C3BDB","s":"_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEji"},{"b":"564F4BC43000","o":"29C3E15","s":"_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEj"},{"b":"564F4BC43000","o":"EBAB3D"},{"b":"564F4BC43000","o":"EBB13E","s":"__wt_stash_add"},{"b":"564F4BC43000","o":"F142FE"},{"b":"564F4BC43000","o":"F176D6"},{"b":"564F4BC43000","o":"F1B7E3","s":"__wt_split_reverse"},{"b":"564F4BC43000","o":"E6CA54"},{"b":"564F4BC43000","o":"E6DEFA","s":"__wt_evict"},{"b":"564F4BC43000","o":"E660CA"},{"b":"564F4BC43000","o":"E66975"},{"b":"564F4BC43000","o":"E68A39","s":"__wt_evict_thread_run"},{"b":"564F4BC43000","o":"EC7C99"},{"b":"7FB9CDA85000","o":"7EA5"},{"b":"7FB9CD6B7000","o":"FE8DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.9", "gitVersion" : "06402114114ffc5146fd4b55402c96f1dc9ec4b5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1127.8.2.el7.x86_64", "version" : "#1 SMP Tue May 12 16:57:42 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "564F4BC43000", "elfType" : 3, "buildId" : "3E8D7C55441120339684C6FD3BDDC6BD418EDCB0" }, { "b" : "7FFDCF4C2000", "elfType" : 3, "buildId" : "64B236454ED57A3FB07532D2829ED254FA0599AD" }, { "b" : "7FB9CEEB4000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "56C5F10F267E857EB448E1E823B84552B0D16976" }, { "b" : "7FB9CEC9A000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C3B2DD93CD59A17EA97148EC98C2667ADB9987A3" }, { "b" : "7FB9CE837000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "4CF1939F660008CFA869D8364651F31AACD2C1C4" }, { "b" : "7FB9CE5C5000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "3B305C3BA17FE394862E749763F2956C9C890C2E" }, { "b" : "7FB9CE3C1000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "F2C36986E11A291A0D4BCB3A81632B24AE2359EA" }, { "b" : "7FB9CE1B9000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CCD4BE566DD5A8FC7FA62B224C14B698F51B0D0D" }, { "b" : "7FB9CDEB7000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "085D924F5D23B9F15A8AD28B7231EE93C09E13F1" }, { "b" : "7FB9CDCA1000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DAC0179F4555AEFEC9E97476201802FD20C03EC5" }, { "b" : "7FB9CDA85000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2B482B3BAE79DEF4E5BC9791BC6BBDAE0E93E359" }, { "b" : "7FB9CD6B7000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D78066A9C36F5FD63E2F6AC851AE3515C4C9792A" }, { "b" : "7FB9CF11E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "27FFD1FBC69569C776E666474EED723395E6D727" }, { "b" : "7FB9CD484000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7FB9CD257000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "1AF123CADB2F2910E89CBD540A06D3B33692F95E" }, { "b" : "7FB9CCFFE000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "B6321C434B5C7386B144B925CEE2798D269FDDF5" }, { "b" : "7FB9CCDD6000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "BDA454441F59F41D2DA36E13CEA1FC4CE95B2BBB" }, { "b" : "7FB9CCAA7000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "DC3B36B530F506DE4FC1A6612D7DF44D4A3DDCDB" }, { "b" : "7FB9CC877000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "32C8FB6C2768FFE41E0A15CBF2089A4202CA2290" }, { "b" : "7FB9CC673000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "325B8CE57A776DE0B24B362A7E0C90E903B1A4B8" }, { "b" : "7FB9CC46E000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "0460FF10A3C63749113D380C40E10DFCF066C76E" }, { "b" : "7FB9CC230000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "8840B019EDB66B0CFBD2F77EF196440F7928106E" }, { "b" : "7FB9CBFE3000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "C3914975092B29D330453950350E254AA562D642" }, { "b" : "7FB9CBCFA000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "872EAC9F0CE30D5C4C37ECCCE3C586296D4FA1F0" }, { "b" : "7FB9CBAC7000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "7856E751772E8538A33113BA62145A8B23314093" }, { "b" : "7FB9CB8C3000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "E4C7298B74FEEADC4DDE40CDD8C4D6B85FE09ADE" }, { "b" : "7FB9CB6B4000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7FB9CB45F000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7FB9CB249000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7FB9CB039000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AED31F16223CE52AE079AB1ED4C09AC4C98F86B8" }, { "b" : "7FB9CAE35000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FB9CAC18000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7FB9CA9F1000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "903A0BD0BFB4FEE8C284F41BEB9773DED94CBC52" }, { "b" : "7FB9CA7BA000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "164A07A654E3B6AA09A43BF9ACE3728AB02BD0D7" }, { "b" : "7FB9CA558000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7FB9CA355000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "197680DAE6538245CB99723E57447C4EF2E98362" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x564f4e503421] mongod(+0x28BFA3C) [0x564f4e502a3c] mongod(+0x28BFC20) [0x564f4e502c20] libpthread.so.0(+0xF630) [0x7fb9cda94630] mongod(_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEji+0xEB) [0x564f4e606bdb] mongod(_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEj+0x35) [0x564f4e606e15] mongod(+0xEBAB3D) [0x564f4cafdb3d] mongod(__wt_stash_add+0x11E) [0x564f4cafe13e] mongod(+0xF142FE) [0x564f4cb572fe] mongod(+0xF176D6) [0x564f4cb5a6d6] mongod(__wt_split_reverse+0xA3) [0x564f4cb5e7e3] mongod(+0xE6CA54) [0x564f4caafa54] mongod(__wt_evict+0x144A) [0x564f4cab0efa] mongod(+0xE660CA) [0x564f4caa90ca] mongod(+0xE66975) [0x564f4caa9975] mongod(__wt_evict_thread_run+0x79) [0x564f4caaba39] mongod(+0xEC7C99) [0x564f4cb0ac99] libpthread.so.0(+0x7EA5) [0x7fb9cda8cea5] libc.so.6(clone+0x6D) [0x7fb9cd7b58dd] ----- END BACKTRACE -----



 Comments   
Comment by Edwin Zhou [ 29/Mar/22 ]

Hi himizes2@gmail.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Best,
Edwin

Comment by Edwin Zhou [ 10/Mar/22 ]

Hi himizes2@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the information my colleague dmitry.agranat requested via the upload portal?

Best,
Edwin

Comment by Dmitry Agranat [ 28/Feb/22 ]

Hi himizes2@gmail.com

Thanks for opening this ticket. To continue to investigate, would you please provide the following information covering the time of the event:

  • The full mongod.log file
  • The content of the diagnostic.data directory
  • Syslog messages
  • The results of running "ls -l" on the database home directory

I've created an upload portal for you to use here.

Thank you,
Dima

Generated at Thu Feb 08 05:59:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.