Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41830

Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Incomplete
    • Affects Version/s: 3.4.17
    • Fix Version/s: None
    • Component/s: Replication, WiredTiger
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide
      1. start initial sync for one of the instacne.
      2. Crashing with above error message.

       

      Show
      start initial sync for one of the instacne. Crashing with above error message.  

      Description

      Hi

      I have deployed 6 mongod instances in one my hosts and suddenly one instance was crashing down. I cleared the dbpath and started again, still the db instance was getting crashed. All 6 instances are secondaries replicaSet members.

      From the log, I can say that during initial sync, the instance was getting crashed.

      MongoDB version : 3.4.17

      Error message:

      2019-06-20T07:06:55.145+0000 I ACCESS   [conn5] Successfully authenticated as principal mongoadmin on admin
       
      2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
       
      2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
       
      2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
       
      2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (-31804) ###
       
      2019-06-20T07:06:55.472+0000 I -        [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
       
      2019-06-20T07:06:55.472+0000 I -        [InitialSyncInserters-csilogcacheuat.sessions0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
       
      2019-06-20T07:06:55.472+0000 I -        [thread2] 
       
       
       
      ***aborting after fassert() failure
       
       
       
       
       
      2019-06-20T07:06:55.472+0000 I -        [InitialSyncInserters-csilogcacheuat.sessions0] 
       
       
       
      ***aborting after fassert() failure
       
       
       
       
       
      2019-06-20T07:06:55.479+0000 F -        [thread2] Got signal: 6 (Aborted).
       
       
       
       0x5587074abac1 0x5587074aacd9 0x5587074ab1bd 0x7feb19d857e0 0x7feb19a144f5 0x7feb19a15cd5 0x55870667be87 0x5587071afd66 0x5587066865dc 0x5587066867f8 0x558706686a5a 0x558707e1010b 0x7feb19d7daa1 0x7feb19acac4d
       
      ----- BEGIN BACKTRACE -----
       
      {"backtrace":[\{"b":"558705E10000","o":"169BAC1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"558705E10000","o":"169ACD9"},\{"b":"558705E10000","o":"169B1BD"},\{"b":"7FEB19D76000","o":"F7E0"},\{"b":"7FEB199E2000","o":"324F5","s":"gsignal"},\{"b":"7FEB199E2000","o":"33CD5","s":"abort"},\{"b":"558705E10000","o":"86BE87","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"558705E10000","o":"139FD66"},\{"b":"558705E10000","o":"8765DC","s":"__wt_eventv"},\{"b":"558705E10000","o":"8767F8","s":"__wt_err"},\{"b":"558705E10000","o":"876A5A","s":"__wt_panic"},\{"b":"558705E10000","o":"200010B"},\{"b":"7FEB19D76000","o":"7AA1"},\{"b":"7FEB199E2000","o":"E8C4D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "3.4.17", "gitVersion" : "7c14a47868643bb691a507a92fe25541f998eca4", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.14.2.el6.x86_64", "version" : "#1 SMP Tue May 14 15:07:36 PDT 2019", "machine" : "x86_64" }, "somap" : [ \{ "b" : "558705E10000", "elfType" : 3, "buildId" : "4AC2BE3D1E5BD9978C56B5917B77AA7E10DEC18F" }, \{ "b" : "7FFFA1D85000", "elfType" : 3, "buildId" : "A7FC5705F7F6DE181634D3E2DB49958368954175" }, \{ "b" : "7FEB1C361000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "D28AB8040B8D7DEFFF17AF81398D2E9F7937224E" }, \{ "b" : "7FEB1C152000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "04C0A5933873AB7117968F72138C68EA610BE27A" }, \{ "b" : "7FEB1BF38000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, \{ "b" : "7FEB1BCF4000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2CBD7B734972204176F168E3BC983371F08D46B0" }, \{ "b" : "7FEB1BA9F000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D56AD8E7A3B883EC7E31741DCC5A8A822E174ED8" }, \{ "b" : "7FEB1B853000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "454217BFFFA4753CC7A84320332ADBBAB64E1ABE" }, \{ "b" : "7FEB1B62D000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "E0F7FCE96C92B40C81580796E390FE1421849768" }, \{ "b" : "7FEB1B165000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "627E0627013003AC8A1889FCA97092542CA1D15F" }, \{ "b" : "7FEB1AE8A000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "1CA660DD5683CF31870475F5478B36A137C94B9D" }, \{ "b" : "7FEB1AC1E000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "0747A4EB0E12BB2405CF203C5F3AF9C2EDBF085C" }, \{ "b" : "7FEB1A839000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "974944833D88BB2A9B0E58223FB7FB40AAD7A8A4" }, \{ "b" : "7FEB1A631000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "F9D53A5992AD2393312DF3E9432CA48DF910A8CB" }, \{ "b" : "7FEB1A42D000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5A32931F43A32D2A7E17FF9D12240BA9EBFE5D7F" }, \{ "b" : "7FEB1A1A9000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0711CDD9A5BCC0879F2E1FD701769DB9708EEF5B" }, \{ "b" : "7FEB19F93000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C17B330215ACDEF76FB4C3722447A7E02C10E468" }, \{ "b" : "7FEB19D76000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2C6CFB7232FD5EF7519BCB41D4951F2A70141B35" }, \{ "b" : "7FEB199E2000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EA77452FE68EC731C702DA4EEAC9D362C495BAB2" }, \{ "b" : "7FEB1C5B2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D22090C72C839687EDD16B71CCBAFB5D8886A260" }, \{ "b" : "7FEB197C8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "CEF784F02AB1359774DA6697A659BF45BD6F1BBC" }, \{ "b" : "7FEB19575000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "035BA67841E482E54C743E398B80547E309F408B" }, \{ "b" : "7FEB19348000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "A17A9F1AFFEF3C27AC10D0C06D68EE539DB7E323" }, \{ "b" : "7FEB19000000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "67BFD154A5EA7C400AE39F59B7F4FF26B9E27883" }, \{ "b" : "7FEB18DD1000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "B2308B5F3B24A982D71662644805F3695C55EED9" }, \{ "b" : "7FEB18BCD000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "0D43C518C544BCE33E94B4B419A1996C10194A40" }, \{ "b" : "7FEB189C8000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "E54A5709B6D38BEEBE361C3CDC0CD596BEEA957D" }, \{ "b" : "7FEB18789000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "86AA37D47F155C625A881A8EF3FF878058DD59A7" }, \{ "b" : "7FEB18552000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "A98CAC454536D118561AABB98F1213B5DBA3C3B3" }, \{ "b" : "7FEB1826B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "68864AF40B36D187E9B61CCAD2741DB1A1E647A9" }, \{ "b" : "7FEB1803F000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "13CCE164ACE084DAB18CB6280557C0535FCBDE80" }, \{ "b" : "7FEB17E3B000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D593146D2D8A933B3782E63E021B13E72169426F" }, \{ "b" : "7FEB17C30000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3E8E0EB7EDCBF0C22F201C8E56EB2CB7F5017490" }, \{ "b" : "7FEB17A2D000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7FEB177FB000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "65E4BD3DC00C6CB5B31BE35CECFAEC28C57FA65F" }, \{ "b" : "7FEB175E5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7FEB173BD000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "52F6E18262B6DFAC418D2B474D51F8C927A3AC91" }, \{ "b" : "7FEB171B2000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "0FA4E248AED9CA89083D3D0F92BED84BDC34DFAE" }, \{ "b" : "7FEB16E47000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "85A5354FE6CFFCD1F426FDA56DCDD032295482B2" }, \{ "b" : "7FEB16C2E000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "00C397D9D891B74F726C098D32F6645BF6D3CBA5" }, \{ "b" : "7FEB16A2B000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "B41731AC40E067FC8625681C165B047EEC5B9FE9" }, \{ "b" : "7FEB167BF000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "FDB3B1273012F8918ECE26315793CB1CB66005A1" }, \{ "b" : "7FEB16590000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "DAB3811A4C8112FA8ABFC2406B04A3D64C6CA919" }, \{ "b" : "7FEB16387000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "13A7A55637B7F9EE91CBEEE8290DB3CBF05A8CB4" }, \{ "b" : "7FEB16177000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "145222F431C8F3461073C33D8548A8D45A12541E" }, \{ "b" : "7FEB15F74000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, \{ "b" : "7FEB15D55000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, \{ "b" : "7FEB15B44000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, \{ "b" : "7FEB1592E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "41EBCB50B31272B92BBB1BF662F9FE9914E205A8" }, \{ "b" : "7FEB1570D000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "C410ED8FEE037CAD13CF3A984ED072A41428D829" }, \{ "b" : "7FEB154E0000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "EBFA410D9187BD7C005C59ADB0E8F37C4592B492" }, \{ "b" : "7FEB152DC000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "F0C1DEF9ADED4895E882F028CA60D57DEECF195B" }, \{ "b" : "7FEB150D4000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "89FDBC49F99100E42556B961E8CF1254D69E370E" }, \{ "b" : "7FEB14D5F000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "30ADD6FC1CA5DB697FFEF49E7803D058FC4BF897" }, \{ "b" : "7FEB14B5A000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" }, \{ "b" : "7FEB14955000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "0F07393788EE03EE7B9E6EDEF596D3391A9DFAF4" }, \{ "b" : "7FEB1474F000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "FBF0F65D6B48974FC73A1D85C26D36E6FF6E0A58" }, \{ "b" : "7FEB1454A000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "5634FF39B695A423B348895DA7733085EAB2F4AD" }, \{ "b" : "7FEB14342000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "2CD3974E3D1BC903B97533F5158B1E830CCC3FF2" }, \{ "b" : "7FEB1413D000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "353EE8BA1C14EACF4D9742B681B3C5C5BBE21BC2" } ] }}
       
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5587074abac1]
       
       mongod(+0x169ACD9) [0x5587074aacd9]
       
       mongod(+0x169B1BD) [0x5587074ab1bd]
       
       libpthread.so.0(+0xF7E0) [0x7feb19d857e0]
       
       libc.so.6(gsignal+0x35) [0x7feb19a144f5]
       
       libc.so.6(abort+0x175) [0x7feb19a15cd5]
       
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55870667be87]
       
       mongod(+0x139FD66) [0x5587071afd66]
       
       mongod(__wt_eventv+0x3D7) [0x5587066865dc]
       
       mongod(__wt_err+0x9D) [0x5587066867f8]
       
       mongod(__wt_panic+0x2E) [0x558706686a5a]
       
       mongod(+0x200010B) [0x558707e1010b]
       
       libpthread.so.0(+0x7AA1) [0x7feb19d7daa1]
       
       libc.so.6(clone+0x6D) [0x7feb19acac4d]
       
      -----  END BACKTRACE  -----
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: