[SERVER-41830] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365 Created: 20/Jun/19  Updated: 04/Sep/19  Resolved: 04/Sep/19

Status: Closed
Project: Core Server
Component/s: Replication, WiredTiger
Affects Version/s: 3.4.17
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: R Dhanunjaya Assignee: Danny Hatcher (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:
  1. start initial sync for one of the instacne.
  2. Crashing with above error message.

 

Participants:

 Description   

Hi

I have deployed 6 mongod instances in one my hosts and suddenly one instance was crashing down. I cleared the dbpath and started again, still the db instance was getting crashed. All 6 instances are secondaries replicaSet members.

From the log, I can say that during initial sync, the instance was getting crashed.

MongoDB version : 3.4.17

Error message:

2019-06-20T07:06:55.145+0000 I ACCESS   [conn5] Successfully authenticated as principal mongoadmin on admin
 
2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
 
2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
 
2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (22) ###
 
2019-06-20T07:06:55.472+0000 E STORAGE  [thread2] WiredTiger error (-31804) ###
 
2019-06-20T07:06:55.472+0000 I -        [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
 
2019-06-20T07:06:55.472+0000 I -        [InitialSyncInserters-csilogcacheuat.sessions0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
 
2019-06-20T07:06:55.472+0000 I -        [thread2] 
 
 
 
***aborting after fassert() failure
 
 
 
 
 
2019-06-20T07:06:55.472+0000 I -        [InitialSyncInserters-csilogcacheuat.sessions0] 
 
 
 
***aborting after fassert() failure
 
 
 
 
 
2019-06-20T07:06:55.479+0000 F -        [thread2] Got signal: 6 (Aborted).
 
 
 
 0x5587074abac1 0x5587074aacd9 0x5587074ab1bd 0x7feb19d857e0 0x7feb19a144f5 0x7feb19a15cd5 0x55870667be87 0x5587071afd66 0x5587066865dc 0x5587066867f8 0x558706686a5a 0x558707e1010b 0x7feb19d7daa1 0x7feb19acac4d
 
----- BEGIN BACKTRACE -----
 
{"backtrace":[\{"b":"558705E10000","o":"169BAC1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"558705E10000","o":"169ACD9"},\{"b":"558705E10000","o":"169B1BD"},\{"b":"7FEB19D76000","o":"F7E0"},\{"b":"7FEB199E2000","o":"324F5","s":"gsignal"},\{"b":"7FEB199E2000","o":"33CD5","s":"abort"},\{"b":"558705E10000","o":"86BE87","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"558705E10000","o":"139FD66"},\{"b":"558705E10000","o":"8765DC","s":"__wt_eventv"},\{"b":"558705E10000","o":"8767F8","s":"__wt_err"},\{"b":"558705E10000","o":"876A5A","s":"__wt_panic"},\{"b":"558705E10000","o":"200010B"},\{"b":"7FEB19D76000","o":"7AA1"},\{"b":"7FEB199E2000","o":"E8C4D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "3.4.17", "gitVersion" : "7c14a47868643bb691a507a92fe25541f998eca4", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.14.2.el6.x86_64", "version" : "#1 SMP Tue May 14 15:07:36 PDT 2019", "machine" : "x86_64" }, "somap" : [ \{ "b" : "558705E10000", "elfType" : 3, "buildId" : "4AC2BE3D1E5BD9978C56B5917B77AA7E10DEC18F" }, \{ "b" : "7FFFA1D85000", "elfType" : 3, "buildId" : "A7FC5705F7F6DE181634D3E2DB49958368954175" }, \{ "b" : "7FEB1C361000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "D28AB8040B8D7DEFFF17AF81398D2E9F7937224E" }, \{ "b" : "7FEB1C152000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "04C0A5933873AB7117968F72138C68EA610BE27A" }, \{ "b" : "7FEB1BF38000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, \{ "b" : "7FEB1BCF4000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2CBD7B734972204176F168E3BC983371F08D46B0" }, \{ "b" : "7FEB1BA9F000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D56AD8E7A3B883EC7E31741DCC5A8A822E174ED8" }, \{ "b" : "7FEB1B853000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "454217BFFFA4753CC7A84320332ADBBAB64E1ABE" }, \{ "b" : "7FEB1B62D000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "E0F7FCE96C92B40C81580796E390FE1421849768" }, \{ "b" : "7FEB1B165000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "627E0627013003AC8A1889FCA97092542CA1D15F" }, \{ "b" : "7FEB1AE8A000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "1CA660DD5683CF31870475F5478B36A137C94B9D" }, \{ "b" : "7FEB1AC1E000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "0747A4EB0E12BB2405CF203C5F3AF9C2EDBF085C" }, \{ "b" : "7FEB1A839000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "974944833D88BB2A9B0E58223FB7FB40AAD7A8A4" }, \{ "b" : "7FEB1A631000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "F9D53A5992AD2393312DF3E9432CA48DF910A8CB" }, \{ "b" : "7FEB1A42D000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5A32931F43A32D2A7E17FF9D12240BA9EBFE5D7F" }, \{ "b" : "7FEB1A1A9000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0711CDD9A5BCC0879F2E1FD701769DB9708EEF5B" }, \{ "b" : "7FEB19F93000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C17B330215ACDEF76FB4C3722447A7E02C10E468" }, \{ "b" : "7FEB19D76000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2C6CFB7232FD5EF7519BCB41D4951F2A70141B35" }, \{ "b" : "7FEB199E2000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EA77452FE68EC731C702DA4EEAC9D362C495BAB2" }, \{ "b" : "7FEB1C5B2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D22090C72C839687EDD16B71CCBAFB5D8886A260" }, \{ "b" : "7FEB197C8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "CEF784F02AB1359774DA6697A659BF45BD6F1BBC" }, \{ "b" : "7FEB19575000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "035BA67841E482E54C743E398B80547E309F408B" }, \{ "b" : "7FEB19348000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "A17A9F1AFFEF3C27AC10D0C06D68EE539DB7E323" }, \{ "b" : "7FEB19000000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "67BFD154A5EA7C400AE39F59B7F4FF26B9E27883" }, \{ "b" : "7FEB18DD1000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "B2308B5F3B24A982D71662644805F3695C55EED9" }, \{ "b" : "7FEB18BCD000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "0D43C518C544BCE33E94B4B419A1996C10194A40" }, \{ "b" : "7FEB189C8000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "E54A5709B6D38BEEBE361C3CDC0CD596BEEA957D" }, \{ "b" : "7FEB18789000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "86AA37D47F155C625A881A8EF3FF878058DD59A7" }, \{ "b" : "7FEB18552000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "A98CAC454536D118561AABB98F1213B5DBA3C3B3" }, \{ "b" : "7FEB1826B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "68864AF40B36D187E9B61CCAD2741DB1A1E647A9" }, \{ "b" : "7FEB1803F000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "13CCE164ACE084DAB18CB6280557C0535FCBDE80" }, \{ "b" : "7FEB17E3B000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D593146D2D8A933B3782E63E021B13E72169426F" }, \{ "b" : "7FEB17C30000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3E8E0EB7EDCBF0C22F201C8E56EB2CB7F5017490" }, \{ "b" : "7FEB17A2D000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7FEB177FB000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "65E4BD3DC00C6CB5B31BE35CECFAEC28C57FA65F" }, \{ "b" : "7FEB175E5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7FEB173BD000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "52F6E18262B6DFAC418D2B474D51F8C927A3AC91" }, \{ "b" : "7FEB171B2000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "0FA4E248AED9CA89083D3D0F92BED84BDC34DFAE" }, \{ "b" : "7FEB16E47000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "85A5354FE6CFFCD1F426FDA56DCDD032295482B2" }, \{ "b" : "7FEB16C2E000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "00C397D9D891B74F726C098D32F6645BF6D3CBA5" }, \{ "b" : "7FEB16A2B000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "B41731AC40E067FC8625681C165B047EEC5B9FE9" }, \{ "b" : "7FEB167BF000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "FDB3B1273012F8918ECE26315793CB1CB66005A1" }, \{ "b" : "7FEB16590000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "DAB3811A4C8112FA8ABFC2406B04A3D64C6CA919" }, \{ "b" : "7FEB16387000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "13A7A55637B7F9EE91CBEEE8290DB3CBF05A8CB4" }, \{ "b" : "7FEB16177000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "145222F431C8F3461073C33D8548A8D45A12541E" }, \{ "b" : "7FEB15F74000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, \{ "b" : "7FEB15D55000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, \{ "b" : "7FEB15B44000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, \{ "b" : "7FEB1592E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "41EBCB50B31272B92BBB1BF662F9FE9914E205A8" }, \{ "b" : "7FEB1570D000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "C410ED8FEE037CAD13CF3A984ED072A41428D829" }, \{ "b" : "7FEB154E0000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "EBFA410D9187BD7C005C59ADB0E8F37C4592B492" }, \{ "b" : "7FEB152DC000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "F0C1DEF9ADED4895E882F028CA60D57DEECF195B" }, \{ "b" : "7FEB150D4000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "89FDBC49F99100E42556B961E8CF1254D69E370E" }, \{ "b" : "7FEB14D5F000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "30ADD6FC1CA5DB697FFEF49E7803D058FC4BF897" }, \{ "b" : "7FEB14B5A000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" }, \{ "b" : "7FEB14955000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "0F07393788EE03EE7B9E6EDEF596D3391A9DFAF4" }, \{ "b" : "7FEB1474F000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "FBF0F65D6B48974FC73A1D85C26D36E6FF6E0A58" }, \{ "b" : "7FEB1454A000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "5634FF39B695A423B348895DA7733085EAB2F4AD" }, \{ "b" : "7FEB14342000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "2CD3974E3D1BC903B97533F5158B1E830CCC3FF2" }, \{ "b" : "7FEB1413D000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "353EE8BA1C14EACF4D9742B681B3C5C5BBE21BC2" } ] }}
 
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5587074abac1]
 
 mongod(+0x169ACD9) [0x5587074aacd9]
 
 mongod(+0x169B1BD) [0x5587074ab1bd]
 
 libpthread.so.0(+0xF7E0) [0x7feb19d857e0]
 
 libc.so.6(gsignal+0x35) [0x7feb19a144f5]
 
 libc.so.6(abort+0x175) [0x7feb19a15cd5]
 
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55870667be87]
 
 mongod(+0x139FD66) [0x5587071afd66]
 
 mongod(__wt_eventv+0x3D7) [0x5587066865dc]
 
 mongod(__wt_err+0x9D) [0x5587066867f8]
 
 mongod(__wt_panic+0x2E) [0x558706686a5a]
 
 mongod(+0x200010B) [0x558707e1010b]
 
 libpthread.so.0(+0x7AA1) [0x7feb19d7daa1]
 
 libc.so.6(clone+0x6D) [0x7feb19acac4d]
 
-----  END BACKTRACE  -----



 Comments   
Comment by Danny Hatcher (Inactive) [ 29/Jul/19 ]

Have you had the chance to upload the complete mongod log?

Comment by Dmitry Agranat [ 30/Jun/19 ]

Hi mruthyunjaya117@gmail.com, you can upload the requested information to this secure portal

Comment by Danny Hatcher (Inactive) [ 20/Jun/19 ]

In order for us to fully determine the cause of the errors and any potential fixes, please upload the complete mongod log file from the start of the initial sync until the crash.

Generated at Thu Feb 08 04:58:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.