-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.4.17
-
Component/s: Replication, WiredTiger
-
Labels:None
-
ALL
-
Hi
I have deployed 6 mongod instances in one my hosts and suddenly one instance was crashing down. I cleared the dbpath and started again, still the db instance was getting crashed. All 6 instances are secondaries replicaSet members.
From the log, I can say that during initial sync, the instance was getting crashed.
MongoDB version : 3.4.17
Error message:
2019-06-20T07:06:55.145+0000 I ACCESS [conn5] Successfully authenticated as principal mongoadmin on admin 2019-06-20T07:06:55.472+0000 E STORAGE [thread2] WiredTiger error (22) ### 2019-06-20T07:06:55.472+0000 E STORAGE [thread2] WiredTiger error (22) ### 2019-06-20T07:06:55.472+0000 E STORAGE [thread2] WiredTiger error (22) ### 2019-06-20T07:06:55.472+0000 E STORAGE [thread2] WiredTiger error (-31804) ### 2019-06-20T07:06:55.472+0000 I - [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365 2019-06-20T07:06:55.472+0000 I - [InitialSyncInserters-csilogcacheuat.sessions0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2019-06-20T07:06:55.472+0000 I - [thread2] ***aborting after fassert() failure 2019-06-20T07:06:55.472+0000 I - [InitialSyncInserters-csilogcacheuat.sessions0] ***aborting after fassert() failure 2019-06-20T07:06:55.479+0000 F - [thread2] Got signal: 6 (Aborted). 0x5587074abac1 0x5587074aacd9 0x5587074ab1bd 0x7feb19d857e0 0x7feb19a144f5 0x7feb19a15cd5 0x55870667be87 0x5587071afd66 0x5587066865dc 0x5587066867f8 0x558706686a5a 0x558707e1010b 0x7feb19d7daa1 0x7feb19acac4d ----- BEGIN BACKTRACE ----- {"backtrace":[\{"b":"558705E10000","o":"169BAC1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"558705E10000","o":"169ACD9"},\{"b":"558705E10000","o":"169B1BD"},\{"b":"7FEB19D76000","o":"F7E0"},\{"b":"7FEB199E2000","o":"324F5","s":"gsignal"},\{"b":"7FEB199E2000","o":"33CD5","s":"abort"},\{"b":"558705E10000","o":"86BE87","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"558705E10000","o":"139FD66"},\{"b":"558705E10000","o":"8765DC","s":"__wt_eventv"},\{"b":"558705E10000","o":"8767F8","s":"__wt_err"},\{"b":"558705E10000","o":"876A5A","s":"__wt_panic"},\{"b":"558705E10000","o":"200010B"},\{"b":"7FEB19D76000","o":"7AA1"},\{"b":"7FEB199E2000","o":"E8C4D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "3.4.17", "gitVersion" : "7c14a47868643bb691a507a92fe25541f998eca4", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.14.2.el6.x86_64", "version" : "#1 SMP Tue May 14 15:07:36 PDT 2019", "machine" : "x86_64" }, "somap" : [ \{ "b" : "558705E10000", "elfType" : 3, "buildId" : "4AC2BE3D1E5BD9978C56B5917B77AA7E10DEC18F" }, \{ "b" : "7FFFA1D85000", "elfType" : 3, "buildId" : "A7FC5705F7F6DE181634D3E2DB49958368954175" }, \{ "b" : "7FEB1C361000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "D28AB8040B8D7DEFFF17AF81398D2E9F7937224E" }, \{ "b" : "7FEB1C152000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "04C0A5933873AB7117968F72138C68EA610BE27A" }, \{ "b" : "7FEB1BF38000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, \{ "b" : "7FEB1BCF4000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2CBD7B734972204176F168E3BC983371F08D46B0" }, \{ "b" : "7FEB1BA9F000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D56AD8E7A3B883EC7E31741DCC5A8A822E174ED8" }, \{ "b" : "7FEB1B853000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "454217BFFFA4753CC7A84320332ADBBAB64E1ABE" }, \{ "b" : "7FEB1B62D000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "E0F7FCE96C92B40C81580796E390FE1421849768" }, \{ "b" : "7FEB1B165000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "627E0627013003AC8A1889FCA97092542CA1D15F" }, \{ "b" : "7FEB1AE8A000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "1CA660DD5683CF31870475F5478B36A137C94B9D" }, \{ "b" : "7FEB1AC1E000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "0747A4EB0E12BB2405CF203C5F3AF9C2EDBF085C" }, \{ "b" : "7FEB1A839000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "974944833D88BB2A9B0E58223FB7FB40AAD7A8A4" }, \{ "b" : "7FEB1A631000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "F9D53A5992AD2393312DF3E9432CA48DF910A8CB" }, \{ "b" : "7FEB1A42D000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5A32931F43A32D2A7E17FF9D12240BA9EBFE5D7F" }, \{ "b" : "7FEB1A1A9000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0711CDD9A5BCC0879F2E1FD701769DB9708EEF5B" }, \{ "b" : "7FEB19F93000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C17B330215ACDEF76FB4C3722447A7E02C10E468" }, \{ "b" : "7FEB19D76000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2C6CFB7232FD5EF7519BCB41D4951F2A70141B35" }, \{ "b" : "7FEB199E2000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EA77452FE68EC731C702DA4EEAC9D362C495BAB2" }, \{ "b" : "7FEB1C5B2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D22090C72C839687EDD16B71CCBAFB5D8886A260" }, \{ "b" : "7FEB197C8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "CEF784F02AB1359774DA6697A659BF45BD6F1BBC" }, \{ "b" : "7FEB19575000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "035BA67841E482E54C743E398B80547E309F408B" }, \{ "b" : "7FEB19348000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "A17A9F1AFFEF3C27AC10D0C06D68EE539DB7E323" }, \{ "b" : "7FEB19000000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "67BFD154A5EA7C400AE39F59B7F4FF26B9E27883" }, \{ "b" : "7FEB18DD1000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "B2308B5F3B24A982D71662644805F3695C55EED9" }, \{ "b" : "7FEB18BCD000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "0D43C518C544BCE33E94B4B419A1996C10194A40" }, \{ "b" : "7FEB189C8000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "E54A5709B6D38BEEBE361C3CDC0CD596BEEA957D" }, \{ "b" : "7FEB18789000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "86AA37D47F155C625A881A8EF3FF878058DD59A7" }, \{ "b" : "7FEB18552000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "A98CAC454536D118561AABB98F1213B5DBA3C3B3" }, \{ "b" : "7FEB1826B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "68864AF40B36D187E9B61CCAD2741DB1A1E647A9" }, \{ "b" : "7FEB1803F000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "13CCE164ACE084DAB18CB6280557C0535FCBDE80" }, \{ "b" : "7FEB17E3B000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D593146D2D8A933B3782E63E021B13E72169426F" }, \{ "b" : "7FEB17C30000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3E8E0EB7EDCBF0C22F201C8E56EB2CB7F5017490" }, \{ "b" : "7FEB17A2D000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7FEB177FB000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "65E4BD3DC00C6CB5B31BE35CECFAEC28C57FA65F" }, \{ "b" : "7FEB175E5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7FEB173BD000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "52F6E18262B6DFAC418D2B474D51F8C927A3AC91" }, \{ "b" : "7FEB171B2000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "0FA4E248AED9CA89083D3D0F92BED84BDC34DFAE" }, \{ "b" : "7FEB16E47000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "85A5354FE6CFFCD1F426FDA56DCDD032295482B2" }, \{ "b" : "7FEB16C2E000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "00C397D9D891B74F726C098D32F6645BF6D3CBA5" }, \{ "b" : "7FEB16A2B000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "B41731AC40E067FC8625681C165B047EEC5B9FE9" }, \{ "b" : "7FEB167BF000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "FDB3B1273012F8918ECE26315793CB1CB66005A1" }, \{ "b" : "7FEB16590000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "DAB3811A4C8112FA8ABFC2406B04A3D64C6CA919" }, \{ "b" : "7FEB16387000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "13A7A55637B7F9EE91CBEEE8290DB3CBF05A8CB4" }, \{ "b" : "7FEB16177000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "145222F431C8F3461073C33D8548A8D45A12541E" }, \{ "b" : "7FEB15F74000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, \{ "b" : "7FEB15D55000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, \{ "b" : "7FEB15B44000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, \{ "b" : "7FEB1592E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "41EBCB50B31272B92BBB1BF662F9FE9914E205A8" }, \{ "b" : "7FEB1570D000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "C410ED8FEE037CAD13CF3A984ED072A41428D829" }, \{ "b" : "7FEB154E0000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "EBFA410D9187BD7C005C59ADB0E8F37C4592B492" }, \{ "b" : "7FEB152DC000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "F0C1DEF9ADED4895E882F028CA60D57DEECF195B" }, \{ "b" : "7FEB150D4000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "89FDBC49F99100E42556B961E8CF1254D69E370E" }, \{ "b" : "7FEB14D5F000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "30ADD6FC1CA5DB697FFEF49E7803D058FC4BF897" }, \{ "b" : "7FEB14B5A000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" }, \{ "b" : "7FEB14955000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "0F07393788EE03EE7B9E6EDEF596D3391A9DFAF4" }, \{ "b" : "7FEB1474F000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "FBF0F65D6B48974FC73A1D85C26D36E6FF6E0A58" }, \{ "b" : "7FEB1454A000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "5634FF39B695A423B348895DA7733085EAB2F4AD" }, \{ "b" : "7FEB14342000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "2CD3974E3D1BC903B97533F5158B1E830CCC3FF2" }, \{ "b" : "7FEB1413D000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "353EE8BA1C14EACF4D9742B681B3C5C5BBE21BC2" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5587074abac1] mongod(+0x169ACD9) [0x5587074aacd9] mongod(+0x169B1BD) [0x5587074ab1bd] libpthread.so.0(+0xF7E0) [0x7feb19d857e0] libc.so.6(gsignal+0x35) [0x7feb19a144f5] libc.so.6(abort+0x175) [0x7feb19a15cd5] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55870667be87] mongod(+0x139FD66) [0x5587071afd66] mongod(__wt_eventv+0x3D7) [0x5587066865dc] mongod(__wt_err+0x9D) [0x5587066867f8] mongod(__wt_panic+0x2E) [0x558706686a5a] mongod(+0x200010B) [0x558707e1010b] libpthread.so.0(+0x7AA1) [0x7feb19d7daa1] libc.so.6(clone+0x6D) [0x7feb19acac4d] ----- END BACKTRACE -----