[SERVER-36837] WT Error, Fatal Assertion 28558 Created: 24/Aug/18  Updated: 15/Sep/18  Resolved: 24/Aug/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Martin Bak Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: envc, lxc, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repair-attempt.tar.gz     File wt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi, my mongodb instance cannot start after restarting the server - without stopping the mongodb in advance. Here is log when I try to start it manually:

 

# mongod --dbpath=/var/lib/mongodb 
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] MongoDB starting : pid=2504 port=27017 dbpath=/var/lib/mongodb 64-bit host=vopicka.dev.dszn.cz
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] db version v3.2.11
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2l  25 May 2017
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] modules: none
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] build environment:
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten]     distarch: x86_64
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2018-08-24T15:06:37.183+0200 I CONTROL  [initandlisten] options: { storage: { dbPath: "/var/lib/mongodb" } }
2018-08-24T15:06:37.214+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-08-24T15:06:37.214+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-08-24T15:06:37.258+0200 E STORAGE  [initandlisten] WiredTiger (0) [1535115997:258251][2504:0x7fa9fcb9ccc0], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 16384B block at offset 28672: block header checksum of 979725410 doesn't match expected checksum of 2731250473
2018-08-24T15:06:37.258+0200 E STORAGE  [initandlisten] WiredTiger (0) [1535115997:258316][2504:0x7fa9fcb9ccc0], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
2018-08-24T15:06:37.258+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1535115997:258343][2504:0x7fa9fcb9ccc0], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-08-24T15:06:37.258+0200 I -        [initandlisten] Fatal Assertion 28558
2018-08-24T15:06:37.258+0200 I -        [initandlisten] ***aborting after fassert() failure
2018-08-24T15:06:37.274+0200 F -        [initandlisten] Got signal: 6 (Aborted). 0x559efba01e2a 0x559efba00c79 0x559efba0159f 0x7fa9f96490c0 0x7fa9f92cbfcf 0x7fa9f92cd3fa 0x559efb97de62 0x559efb71c272 0x559efae01e06 0x559efae01ef6 0x559efae02154 0x559efbf5eb65 0x559efbf7b57d 0x559efbf7f8e8 0x559efbf9ffc1 0x559efbf6e33e 0x559efbfbe7c6 0x559efc051476 0x559efbfe513c 0x559efc051ce8 0x559efbfb3e27 0x559efbfac047 0x559efb6ff569 0x559efb6f8914 0x559efb5e6f4e 0x559efae4f43b 0x559efae02fbf 0x7fa9f92b92b1 0x559efae4975a
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"559EFA817000","o":"11EAE2A","s":"_ZN5mongo15printStackTraceERSo"},{"b":"559EFA817000","o":"11E9C79"},{"b":"559EFA817000","o":"11EA59F"},{"b":"7FA9F9638000","o":"110C0"},{"b":"7FA9F9299000","o":"32FCF","s":"gsignal"},{"b":"7FA9F9299000","o":"343FA","s":"abort"},{"b":"559EFA817000","o":"1166E62","s":"_ZN5mongo13fassertFailedEi"},{"b":"559EFA817000","o":"F05272"},{"b":"559EFA817000","o":"5EAE06","s":"__wt_eventv"},{"b":"559EFA817000","o":"5EAEF6","s":"__wt_err"},{"b":"559EFA817000","o":"5EB154","s":"__wt_panic"},{"b":"559EFA817000","o":"1747B65","s":"__wt_bm_read"},{"b":"559EFA817000","o":"176457D","s":"__wt_bt_read"},{"b":"559EFA817000","o":"17688E8","s":"__wt_page_in_func"},{"b":"559EFA817000","o":"1788FC1","s":"__wt_row_search"},{"b":"559EFA817000","o":"175733E","s":"__wt_btcur_insert"},{"b":"559EFA817000","o":"17A77C6"},{"b":"559EFA817000","o":"183A476"},{"b":"559EFA817000","o":"17CE13C","s":"__wt_log_scan"},{"b":"559EFA817000","o":"183ACE8","s":"__wt_txn_recover"},{"b":"559EFA817000","o":"179CE27","s":"__wt_connection_workers"},{"b":"559EFA817000","o":"1795047","s":"wiredtiger_open"},{"b":"559EFA817000","o":"EE8569","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"559EFA817000","o":"EE1914"},{"b":"559EFA817000","o":"DCFF4E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"559EFA817000","o":"63843B","s":"_ZN5mongo13initAndListenEi"},{"b":"559EFA817000","o":"5EBFBF","s":"main"},{"b":"7FA9F9299000","o":"202B1","s":"__libc_start_main"},{"b":"559EFA817000","o":"63275A","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.108-zen", "version" : "#1 SMP Fri Jun 15 09:22:41 CEST 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "559EFA817000", "elfType" : 3, "buildId" : "E3D43627426AEF80AC6422132B85090B61C6488B" }, { "b" : "7FFD5C7B0000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "B09B80DC0D1E32873BDE68F09018E27BD9A5E30E" }, { "b" : "7FA9FC712000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0", "elfType" : 3, "buildId" : "24B12C5DB7EB27CA6913E110036B6D501DE43D58" }, { "b" : "7FA9FC4F9000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0", "elfType" : 3, "buildId" : "3A8C0229F2AA03F8A062D06D70C30A7C7F49ED6A" }, { "b" : "7FA9FC2D1000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0", "elfType" : 3, "buildId" : "559760FAD4E5057673E6A198DCA0682077CA73AD" }, { "b" : "7FA9FC0CD000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0", "elfType" : 3, "buildId" : "9CEA3F7F5BCDA68EFA83A109C7C1C5C95EE5F9EE" }, { "b" : "7FA9FBEC6000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0", "elfType" : 3, "buildId" : "F7914F6A45BD6B1D843FF9B73651D7C85584D02F" }, { "b" : "7FA9FBBAD000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0", "elfType" : 3, "buildId" : "3FF0F4ABFF7BE32CCA250E9592555231B0960605" }, { "b" : "7FA9FB92F000", "path" : "/usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "B658AFC5A5081D3BC01514CD135E11A630ACADF9" }, { "b" : "7FA9FB727000", "path" : "/usr/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "2E75BB07FA335FA51E4A1F9EEFB8040CFF8639E1" }, { "b" : "7FA9FB4B4000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "1B72306EF00166FE1511D57140B50F5CE341170E" }, { "b" : "7FA9FB2AA000", "path" : "/usr/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "F9E20B1533BE511A4666D245232091723BD56BA9" }, { "b" : "7FA9FB036000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "63E43AB68BFBAD8631A9BC8549ADFA9BB8ED0DDC" }, { "b" : "7FA9FADE5000", "path" : "/usr/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "9CFDCA3EC241381801BA84204465310304FC7819" }, { "b" : "7FA9FABCB000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "908B5A955D0A73FB8D31E0F927D0CDBA810CB300" }, { "b" : "7FA9FA962000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.2", "elfType" : 3, "buildId" : "721B12B64868AFD74C3CC343086B69438BCF8A68" }, { "b" : "7FA9FA4FE000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2", "elfType" : 3, "buildId" : "E10962031A160EA04E49C7CD52C7A280FAEE19BD" }, { "b" : "7FA9FA2F6000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "FE41526A83999F2FE9D0F8AADCD61D03A92CBB70" }, { "b" : "7FA9FA0F2000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "6A5D98612129B8186F21E800AFDFAAA627082F46" }, { "b" : "7FA9F9D70000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "9E79D92950117355FD694E178C26ADE013CAE38B" }, { "b" : "7FA9F9A6C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "90DA054E12EA1A53EE0CBB5BB5E65F7069AEEE44" }, { "b" : "7FA9F9855000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "036DAE71A7197C847FC5B720634642B922C74398" }, { "b" : "7FA9F9638000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "968DF33F83963B559243653D74D27D89605BED02" }, { "b" : "7FA9F9299000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "79450F6E36287865D093EA209B85A222209925FF" }, { "b" : "7FA9FC991000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F150F33B150D6A81E26A425DD47D713D00F2D29" }, { "b" : "7FA9F781C000", "path" : "/usr/lib/x86_64-linux-gnu/libicudata.so.57", "elfType" : 3, "buildId" : "2DCEE0955E6639060DE44DEF87E349D608549DEA" }, { "b" : "7FA9F73A2000", "path" : "/usr/lib/x86_64-linux-gnu/libicui18n.so.57", "elfType" : 3, "buildId" : "DC010724B2591402197199B373006C25AA7BD8D7" }, { "b" : "7FA9F6FFA000", "path" : "/usr/lib/x86_64-linux-gnu/libicuuc.so.57", "elfType" : 3, "buildId" : "4E0664B37A77B3EFED5E9C4E7A4E2E1BDB2B4FD9" }, { "b" : "7FA9F6DDF000", "path" : "/usr/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "D018350F8DD20CC8574A21B2030049AB13E7CB1D" }, { "b" : "7FA9F6BB9000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "D94951F95E154271AE7CF843CC9D6D67EA502F5D" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x559efba01e2a]
 mongod(+0x11E9C79) [0x559efba00c79]
 mongod(+0x11EA59F) [0x559efba0159f]
 libpthread.so.0(+0x110C0) [0x7fa9f96490c0]
 libc.so.6(gsignal+0xCF) [0x7fa9f92cbfcf]
 libc.so.6(abort+0x16A) [0x7fa9f92cd3fa]
 mongod(_ZN5mongo13fassertFailedEi+0xA2) [0x559efb97de62]
 mongod(+0xF05272) [0x559efb71c272]
 mongod(__wt_eventv+0x42B) [0x559efae01e06]
 mongod(__wt_err+0xA0) [0x559efae01ef6]
 mongod(__wt_panic+0x24) [0x559efae02154]
 mongod(__wt_bm_read+0x135) [0x559efbf5eb65]
 mongod(__wt_bt_read+0x20D) [0x559efbf7b57d]
 mongod(__wt_page_in_func+0x1198) [0x559efbf7f8e8]
 mongod(__wt_row_search+0x701) [0x559efbf9ffc1]
 mongod(__wt_btcur_insert+0x4AE) [0x559efbf6e33e]
 mongod(+0x17A77C6) [0x559efbfbe7c6]
 mongod(+0x183A476) [0x559efc051476]
 mongod(__wt_log_scan+0xAEC) [0x559efbfe513c]
 mongod(__wt_txn_recover+0x398) [0x559efc051ce8]
 mongod(__wt_connection_workers+0x37) [0x559efbfb3e27]
 mongod(wiredtiger_open+0x1657) [0x559efbfac047]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xEA9) [0x559efb6ff569]
 mongod(+0xEE1914) [0x559efb6f8914]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3CE) [0x559efb5e6f4e]
 mongod(_ZN5mongo13initAndListenEi+0x48B) [0x559efae4f43b]
 mongod(main+0x72F) [0x559efae02fbf]
 libc.so.6(__libc_start_main+0xF1) [0x7fa9f92b92b1]
 mongod(_start+0x2A) [0x559efae4975a]
-----  END BACKTRACE  -----
Aborted

 

I also tried to run it with --repair, unfortunately without any luck:

 

# mongod --repair --dbpath=/var/lib/mongodb --storageEngine wiredTiger
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] MongoDB starting : pid=2509 port=27017 dbpath=/var/lib/mongodb 64-bit host=vopicka.dev.dszn.cz
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] db version v3.2.11
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2l 25 May 2017
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] allocator: tcmalloc
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] modules: none
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] build environment:
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] distarch: x86_64
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] target_arch: x86_64
2018-08-24T15:06:49.948+0200 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger" } }
2018-08-24T15:06:49.974+0200 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-08-24T15:06:49.974+0200 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-08-24T15:06:50.012+0200 E STORAGE [initandlisten] WiredTiger (0) [1535116010:12938][2509:0x7f73d370fcc0], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 16384B block at offset 28672: block header checksum of 979725410 doesn't match expected checksum of 2731250473
2018-08-24T15:06:50.013+0200 E STORAGE [initandlisten] WiredTiger (0) [1535116010:13002][2509:0x7f73d370fcc0], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
2018-08-24T15:06:50.013+0200 E STORAGE [initandlisten] WiredTiger (-31804) [1535116010:13017][2509:0x7f73d370fcc0], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-08-24T15:06:50.013+0200 I - [initandlisten] Fatal Assertion 28558
2018-08-24T15:06:50.013+0200 I - [initandlisten]
***aborting after fassert() failure
 
2018-08-24T15:06:50.028+0200 F - [initandlisten] Got signal: 6 (Aborted).
0x555de9ca0e2a 0x555de9c9fc79 0x555de9ca059f 0x7f73d01bc0c0 0x7f73cfe3efcf 0x7f73cfe403fa 0x555de9c1ce62 0x555de99bb272 0x555de90a0e06 0x555de90a0ef6 0x555de90a1154 0x555dea1fdb65 0x555dea21a57d 0x555dea21e8e8 0x555dea23efc1 0x555dea20d33e 0x555dea25d7c6 0x555dea2f0476 0x555dea28413c 0x555dea2f0ce8 0x555dea252e27 0x555dea24b047 0x555de999e300 0x555de9997914 0x555de9885f4e 0x555de90ee43b 0x555de90a1fbf 0x7f73cfe2c2b1 0x555de90e875a
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"555DE8AB6000","o":"11EAE2A","s":"_ZN5mongo15printStackTraceERSo"},{"b":"555DE8AB6000","o":"11E9C79"},{"b":"555DE8AB6000","o":"11EA59F"},{"b":"7F73D01AB000","o":"110C0"},{"b":"7F73CFE0C000","o":"32FCF","s":"gsignal"},{"b":"7F73CFE0C000","o":"343FA","s":"abort"},{"b":"555DE8AB6000","o":"1166E62","s":"_ZN5mongo13fassertFailedEi"},{"b":"555DE8AB6000","o":"F05272"},{"b":"555DE8AB6000","o":"5EAE06","s":"__wt_eventv"},{"b":"555DE8AB6000","o":"5EAEF6","s":"__wt_err"},{"b":"555DE8AB6000","o":"5EB154","s":"__wt_panic"},{"b":"555DE8AB6000","o":"1747B65","s":"__wt_bm_read"},{"b":"555DE8AB6000","o":"176457D","s":"__wt_bt_read"},{"b":"555DE8AB6000","o":"17688E8","s":"__wt_page_in_func"},{"b":"555DE8AB6000","o":"1788FC1","s":"__wt_row_search"},{"b":"555DE8AB6000","o":"175733E","s":"__wt_btcur_insert"},{"b":"555DE8AB6000","o":"17A77C6"},{"b":"555DE8AB6000","o":"183A476"},{"b":"555DE8AB6000","o":"17CE13C","s":"__wt_log_scan"},{"b":"555DE8AB6000","o":"183ACE8","s":"__wt_txn_recover"},{"b":"555DE8AB6000","o":"179CE27","s":"__wt_connection_workers"},{"b":"555DE8AB6000","o":"1795047","s":"wiredtiger_open"},{"b":"555DE8AB6000","o":"EE8300","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"555DE8AB6000","o":"EE1914"},{"b":"555DE8AB6000","o":"DCFF4E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"555DE8AB6000","o":"63843B","s":"_ZN5mongo13initAndListenEi"},{"b":"555DE8AB6000","o":"5EBFBF","s":"main"},{"b":"7F73CFE0C000","o":"202B1","s":"__libc_start_main"},{"b":"555DE8AB6000","o":"63275A","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.108-zen", "version" : "#1 SMP Fri Jun 15 09:22:41 CEST 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "555DE8AB6000", "elfType" : 3, "buildId" : "E3D43627426AEF80AC6422132B85090B61C6488B" }, { "b" : "7FFC45FEA000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "B09B80DC0D1E32873BDE68F09018E27BD9A5E30E" }, { "b" : "7F73D3285000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0", "elfType" : 3, "buildId" : "24B12C5DB7EB27CA6913E110036B6D501DE43D58" }, { "b" : "7F73D306C000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0", "elfType" : 3, "buildId" : "3A8C0229F2AA03F8A062D06D70C30A7C7F49ED6A" }, { "b" : "7F73D2E44000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0", "elfType" : 3, "buildId" : "559760FAD4E5057673E6A198DCA0682077CA73AD" }, { "b" : "7F73D2C40000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0", "elfType" : 3, "buildId" : "9CEA3F7F5BCDA68EFA83A109C7C1C5C95EE5F9EE" }, { "b" : "7F73D2A39000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0", "elfType" : 3, "buildId" : "F7914F6A45BD6B1D843FF9B73651D7C85584D02F" }, { "b" : "7F73D2720000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0", "elfType" : 3, "buildId" : "3FF0F4ABFF7BE32CCA250E9592555231B0960605" }, { "b" : "7F73D24A2000", "path" : "/usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "B658AFC5A5081D3BC01514CD135E11A630ACADF9" }, { "b" : "7F73D229A000", "path" : "/usr/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "2E75BB07FA335FA51E4A1F9EEFB8040CFF8639E1" }, { "b" : "7F73D2027000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "1B72306EF00166FE1511D57140B50F5CE341170E" }, { "b" : "7F73D1E1D000", "path" : "/usr/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "F9E20B1533BE511A4666D245232091723BD56BA9" }, { "b" : "7F73D1BA9000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "63E43AB68BFBAD8631A9BC8549ADFA9BB8ED0DDC" }, { "b" : "7F73D1958000", "path" : "/usr/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "9CFDCA3EC241381801BA84204465310304FC7819" }, { "b" : "7F73D173E000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "908B5A955D0A73FB8D31E0F927D0CDBA810CB300" }, { "b" : "7F73D14D5000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.2", "elfType" : 3, "buildId" : "721B12B64868AFD74C3CC343086B69438BCF8A68" }, { "b" : "7F73D1071000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2", "elfType" : 3, "buildId" : "E10962031A160EA04E49C7CD52C7A280FAEE19BD" }, { "b" : "7F73D0E69000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "FE41526A83999F2FE9D0F8AADCD61D03A92CBB70" }, { "b" : "7F73D0C65000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "6A5D98612129B8186F21E800AFDFAAA627082F46" }, { "b" : "7F73D08E3000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "9E79D92950117355FD694E178C26ADE013CAE38B" }, { "b" : "7F73D05DF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "90DA054E12EA1A53EE0CBB5BB5E65F7069AEEE44" }, { "b" : "7F73D03C8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "036DAE71A7197C847FC5B720634642B922C74398" }, { "b" : "7F73D01AB000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "968DF33F83963B559243653D74D27D89605BED02" }, { "b" : "7F73CFE0C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "79450F6E36287865D093EA209B85A222209925FF" }, { "b" : "7F73D3504000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F150F33B150D6A81E26A425DD47D713D00F2D29" }, { "b" : "7F73CE38F000", "path" : "/usr/lib/x86_64-linux-gnu/libicudata.so.57", "elfType" : 3, "buildId" : "2DCEE0955E6639060DE44DEF87E349D608549DEA" }, { "b" : "7F73CDF15000", "path" : "/usr/lib/x86_64-linux-gnu/libicui18n.so.57", "elfType" : 3, "buildId" : "DC010724B2591402197199B373006C25AA7BD8D7" }, { "b" : "7F73CDB6D000", "path" : "/usr/lib/x86_64-linux-gnu/libicuuc.so.57", "elfType" : 3, "buildId" : "4E0664B37A77B3EFED5E9C4E7A4E2E1BDB2B4FD9" }, { "b" : "7F73CD952000", "path" : "/usr/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "D018350F8DD20CC8574A21B2030049AB13E7CB1D" }, { "b" : "7F73CD72C000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "D94951F95E154271AE7CF843CC9D6D67EA502F5D" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x555de9ca0e2a]
 mongod(+0x11E9C79) [0x555de9c9fc79]
 mongod(+0x11EA59F) [0x555de9ca059f]
 libpthread.so.0(+0x110C0) [0x7f73d01bc0c0]
 libc.so.6(gsignal+0xCF) [0x7f73cfe3efcf]
 libc.so.6(abort+0x16A) [0x7f73cfe403fa]
 mongod(_ZN5mongo13fassertFailedEi+0xA2) [0x555de9c1ce62]
 mongod(+0xF05272) [0x555de99bb272]
 mongod(__wt_eventv+0x42B) [0x555de90a0e06]
 mongod(__wt_err+0xA0) [0x555de90a0ef6]
 mongod(__wt_panic+0x24) [0x555de90a1154]
 mongod(__wt_bm_read+0x135) [0x555dea1fdb65]
 mongod(__wt_bt_read+0x20D) [0x555dea21a57d]
 mongod(__wt_page_in_func+0x1198) [0x555dea21e8e8]
 mongod(__wt_row_search+0x701) [0x555dea23efc1]
 mongod(__wt_btcur_insert+0x4AE) [0x555dea20d33e]
 mongod(+0x17A77C6) [0x555dea25d7c6]
 mongod(+0x183A476) [0x555dea2f0476]
 mongod(__wt_log_scan+0xAEC) [0x555dea28413c]
 mongod(__wt_txn_recover+0x398) [0x555dea2f0ce8]
 mongod(__wt_connection_workers+0x37) [0x555dea252e27]
 mongod(wiredtiger_open+0x1657) [0x555dea24b047]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xC40) [0x555de999e300]
 mongod(+0xEE1914) [0x555de9997914]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3CE) [0x555de9885f4e]
 mongod(_ZN5mongo13initAndListenEi+0x48B) [0x555de90ee43b]
 mongod(main+0x72F) [0x555de90a1fbf]
 libc.so.6(__libc_start_main+0xF1) [0x7f73cfe2c2b1]
 mongod(_start+0x2A) [0x555de90e875a]
----- END BACKTRACE -----
Aborted

 

 

Can you please have a look at it and give any advice? All WiredTiger.* files from dpath are attached.

The server is LXC container running Debian Stretch:

 

# uname -a
Linux vopicka.dev.dszn.cz 4.9.108-zen #1 SMP Fri Jun 15 09:22:41 CEST 2018 x86_64 GNU/Linux

 

Thanks a lot.



 Comments   
Comment by Nick Brewer [ 24/Aug/18 ]

mbcko We do make use of an internal tool to fix these files. In terms of what you can do to avoid a similar problem moving forward, it's true that unexpected shutdowns are likely the cause here. The exact would depend on how the LXC container is actually being stopped, as well as the configuration of mongod on the system; lxc-stop should be used as opposed to simply killing the container. If this isn't possible, I'd suggest taking regular backups to more easily recover from an unclean shutdown when it occurs.

-Nick

Comment by Martin Bak [ 24/Aug/18 ]

Hi Nick, thanks for your super fast response. The mongodb successfully started with provided repaired files. Thank you .-)

Could I solve this problem myself or do you have some internal tools to repair WT files?

Also, this happened to me already before, can I prevent this somehow in the future? Manually stopping the instance would probably help, but since this is LXC container on a shared server, it might get restarted without knowing in advance..

Comment by Nick Brewer [ 24/Aug/18 ]

mbcko I've uploaded the files after a repair attempt. Can you please substitute them for the equivalent files currently in your dbpath, and let us know if this resolves the issue?

Thanks,
-Nick

repair-attempt.tar.gz

Generated at Thu Feb 08 04:44:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.