[SERVER-35033] Fatal Assertion 28558 Created: 16/May/18  Updated: 28/Jun/18  Resolved: 21/May/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Abdelouahed TOUIMI Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Operating System: ALL
Participants:

 Description   

Hi, after several attempt, I can't fix my database.

the server crashed. here's what happens.

root@vmi157799:~# mongod --repair --dbpath /var/lib/mongodb/
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] MongoDB starting : pid=22978 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=vmi157799.contaboserver.net
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] db version v3.2.18
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] git version: 4c1bae566c0c00f996a2feb16febf84936ecaf6f
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] modules: none
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] build environment:
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     distarch: x86_64
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
2018-05-16T23:50:02.996+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-05-16T23:50:02.996+0200 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-05-16T23:50:02.996+0200 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=28G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-05-16T23:50:03.007+0200 E STORAGE  [initandlisten] WiredTiger (0) [1526507403:7255][22978:0x7f9783a9ac80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-05-16T23:50:03.007+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1526507403:7331][22978:0x7f9783a9ac80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-05-16T23:50:03.007+0200 I -        [initandlisten] Fatal Assertion 28558
2018-05-16T23:50:03.007+0200 I -        [initandlisten]***aborting after fassert() failure
2018-05-16T23:50:03.023+0200 F -        [initandlisten] Got signal: 6 (Aborted). 0x1559492 0x1558439 0x1558ca2 0x7f978269f390 0x7f97822f9428 0x7f97822fb02a 0x14d5563 0x1275b5a 0x97de59 0x97e03f 0x97e205 0x1c3988c 0x1c381da 0x1bef994 0x1c75add 0x1c76009 0x1c762ac 0x1bfdae1 0x1c72978 0x1c379af 0x1c37a6c 0x1bec12c 0x125a57a 0x1253065 0x114e28e 0x9cb181 0x97efda 0x7f97822e4830 0x9c6479
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1159492","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"1158439"},{"b":"400000","o":"1158CA2"},{"b":"7F978268E000","o":"11390"},{"b":"7F97822C4000","o":"35428","s":"gsignal"},{"b":"7F97822C4000","o":"3702A","s":"abort"},{"b":"400000","o":"10D5563","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E75B5A"},{"b":"400000","o":"57DE59","s":"__wt_eventv"},{"b":"400000","o":"57E03F","s":"__wt_err"},{"b":"400000","o":"57E205","s":"__wt_panic"},{"b":"400000","o":"183988C","s":"__wt_turtle_read"},{"b":"400000","o":"18381DA","s":"__wt_metadata_search"},{"b":"400000","o":"17EF994","s":"__wt_conn_btree_open"},{"b":"400000","o":"1875ADD","s":"__wt_session_get_btree"},{"b":"400000","o":"1876009","s":"__wt_session_get_btree"},{"b":"400000","o":"18762AC","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17FDAE1","s":"__wt_curfile_open"},{"b":"400000","o":"1872978"},{"b":"400000","o":"18379AF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1837A6C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17EC12C","s":"wiredtiger_open"},{"b":"400000","o":"E5A57A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E53065"},{"b":"400000","o":"D4E28E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5CB181"},{"b":"400000","o":"57EFDA","s":"main"},{"b":"7F97822C4000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5C6479","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.18", "gitVersion" : "4c1bae566c0c00f996a2feb16febf84936ecaf6f", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-104-generic", "version" : "#127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "3C61A0DBFA07F1A1069E23BC3A1ADF2E695605CE" }, { "b" : "7FFFC5FEF000", "elfType" : 3, "buildId" : "BF4B5E36B7E2464DCF29A888C247A27A3FF2BC5C" }, { "b" : "7F978361A000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F97831D6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F9782FCE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9782DCA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9782AC1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F97828AB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F978268E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F97822C4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9783883000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F9781DA1000", "path" : "/usr/lib/x86_64-linux-gnu/gconv/ISO8859-1.so", "elfType" : 3, "buildId" : "BFA0A1254EC586C3AA34328A7DDAA0A4DDAEF21D" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1559492]
 mongod(+0x1158439) [0x1558439]
 mongod(+0x1158CA2) [0x1558ca2]
 libpthread.so.0(+0x11390) [0x7f978269f390]
 libc.so.6(gsignal+0x38) [0x7f97822f9428]
 libc.so.6(abort+0x16A) [0x7f97822fb02a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d5563]
 mongod(+0xE75B5A) [0x1275b5a]
 mongod(__wt_eventv+0x3BA) [0x97de59]
 mongod(__wt_err+0x8B) [0x97e03f]
 mongod(__wt_panic+0x24) [0x97e205]
 mongod(__wt_turtle_read+0x24C) [0x1c3988c]
 mongod(__wt_metadata_search+0x8A) [0x1c381da]
 mongod(__wt_conn_btree_open+0x64) [0x1bef994]
 mongod(__wt_session_get_btree+0xDD) [0x1c75add]
 mongod(__wt_session_get_btree+0x609) [0x1c76009]
 mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c762ac]
 mongod(__wt_curfile_open+0x151) [0x1bfdae1]
 mongod(+0x1872978) [0x1c72978]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1c379af]
 mongod(__wt_metadata_cursor+0x3C) [0x1c37a6c]
 mongod(wiredtiger_open+0x18DC) [0x1bec12c]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x86A) [0x125a57a]
 mongod(+0xE53065) [0x1253065]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x114e28e]
 mongod(+0x5CB181) [0x9cb181]
 mongod(main+0x73A) [0x97efda]
 libc.so.6(__libc_start_main+0xF0) [0x7f97822e4830]
 mongod(_start+0x29) [0x9c6479]
-----  END BACKTRACE  -----
Aborted

Any suggestoin how i can fix it? I already know that some topics here have similar issue. What files do i need to upload to fix?

Thank you.



 Comments   
Comment by Kelsey Schubert [ 21/May/18 ]

Hi atouimi,

Unfortunately, manual inspection indicates that the WiredTiger.turtle file has been zeroed, which typically suggests some type of filesystem or operating system failure. In this case, it is not possible successfully restart the mongod with these files and our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Abdelouahed TOUIMI [ 20/May/18 ]

Hi
Is there still any hope to repair it.

Comment by Abdelouahed TOUIMI [ 17/May/18 ]

Hi

Thank you for your effort, I've replaced the file's + changed permissions

I think it's the same error 

 

root@vmi157799:/apps# mongod --repair --dbpath /var/lib/mongodb/
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] MongoDB starting : pid=2978 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=vmi157799.contaboserver.net
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] db version v3.2.18
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] git version: 4c1bae566c0c00f996a2feb16febf84936ecaf6f
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] modules: none
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] build environment:
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten]     distarch: x86_64
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2018-05-18T01:29:36.511+0200 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
2018-05-18T01:29:36.540+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-05-18T01:29:36.540+0200 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-05-18T01:29:36.540+0200 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=28G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-05-18T01:29:36.564+0200 E STORAGE  [initandlisten] WiredTiger (0) [1526599776:564721][2978:0x7fe923846c80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-05-18T01:29:36.564+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1526599776:564814][2978:0x7fe923846c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-05-18T01:29:36.564+0200 I -        [initandlisten] Fatal Assertion 28558
2018-05-18T01:29:36.564+0200 I -        [initandlisten]***aborting after fassert() failure
2018-05-18T01:29:36.589+0200 F -        [initandlisten] Got signal: 6 (Aborted). 0x1559492 0x1558439 0x1558ca2 0x7fe92244b390 0x7fe9220a5428 0x7fe9220a702a 0x14d5563 0x1275b5a 0x97de59 0x97e03f 0x97e205 0x1c3988c 0x1c381da 0x1bef994 0x1c75add 0x1c76009 0x1c762ac 0x1bfdae1 0x1c72978 0x1c379af 0x1c37a6c 0x1bec12c 0x125a57a 0x1253065 0x114e28e 0x9cb181 0x97efda 0x7fe922090830 0x9c6479
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1159492","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"1158439"},{"b":"400000","o":"1158CA2"},{"b":"7FE92243A000","o":"11390"},{"b":"7FE922070000","o":"35428","s":"gsignal"},{"b":"7FE922070000","o":"3702A","s":"abort"},{"b":"400000","o":"10D5563","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E75B5A"},{"b":"400000","o":"57DE59","s":"__wt_eventv"},{"b":"400000","o":"57E03F","s":"__wt_err"},{"b":"400000","o":"57E205","s":"__wt_panic"},{"b":"400000","o":"183988C","s":"__wt_turtle_read"},{"b":"400000","o":"18381DA","s":"__wt_metadata_search"},{"b":"400000","o":"17EF994","s":"__wt_conn_btree_open"},{"b":"400000","o":"1875ADD","s":"__wt_session_get_btree"},{"b":"400000","o":"1876009","s":"__wt_session_get_btree"},{"b":"400000","o":"18762AC","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17FDAE1","s":"__wt_curfile_open"},{"b":"400000","o":"1872978"},{"b":"400000","o":"18379AF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1837A6C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17EC12C","s":"wiredtiger_open"},{"b":"400000","o":"E5A57A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E53065"},{"b":"400000","o":"D4E28E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5CB181"},{"b":"400000","o":"57EFDA","s":"main"},{"b":"7FE922070000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5C6479","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.18", "gitVersion" : "4c1bae566c0c00f996a2feb16febf84936ecaf6f", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-104-generic", "version" : "#127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "3C61A0DBFA07F1A1069E23BC3A1ADF2E695605CE" }, { "b" : "7FFE21F5C000", "elfType" : 3, "buildId" : "BF4B5E36B7E2464DCF29A888C247A27A3FF2BC5C" }, { "b" : "7FE9233C6000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7FE922F82000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7FE922D7A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FE922B76000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FE92286D000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FE922657000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE92243A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FE922070000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FE92362F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7FE921B4D000", "path" : "/usr/lib/x86_64-linux-gnu/gconv/ISO8859-1.so", "elfType" : 3, "buildId" : "BFA0A1254EC586C3AA34328A7DDAA0A4DDAEF21D" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1559492]
 mongod(+0x1158439) [0x1558439]
 mongod(+0x1158CA2) [0x1558ca2]
 libpthread.so.0(+0x11390) [0x7fe92244b390]
 libc.so.6(gsignal+0x38) [0x7fe9220a5428]
 libc.so.6(abort+0x16A) [0x7fe9220a702a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d5563]
 mongod(+0xE75B5A) [0x1275b5a]
 mongod(__wt_eventv+0x3BA) [0x97de59]
 mongod(__wt_err+0x8B) [0x97e03f]
 mongod(__wt_panic+0x24) [0x97e205]
 mongod(__wt_turtle_read+0x24C) [0x1c3988c]
 mongod(__wt_metadata_search+0x8A) [0x1c381da]
 mongod(__wt_conn_btree_open+0x64) [0x1bef994]
 mongod(__wt_session_get_btree+0xDD) [0x1c75add]
 mongod(__wt_session_get_btree+0x609) [0x1c76009]
 mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c762ac]
 mongod(__wt_curfile_open+0x151) [0x1bfdae1]
 mongod(+0x1872978) [0x1c72978]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1c379af]
 mongod(__wt_metadata_cursor+0x3C) [0x1c37a6c]
 mongod(wiredtiger_open+0x18DC) [0x1bec12c]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x86A) [0x125a57a]
 mongod(+0xE53065) [0x1253065]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x114e28e]
 mongod(+0x5CB181) [0x9cb181]
 mongod(main+0x73A) [0x97efda]
 libc.so.6(__libc_start_main+0xF0) [0x7fe922090830]
 mongod(_start+0x29) [0x9c6479]
-----  END BACKTRACE  -----
Aborted

 

 

Comment by Kelsey Schubert [ 17/May/18 ]

Hi atouimi,

Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Kelsey

Comment by Abdelouahed TOUIMI [ 16/May/18 ]

Any help pls

Generated at Thu Feb 08 04:38:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.