[SERVER-33137] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 05/Feb/18  Updated: 19/Feb/19  Resolved: 06/Feb/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Hon Wai Leong Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rdi, rpu, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-26103-WiredTiger-files.tar.gz     Text File mongod.log     File repair-SERVER-33137.tar.gz    
Operating System: Linux
Participants:

 Description   

Hello,

Due to /var space filling up, my mongodb failed to restart. Errors are seen as below:

2018-02-05T16:26:38.000-0600 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-02-05T16:26:38.000-0600 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-02-05T16:26:38.000-0600 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=7561M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archiv
e=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-02-05T16:26:38.015-0600 E STORAGE  [initandlisten] WiredTiger error (0) [1517869598:15394][27893:0x7f7077c1dd00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-02-05T16:26:38.015-0600 E STORAGE  [initandlisten] WiredTiger error (-31804) [1517869598:15426][27893:0x7f7077c1dd00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-02-05T16:26:38.015-0600 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361

How do I fix this issue?



 Comments   
Comment by Rafał Dołęga [ 19/Feb/19 ]

I've got the same error and has the same problem and it happend second time in the last month. What is reason of these problems ?! 

If I haven't got backup these mean that my data has been completely lost forever? 

Comment by Hon Wai Leong [ 07/Feb/18 ]

Thank you, Mark for looking into this.

Comment by Mark Agarunov [ 06/Feb/18 ]

Hello hwleong,

Unfortunately, this error indicates that there was corruption on the disk, most often cause by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thanks,
Mark

Comment by Hon Wai Leong [ 06/Feb/18 ]

mongod.log
Attached mongod.log. Thanks.

Comment by Hon Wai Leong [ 05/Feb/18 ]

I couldn't attach the log file for some reason. So, I paste it here:

2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] MongoDB starting : pid=15922 port=27017 dbpath=/var/lib/mongo 64-bit host=jyc1
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] db version v3.4.7
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 0.9.8j-fips 07 Jan 2009
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] allocator: tcmalloc
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] modules: none
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] build environment:
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten]     distmod: suse11
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten]     distarch: x86_64
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten]     target_arch: x86_64
2018-02-05T17:19:28.606-0600 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-02-05T17:19:28.625-0600 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-02-05T17:19:28.625-0600 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7561M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-02-05T17:19:28.640-0600 E STORAGE  [initandlisten] WiredTiger error (0) [1517872768:640553][15922:0x7f79d73d1d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-02-05T17:19:28.640-0600 E STORAGE  [initandlisten] WiredTiger error (-31804) [1517872768:640587][15922:0x7f79d73d1d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-02-05T17:19:28.640-0600 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-02-05T17:19:28.640-0600 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2018-02-05T17:19:28.658-0600 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x7f79d89d49e1 0x7f79d89d3bf9 0x7f79d89d40dd 0x7f79d61a6850 0x7f79d5e4d875 0x7f79d5e4ee51 0x7f79d7c8376b 0x7f79d86e8c46 0x7f79d7c8da04 0x7f79d7c8dc29 0x7f79d7c8de81 0x7f79d937b75c 0x7f79d9379ec9 0x7f79d932e963 0x7f79d93b92bb 0x7f79d93b97fd 0x7f79d93b9abc 0x7f79d933d151 0x7f79d93af668 0x7f79d937960e 0x7f79d93796eb 0x7f79d932acbd 0x7f79d86cda8f 0x7f79d86c630c 0x7f79d85b9a20 0x7f79d7c6f9ae 0x7f79d7c8f226 0x7f79d5e39c36 0x7f79d7cede11
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F79D7469000","o":"156B9E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F79D7469000","o":"156ABF9"},{"b":"7F79D7469000","o":"156B0DD"},{"b":"7F79D6197000","o":"F850"},{"b":"7F79D5E1B000","o":"32875","s":"gsignal"},{"b":"7F79D5E1B000","o":"33E51","s":"abort"},{"b":"7F79D7469000","o":"81A76B","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7F79D7469000","o":"127FC46"},{"b":"7F79D7469000","o":"824A04","s":"__wt_eventv"},{"b":"7F79D7469000","o":"824C29","s":"__wt_err"},{"b":"7F79D7469000","o":"824E81","s":"__wt_panic"},{"b":"7F79D7469000","o":"1F1275C","s":"__wt_turtle_read"},{"b":"7F79D7469000","o":"1F10EC9","s":"__wt_metadata_search"},{"b":"7F79D7469000","o":"1EC5963","s":"__wt_conn_btree_open"},{"b":"7F79D7469000","o":"1F502BB","s":"__wt_session_get_btree"},{"b":"7F79D7469000","o":"1F507FD","s":"__wt_session_get_btree"},{"b":"7F79D7469000","o":"1F50ABC","s":"__wt_session_get_btree_ckpt"},{"b":"7F79D7469000","o":"1ED4151","s":"__wt_curfile_open"},{"b":"7F79D7469000","o":"1F46668"},{"b":"7F79D7469000","o":"1F1060E","s":"__wt_metadata_cursor_open"},{"b":"7F79D7469000","o":"1F106EB","s":"__wt_metadata_cursor"},{"b":"7F79D7469000","o":"1EC1CBD","s":"wiredtiger_open"},{"b":"7F79D7469000","o":"1264A8F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"7F79D7469000","o":"125D30C"},{"b":"7F79D7469000","o":"1150A20","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"7F79D7469000","o":"8069AE"},{"b":"7F79D7469000","o":"826226","s":"main"},{"b":"7F79D5E1B000","o":"1EC36","s":"__libc_start_main"},{"b":"7F79D7469000","o":"884E11"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.0.101-0.46.1_1.0502.8871-cray_gem_s", "version" : "#1 SMP Fri Oct 20 18:35:26 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "7F79D7469000", "elfType" : 3, "buildId" : "17BC3601D350C4192F6612B026D5FA50B0407937" }, { "b" : "7FFF0AAFF000", "elfType" : 3, "buildId" : "31EB39A6596D81ABDB5219E85E4A9A575B0F7DC1" }, { "b" : "7F79D6FF0000", "path" : "/usr/lib64/libssl.so.0.9.8", "elfType" : 3, "buildId" : "681599EF92593F5D9CE1A8F02F0291FC0B2E5D6B" }, { "b" : "7F79D6C50000", "path" : "/usr/lib64/libcrypto.so.0.9.8", "elfType" : 3, "buildId" : "9DE23C98D49093A7F68F03271847C1EBD1423F35" }, { "b" : "7F79D6A47000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "E1A13ECB56367B69B89D1C9CA1A4C42167336030" }, { "b" : "7F79D6843000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "732292820E69F70459CB927ADE5B49BC56D32B0F" }, { "b" : "7F79D65CA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F0721CB50AB9FBDF06314A53BFF5AF581BBEFE64" }, { "b" : "7F79D63B4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3B149ECCD897F1F37DCE50AD22614043EBA757A2" }, { "b" : "7F79D6197000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "48172710254F4E2549684D7D3E9F9622272D6C66" }, { "b" : "7F79D5E1B000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E2CAB3C95CB1189420734B4AF264B047355BE2E5" }, { "b" : "7F79D7247000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9FDC592B21682A31F460F6F043F50EEA8C8B6821" }, { "b" : "7F79D5C05000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "C96D4398CB0A4F35E94CE172F9536CD5EBD20DD6" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f79d89d49e1]
 mongod(+0x156ABF9) [0x7f79d89d3bf9]
 mongod(+0x156B0DD) [0x7f79d89d40dd]
 libpthread.so.0(+0xF850) [0x7f79d61a6850]
 libc.so.6(gsignal+0x35) [0x7f79d5e4d875]
 libc.so.6(abort+0x181) [0x7f79d5e4ee51]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7f79d7c8376b]
 mongod(+0x127FC46) [0x7f79d86e8c46]
 mongod(__wt_eventv+0x3D7) [0x7f79d7c8da04]
 mongod(__wt_err+0x9D) [0x7f79d7c8dc29]
 mongod(__wt_panic+0x24) [0x7f79d7c8de81]
 mongod(__wt_turtle_read+0x26C) [0x7f79d937b75c]
 mongod(__wt_metadata_search+0x99) [0x7f79d9379ec9]
 mongod(__wt_conn_btree_open+0x73) [0x7f79d932e963]
 mongod(__wt_session_get_btree+0xFB) [0x7f79d93b92bb]
 mongod(__wt_session_get_btree+0x63D) [0x7f79d93b97fd]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x7f79d93b9abc]
 mongod(__wt_curfile_open+0x161) [0x7f79d933d151]
 mongod(+0x1F46668) [0x7f79d93af668]
 mongod(__wt_metadata_cursor_open+0x6E) [0x7f79d937960e]
 mongod(__wt_metadata_cursor+0x4B) [0x7f79d93796eb]
 mongod(wiredtiger_open+0x164D) [0x7f79d932acbd]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x7f79d86cda8f]
 mongod(+0x125D30C) [0x7f79d86c630c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x7f79d85b9a20]
 mongod(+0x8069AE) [0x7f79d7c6f9ae]
 mongod(main+0x966) [0x7f79d7c8f226]
 libc.so.6(__libc_start_main+0xE6) [0x7f79d5e39c36]
 mongod(+0x884E11) [0x7f79d7cede11]
-----  END BACKTRACE  -----

Comment by Hon Wai Leong [ 05/Feb/18 ]

Thank you for providing the repair files, unfortunately it doesn't fix the issue. I am attaching the mongod.log here.
mongod.log

Comment by Mark Agarunov [ 05/Feb/18 ]

Hello hwleong,

Thank you for providing these files. I've attached a repair attempt of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node(s) so that we can further investigate.

Thanks,
Mark

Comment by Hon Wai Leong [ 05/Feb/18 ]

SERVER-26103-WiredTiger-files.tar.gz

Hello Mark,

Thank you for your respond. As per request, I have attached both files. Thanks for looking into this.

Regards,
Hon Wai

Comment by Mark Agarunov [ 05/Feb/18 ]

Hello hwleong,

Thank you for the report. If you can provide the WiredTiger.wt and WiredTiger.turtle files we can attempt a repair of the database, but please keep in mind that this is not a guaranteed fix.

Thanks,
Mark

Generated at Thu Feb 08 04:32:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.