[SERVER-35005] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 15/May/18  Updated: 27/Jul/18  Resolved: 21/Jun/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.3.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tadej Markun Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, rpu, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File MongoLog     Zip Archive WiredTiger.zip     Zip Archive WiredTiger_new.zip     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi,
as many other on jira.mongodb.org I faced with the same situation. Mongodb/server has unexpected shutdown. After startup mongo print out error listed below. I've tried --repair and other suggestions that google finds, but didn't helped.

If some could help me I would be extremely grateful.
Thank you.

 

2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=04d1cfeeb14e
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] db version v3.3.15
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] git version: 520f5571d039b57cf9c319b49654909828971073
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] modules: none
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] build environment:
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten]     distmod: debian81
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-05-15T18:22:34.131+0000 I CONTROL  [initandlisten] options: {}
2018-05-15T18:22:34.135+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-05-15T18:22:34.136+0000 I STORAGE  [initandlisten]
2018-05-15T18:22:34.136+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-05-15T18:22:34.136+0000 I STORAGE  [initandlisten] See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-05-15T18:22:34.136+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=706M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-05-15T18:22:34.148+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1526408554:148875][1:0x7f3ab1b38cc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-05-15T18:22:34.148+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1526408554:148970][1:0x7f3ab1b38cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-05-15T18:22:34.149+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-05-15T18:22:34.149+0000 I -        [initandlisten]***aborting after fassert() failure
2018-05-15T18:22:34.190+0000 F -        [initandlisten] Got signal: 6 (Aborted). 0x55970bee2fe1 0x55970bee20d9 0x55970bee25bd 0x7f3ab07948d0 0x7f3ab040f067 0x7f3ab0410448 0x55970b1a48e5 0x55970bbce796 0x55970b1aee91 0x55970b1aef80 0x55970b1af1d8 0x55970c87ceab 0x55970c87b79e 0x55970c831440 0x55970c8b8c93 0x55970c8b9279 0x55970c8b950c 0x55970c840cd8 0x55970c8afc00 0x55970c87adfe 0x55970c87aeeb 0x55970c82dab8 0x55970bbb2b19 0x55970bbab94c 0x55970ba9c1b0 0x55970b19066c 0x55970b1b07ff 0x7f3ab03fbb45 0x55970b20c8ef
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55970A9A0000","o":"1542FE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55970A9A0000","o":"15420D9"},{"b":"55970A9A0000","o":"15425BD"},{"b":"7F3AB0785000","o":"F8D0"},{"b":"7F3AB03DA000","o":"35067","s":"gsignal"},{"b":"7F3AB03DA000","o":"36448","s":"abort"},{"b":"55970A9A0000","o":"8048E5","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55970A9A0000","o":"122E796"},{"b":"55970A9A0000","o":"80EE91","s":"__wt_eventv"},{"b":"55970A9A0000","o":"80EF80","s":"__wt_err"},{"b":"55970A9A0000","o":"80F1D8","s":"__wt_panic"},{"b":"55970A9A0000","o":"1EDCEAB","s":"__wt_turtle_read"},{"b":"55970A9A0000","o":"1EDB79E","s":"__wt_metadata_search"},{"b":"55970A9A0000","o":"1E91440","s":"__wt_conn_btree_open"},{"b":"55970A9A0000","o":"1F18C93","s":"__wt_session_get_btree"},{"b":"55970A9A0000","o":"1F19279","s":"__wt_session_get_btree"},{"b":"55970A9A0000","o":"1F1950C","s":"__wt_session_get_btree_ckpt"},{"b":"55970A9A0000","o":"1EA0CD8","s":"__wt_curfile_open"},{"b":"55970A9A0000","o":"1F0FC00"},{"b":"55970A9A0000","o":"1EDADFE","s":"__wt_metadata_cursor_open"},{"b":"55970A9A0000","o":"1EDAEEB","s":"__wt_metadata_cursor"},{"b":"55970A9A0000","o":"1E8DAB8","s":"wiredtiger_open"},{"b":"55970A9A0000","o":"1212B19","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55970A9A0000","o":"120B94C"},{"b":"55970A9A0000","o":"10FC1B0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55970A9A0000","o":"7F066C"},{"b":"55970A9A0000","o":"8107FF","s":"main"},{"b":"7F3AB03DA000","o":"21B45","s":"__libc_start_main"},{"b":"55970A9A0000","o":"86C8EF"}],"processInfo":{ "mongodbVersion" : "3.3.15", "gitVersion" : "520f5571d039b57cf9c319b49654909828971073", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-20-generic", "version" : "#21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55970A9A0000", "elfType" : 3, "buildId" : "3577DE2A89C1B5B1EB059231161B733A14CCB52A" }, { "b" : "7FFDAC3BD000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "A0297FE29F8038EF50A10A26C9FCF5249CCC1184" }, { "b" : "7F3AB16C1000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F3AB12C5000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F3AB10BD000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F3AB0EB9000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F3AB0BB8000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F3AB09A2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F3AB0785000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F3AB03DA000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F3AB1922000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55970bee2fe1]
 mongod(+0x15420D9) [0x55970bee20d9]
 mongod(+0x15425BD) [0x55970bee25bd]
 libpthread.so.0(+0xF8D0) [0x7f3ab07948d0]
 libc.so.6(gsignal+0x37) [0x7f3ab040f067]
 libc.so.6(abort+0x148) [0x7f3ab0410448]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55970b1a48e5]
 mongod(+0x122E796) [0x55970bbce796]
 mongod(__wt_eventv+0x496) [0x55970b1aee91]
 mongod(__wt_err+0x9D) [0x55970b1aef80]
 mongod(__wt_panic+0x24) [0x55970b1af1d8]
 mongod(__wt_turtle_read+0x30B) [0x55970c87ceab]
 mongod(__wt_metadata_search+0x23E) [0x55970c87b79e]
 mongod(__wt_conn_btree_open+0x70) [0x55970c831440]
 mongod(__wt_session_get_btree+0xF3) [0x55970c8b8c93]
 mongod(__wt_session_get_btree+0x6D9) [0x55970c8b9279]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55970c8b950c]
 mongod(__wt_curfile_open+0xB8) [0x55970c840cd8]
 mongod(+0x1F0FC00) [0x55970c8afc00]
 mongod(__wt_metadata_cursor_open+0x6E) [0x55970c87adfe]
 mongod(__wt_metadata_cursor+0x4B) [0x55970c87aeeb]
 mongod(wiredtiger_open+0x1698) [0x55970c82dab8]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xAF9) [0x55970bbb2b19]
 mongod(+0x120B94C) [0x55970bbab94c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x55970ba9c1b0]
 mongod(+0x7F066C) [0x55970b19066c]
 mongod(main+0x95F) [0x55970b1b07ff]
 libc.so.6(__libc_start_main+0xF5) [0x7f3ab03fbb45]
 mongod(+0x86C8EF) [0x55970b20c8ef]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 24/May/18 ]

Hi tadejcek,

Unfortunately, this error indicates that there was corruption on the disk, most often caused by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Tadej Markun [ 22/May/18 ]

Hi Kelsey,

Thank you for repaired files, but unfortunaly files didn't resolve the issue. I've attached full log of mongo..

Thank you,

Tadej

 

Comment by Kelsey Schubert [ 21/May/18 ]

Hi tadejcek,

Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Kelsey

Comment by Tadej Markun [ 20/May/18 ]

I have attached new zip with missing WiredTiger_new.zip

Thanks

Comment by Kelsey Schubert [ 20/May/18 ]

Hi tadejcek,

Would you please double check the zip? It looks like the WiredTiger.turtle file is missing.

Thanks,
Kelsey

Comment by Tadej Markun [ 15/May/18 ]

Hi Kelsey,

thank you for quick respons and recommendation. Files are attached.

 

Comment by Kelsey Schubert [ 15/May/18 ]

Hi tadejcek,

Would you please upload the WiredTiger.turtle and WiredTiger.wt files so we can take a look and determine if a repair attempt is possible?

Please note that MongoDB 3.3.15 is a development release and not suitable for production. I would recommend running on MongoDB 3.4 (which is the culmination of the 3.3 development series) or upgrading to MongoDB 3.6, which is the latest stable release.

Kind regards,
Kelsey 

Generated at Thu Feb 08 04:38:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.