[SERVER-30364] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 27/Jul/17  Updated: 27/Jul/18  Resolved: 02/Aug/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rahul Ranjan Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envh, rge, rpu, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-303640-repair.tar.gz     Zip Archive WiredTiger.zip    
Operating System: Linux
Participants:

 Description   

I'm not able to restart mongodb after reboot. I'm uploading the WiredTiger.* files for your reference. I tried running mongod --repair but found the same issue.



 Comments   
Comment by Ramon Fernandez Marina [ 06/May/18 ]

Apologies for the delay Durbala. When the underlying storage layer isn't reliable enough, unfortunately there may be no way to do any further recovery of the data. SERVER-19815 is open to attempt to do more in these cases, but for now doing regular backups and taking advantage of replication are your best bets.

Regards,
Ramón.

Comment by Maxim [X] [ 19/Apr/18 ]

Good day Mark, thank you for quick response.
If there`s no backup for current db state, and one that exist is too old to be restored and usefull, is there any achances to get data back (at least partial collections)?
Regards, Maxim

Comment by Mark Agarunov [ 02/Aug/17 ]

Hello rahul_ranjan,

Thank you for providing the additional information. Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to resync the affected node or restore from a backup if possible.

Thanks,
Mark

Comment by Rahul Ranjan [ 27/Jul/17 ]

1. I'm using SSD. The storage device is locally attached.
2. The disks are running fine.
3. Yes. 3.4.5.
4. No
5. No
6. No backups created.
7. The disk is marked clean.

Comment by Rahul Ranjan [ 27/Jul/17 ]

Hi, Thanks for replying. I replaced the files as provided by you and ran mongod --dbpath /var/lib/mongodb/ --port 27018. I got the following error:

2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] MongoDB starting : pid=3700 port=27018 dbpath=/var/lib/mongodb/ 64-bit host=api-temporary
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] db version v3.4.5
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] git version: 520b8f3092c48d934f0cd78ab5f40fe594f96863
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] modules: none
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] build environment:
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-07-27T15:31:02.924+0000 I CONTROL  [initandlisten] options: { net: { port: 27018 }, storage: { dbPath: "/var/lib/mongodb/" } }
2017-07-27T15:31:02.953+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-07-27T15:31:02.953+0000 I STORAGE  [initandlisten] 
2017-07-27T15:31:02.953+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-07-27T15:31:02.953+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-07-27T15:31:02.953+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15535M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-07-27T15:31:02.964+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1501169462:964318][3700:0x7fe64c2fed00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2017-07-27T15:31:02.964+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1501169462:964394][3700:0x7fe64c2fed00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-07-27T15:31:02.964+0000 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-07-27T15:31:02.964+0000 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-07-27T15:31:02.987+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x560847a14c01 0x560847a13e19 0x560847a142fd 0x7fe64af02390 0x7fe64ab5c428 0x7fe64ab5e02a 0x560846cc35b3 0x5608477288e6 0x560846ccd84c 0x560846ccda71 0x560846ccdcc9 0x5608483bb75c 0x5608483b9ec9 0x56084836e963 0x5608483f91cb 0x5608483f970d 0x5608483f99cc 0x56084837d221 0x5608483ef598 0x5608483b960e 0x5608483b96eb 0x56084836adad 0x56084770c9e9 0x560847705a1c 0x5608475f94c0 0x560846caf7de 0x560846ccf076 0x7fe64ab47830 0x560846d2d8d9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5608464AA000","o":"156AC01","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5608464AA000","o":"1569E19"},{"b":"5608464AA000","o":"156A2FD"},{"b":"7FE64AEF1000","o":"11390"},{"b":"7FE64AB27000","o":"35428","s":"gsignal"},{"b":"7FE64AB27000","o":"3702A","s":"abort"},{"b":"5608464AA000","o":"8195B3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5608464AA000","o":"127E8E6"},{"b":"5608464AA000","o":"82384C","s":"__wt_eventv"},{"b":"5608464AA000","o":"823A71","s":"__wt_err"},{"b":"5608464AA000","o":"823CC9","s":"__wt_panic"},{"b":"5608464AA000","o":"1F1175C","s":"__wt_turtle_read"},{"b":"5608464AA000","o":"1F0FEC9","s":"__wt_metadata_search"},{"b":"5608464AA000","o":"1EC4963","s":"__wt_conn_btree_open"},{"b":"5608464AA000","o":"1F4F1CB","s":"__wt_session_get_btree"},{"b":"5608464AA000","o":"1F4F70D","s":"__wt_session_get_btree"},{"b":"5608464AA000","o":"1F4F9CC","s":"__wt_session_get_btree_ckpt"},{"b":"5608464AA000","o":"1ED3221","s":"__wt_curfile_open"},{"b":"5608464AA000","o":"1F45598"},{"b":"5608464AA000","o":"1F0F60E","s":"__wt_metadata_cursor_open"},{"b":"5608464AA000","o":"1F0F6EB","s":"__wt_metadata_cursor"},{"b":"5608464AA000","o":"1EC0DAD","s":"wiredtiger_open"},{"b":"5608464AA000","o":"12629E9","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5608464AA000","o":"125BA1C"},{"b":"5608464AA000","o":"114F4C0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5608464AA000","o":"8057DE"},{"b":"5608464AA000","o":"825076","s":"main"},{"b":"7FE64AB27000","o":"20830","s":"__libc_start_main"},{"b":"5608464AA000","o":"8838D9","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.5", "gitVersion" : "520b8f3092c48d934f0cd78ab5f40fe594f96863", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.38-std-1", "version" : "#1 SMP Mon Dec 12 10:45:29 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "5608464AA000", "elfType" : 3, "buildId" : "7CA488310DEBB5E4D2FD6653DD93385F07B61791" }, { "b" : "7FFC8BFB2000", "elfType" : 3, "buildId" : "61F1D857B8E1317E89C3AFC5E77136E0B4636272" }, { "b" : "7FE64BE7D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FE64BA39000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7FE64B831000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FE64B62D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FE64B324000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FE64B10E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE64AEF1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FE64AB27000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FE64C0E6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x560847a14c01]
 mongod(+0x1569E19) [0x560847a13e19]
 mongod(+0x156A2FD) [0x560847a142fd]
 libpthread.so.0(+0x11390) [0x7fe64af02390]
 libc.so.6(gsignal+0x38) [0x7fe64ab5c428]
 libc.so.6(abort+0x16A) [0x7fe64ab5e02a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x560846cc35b3]
 mongod(+0x127E8E6) [0x5608477288e6]
 mongod(__wt_eventv+0x3D7) [0x560846ccd84c]
 mongod(__wt_err+0x9D) [0x560846ccda71]
 mongod(__wt_panic+0x24) [0x560846ccdcc9]
 mongod(__wt_turtle_read+0x26C) [0x5608483bb75c]
 mongod(__wt_metadata_search+0x99) [0x5608483b9ec9]
 mongod(__wt_conn_btree_open+0x73) [0x56084836e963]
 mongod(__wt_session_get_btree+0xFB) [0x5608483f91cb]
 mongod(__wt_session_get_btree+0x63D) [0x5608483f970d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x5608483f99cc]
 mongod(__wt_curfile_open+0x161) [0x56084837d221]
 mongod(+0x1F45598) [0x5608483ef598]
 mongod(__wt_metadata_cursor_open+0x6E) [0x5608483b960e]
 mongod(__wt_metadata_cursor+0x4B) [0x5608483b96eb]
 mongod(wiredtiger_open+0x164D) [0x56084836adad]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xA59) [0x56084770c9e9]
 mongod(+0x125BA1C) [0x560847705a1c]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5608475f94c0]
 mongod(+0x8057DE) [0x560846caf7de]
 mongod(main+0x966) [0x560846ccf076]
 libc.so.6(__libc_start_main+0xF0) [0x7fe64ab47830]
 mongod(_start+0x29) [0x560846d2d8d9]
-----  END BACKTRACE  -----
Aborted

Comment by Mark Agarunov [ 27/Jul/17 ]

Hello rahul_ranjan,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:23:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.