[SERVER-31603] WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 17/Oct/17  Updated: 27/Jul/18  Resolved: 06/Nov/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nicolas Alfonso Assignee: Mark Agarunov
Resolution: Incomplete Votes: 0
Labels: envns, rns, rpns, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux Laboratorio 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux


Attachments: HTML File WiredTiger     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File repair-SERVER-31603.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi,

i have got this error on init mongodb:

2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] MongoDB starting : pid=2089 port=27017 dbpath=/var/lib/mongodb 64-bit host=Laboratorio
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] db version v3.4.1
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] git version: 5e103c4f5583e2566a45d740225dc250baacfbd7
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] modules: none
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] build environment:
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten]     distmod: debian81
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-17T14:39:33.638-0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
2017-10-17T14:39:33.642-0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-10-17T14:39:33.643-0300 I STORAGE  [initandlisten]
2017-10-17T14:39:33.643-0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-10-17T14:39:33.643-0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-10-17T14:39:33.643-0300 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-10-17T14:39:33.643-0300 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=256M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-10-17T14:39:33.651-0300 E STORAGE  [initandlisten] WiredTiger error (0) [1508261973:651290][2089:0x7fe6ffeeccc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2017-10-17T14:39:33.651-0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1508261973:651378][2089:0x7fe6ffeeccc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-10-17T14:39:33.651-0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-10-17T14:39:33.651-0300 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-10-17T14:39:33.671-0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x7fe7014742c1 0x7fe7014733b9 0x7fe70147389d 0x7fe6feb4b8d0 0x7fe6fe7c6067 0x7fe6fe7c7448 0x7fe70070eeed 0x7fe70118f626 0x7fe700719219 0x7fe70071930d 0x7fe700719565 0x7fe701e18dab 0x7fe701e1758e 0x7fe701dc9b30 0x7fe701e577c1 0x7fe701e57ed0 0x7fe701e5840c 0x7fe701dd9a99 0x7fe701e4c400 0x7fe701e16aee 0x7fe701e16bdb 0x7fe701dc6095 0x7fe701172f13 0x7fe70116baec 0x7fe70105b1b0 0x7fe7006fa8e3 0x7fe70071a84f 0x7fe6fe7b2b45 0x7fe7007783ff
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7FE6FFEFC000","o":"15782C1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FE6FFEFC000","o":"15773B9"},{"b":"7FE6FFEFC000","o":"157789D"},{"b":"7FE6FEB3C000","o":"F8D0"},{"b":"7FE6FE791000","o":"35067","s":"gsignal"},{"b":"7FE6FE791000","o":"36448","s":"abort"},{"b":"7FE6FFEFC000","o":"812EED","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7FE6FFEFC000","o":"1293626"},{"b":"7FE6FFEFC000","o":"81D219","s":"__wt_eventv"},{"b":"7FE6FFEFC000","o":"81D30D","s":"__wt_err"},{"b":"7FE6FFEFC000","o":"81D565","s":"__wt_panic"},{"b":"7FE6FFEFC000","o":"1F1CDAB","s":"__wt_turtle_read"},{"b":"7FE6FFEFC000","o":"1F1B58E","s":"__wt_metadata_search"},{"b":"7FE6FFEFC000","o":"1ECDB30","s":"__wt_conn_btree_open"},{"b":"7FE6FFEFC000","o":"1F5B7C1","s":"__wt_session_get_btree"},{"b":"7FE6FFEFC000","o":"1F5BED0","s":"__wt_session_get_btree"},{"b":"7FE6FFEFC000","o":"1F5C40C","s":"__wt_session_get_btree_ckpt"},{"b":"7FE6FFEFC000","o":"1EDDA99","s":"__wt_curfile_open"},{"b":"7FE6FFEFC000","o":"1F50400"},{"b":"7FE6FFEFC000","o":"1F1AAEE","s":"__wt_metadata_cursor_open"},{"b":"7FE6FFEFC000","o":"1F1ABDB","s":"__wt_metadata_cursor"},{"b":"7FE6FFEFC000","o":"1ECA095","s":"wiredtiger_open"},{"b":"7FE6FFEFC000","o":"1276F13","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"7FE6FFEFC000","o":"126FAEC"},{"b":"7FE6FFEFC000","o":"115F1B0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"7FE6FFEFC000","o":"7FE8E3"},{"b":"7FE6FFEFC000","o":"81E84F","s":"main"},{"b":"7FE6FE791000","o":"21B45","s":"__libc_start_main"},{"b":"7FE6FFEFC000","o":"87C3FF"}],"processInfo":{ "mongodbVersion" : "3.4.1", "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)", "machine" : "x86_64" }, "somap" : [ { "b" : "7FE6FFEFC000", "elfType" : 3, "buildId" : "38074E101F25D703C8F60A85C8471A7AE340C023" }, { "b" : "7FFF79934000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "89E9BE3B482AEC24D971AFB4916B043DB8772A0A" }, { "b" : "7FE6FFA78000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7FE6FF67C000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7FE6FF474000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7FE6FF270000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7FE6FEF6F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7FE6FED59000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7FE6FEB3C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7FE6FE791000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7FE6FFCD9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7fe7014742c1]
 mongod(+0x15773B9) [0x7fe7014733b9]
 mongod(+0x157789D) [0x7fe70147389d]
 libpthread.so.0(+0xF8D0) [0x7fe6feb4b8d0]
 libc.so.6(gsignal+0x37) [0x7fe6fe7c6067]
 libc.so.6(abort+0x148) [0x7fe6fe7c7448]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7fe70070eeed]
 mongod(+0x1293626) [0x7fe70118f626]
 mongod(__wt_eventv+0x427) [0x7fe700719219]
 mongod(__wt_err+0x9D) [0x7fe70071930d]
 mongod(__wt_panic+0x24) [0x7fe700719565]
 mongod(__wt_turtle_read+0x30B) [0x7fe701e18dab]
 mongod(__wt_metadata_search+0x24E) [0x7fe701e1758e]
 mongod(__wt_conn_btree_open+0x70) [0x7fe701dc9b30]
 mongod(__wt_session_get_btree+0x101) [0x7fe701e577c1]
 mongod(__wt_session_get_btree+0x810) [0x7fe701e57ed0]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x7fe701e5840c]
 mongod(__wt_curfile_open+0x179) [0x7fe701dd9a99]
 mongod(+0x1F50400) [0x7fe701e4c400]
 mongod(__wt_metadata_cursor_open+0x6E) [0x7fe701e16aee]
 mongod(__wt_metadata_cursor+0x4B) [0x7fe701e16bdb]
 mongod(wiredtiger_open+0x1745) [0x7fe701dc6095]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xD33) [0x7fe701172f13]
 mongod(+0x126FAEC) [0x7fe70116baec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x7fe70105b1b0]
 mongod(+0x7FE8E3) [0x7fe7006fa8e3]
 mongod(main+0x95F) [0x7fe70071a84f]
 libc.so.6(__libc_start_main+0xF5) [0x7fe6fe7b2b45]
 mongod(+0x87C3FF) [0x7fe7007783ff]
-----  END BACKTRACE  -----
Aborted

attached files.

Thank you very much.

Nico



 Comments   
Comment by Mark Agarunov [ 06/Nov/17 ]

Hello nick_9015

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 17/Oct/17 ]

Hello nick_9015,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:27:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.