[SERVER-28364] WiredTiger file corrupted after server crash Created: 17/Mar/17  Updated: 27/Jul/18  Resolved: 20/Mar/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrea Ghelfi Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair-SERVER-28364.tar.gz    
Operating System: Linux
Participants:

 Description   

I am running a standalone mongodb server 3.2.6 on Suse 11 x86_64 (production system)
After a server crash, mondodb server don't start with error messages:

{{2017-03-17T10:23:57.850+0100 E STORAGE [initandlisten] WiredTiger (-31802) [1489742637:850794][5845:0x7ffc3af69c80], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2017-03-17T10:23:57.851+0100 E STORAGE [initandlisten] WiredTiger (0) [1489742637:851001][5845:0x7ffc3af69c80], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2017-03-17T10:23:57.851+0100 E STORAGE [initandlisten] WiredTiger (0) [1489742637:851074][5845:0x7ffc3af69c80], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2017-03-17T10:23:57.851+0100 E STORAGE [initandlisten] WiredTiger (0) [1489742637:851148][5845:0x7ffc3af69c80], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options}}

I try to repair using mongod --repair with no success.



 Comments   
Comment by Mark Agarunov [ 20/Mar/17 ]

Hello a_ghelfi,

Unfortunately, this error indicates that the data became corrupted in some way. I would recommend a clean resync from a node that is not affected or restoring from a backup if possible.

Thanks,
Mark

Comment by Andrea Ghelfi [ 17/Mar/17 ]

Hello Mark,
I've tried the new files, now I get a different error but still no success:

2017-03-17T22:22:03.429+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2017-03-17T22:22:03.430+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-17T22:22:05.264+0100 I STORAGE [initandlisten] dropping unused ident: index-0--1819120192517270014
2017-03-17T22:22:05.338+0100 I STORAGE [initandlisten] dropping unused ident: index-1--1819120192517270014
2017-03-17T22:22:05.469+0100 I STORAGE [initandlisten] dropping unused ident: index-2--1819120192517270014
2017-03-17T22:22:05.490+0100 I STORAGE [initandlisten] dropping unused ident: index-3--1819120192517270014
2017-03-17T22:22:05.851+0100 I STORAGE [initandlisten] dropping unused ident: index-4--1819120192517270014
2017-03-17T22:22:05.863+0100 I STORAGE [initandlisten] dropping unused ident: index-5--1819120192517270014
2017-03-17T22:22:05.869+0100 I STORAGE [initandlisten] dropping unused ident: index-6--1819120192517270014
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten]
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten]
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-03-17T22:22:05.928+0100 I CONTROL [initandlisten]
2017-03-17T22:22:05.929+0100 I - [initandlisten] Fatal assertion 28579 NoSuchKey: Unable to find metadata for table:index-1-6525042046909573734
2017-03-17T22:22:05.930+0100 I - [initandlisten]

***aborting after fassert() failure

Thanks,
Andrea

Comment by Mark Agarunov [ 17/Mar/17 ]

Hello a_ghelfi,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Thanks,
Mark

Generated at Thu Feb 08 04:17:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.