[SERVER-25770] WiredTiger.wt File corrupted Created: 24/Aug/16  Updated: 14/Aug/18  Resolved: 23/Sep/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: kuririn Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive 17017.zip     Zip Archive 27017.zip     Zip Archive 37017.zip     File repair_attempt-17017.tar.gz     File repair_attempt-27017.tar.gz     File repair_attempt-37017.tar.gz    
Operating System: Linux
Participants:

 Description   

I am running mongodb on version 3.2.6 , ubuntu 14.04.4. One day my office was power outage, after that my mongod could not start.
The error like this:
···
test
···
klg@koala:~$ tail /var/log/mongodb/wiredtiger_spider.log
2016-08-24T19:15:51.487+0800 I CONTROL [initandlisten] options: { config: "/etc/mongodb/mongodb_spider.conf.bak", net:

{ port: 17017 }

, processManagement:

{ fork: true, pidFilePath: "/tmp/mongo_17017.pid" }

, storage: { dbPath: "/var/lib/mongodb/wiredtiger_spider", directoryPerDB: true, engine: "wiredTiger", journal:

{ enabled: true }

}, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/mongodb/wiredtiger_spider.log", verbosity: 1 }

}
2016-08-24T19:15:51.487+0800 D NETWORK [initandlisten] fd limit hard:4096 soft:1024 max conn: 819
2016-08-24T19:15:51.516+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-08-24T19:15:51.525+0800 E STORAGE [initandlisten] WiredTiger (-31802) [1472037351:525304][49250:0x7fc335dcecc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2016-08-24T19:15:51.525+0800 E STORAGE [initandlisten] WiredTiger (0) [1472037351:525367][49250:0x7fc335dcecc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2016-08-24T19:15:51.525+0800 E STORAGE [initandlisten] WiredTiger (0) [1472037351:525379][49250:0x7fc335dcecc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2016-08-24T19:15:51.525+0800 E STORAGE [initandlisten] WiredTiger (0) [1472037351:525391][49250:0x7fc335dcecc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2016-08-24T19:15:51.526+0800 I - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error
2016-08-24T19:15:51.558+0800 I STORAGE [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2016-08-24T19:15:51.558+0800 I CONTROL [initandlisten] dbexit: rc: 100



 Comments   
Comment by Kelsey Schubert [ 23/Sep/16 ]

Hi gaoxianrui52,

Sorry for the delay getting back to you. We have been discussing this issue internally to determine whether there is anything more that we can do in situations like this. Unfortunately, I cannot suggest any additional actions to take to repair this corruption.

Kind regards,
Thomas

Comment by kuririn [ 25/Aug/16 ]

Sorry i can not recovery that collection.I try to use mondod --repair and wt, but can't work.
mongod logs like this:

2016-08-25T11:38:12.286+0800 I CONTROL [initandlisten] build environment:
2016-08-25T11:38:12.286+0800 I CONTROL [initandlisten] distmod: ubuntu1404
2016-08-25T11:38:12.286+0800 I CONTROL [initandlisten] distarch: x86_64
2016-08-25T11:38:12.286+0800 I CONTROL [initandlisten] target_arch: x86_64
2016-08-25T11:38:12.286+0800 I CONTROL [initandlisten] options: { config: "/etc/mongodb/mongodb_spider.conf", net:

Unknown macro: { port}

, processManagement:

Unknown macro: { fork}

, storage: { dbPath: "/var/lib/mongodb/wiredtiger_spider", directoryPerDB: true, engine: "wiredTiger", journal:

Unknown macro: { enabled}

}, systemLog:

Unknown macro: { destination}

}
2016-08-25T11:38:12.286+0800 D NETWORK [initandlisten] fd limit hard:4096 soft:1024 max conn: 819
2016-08-25T11:38:12.305+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-08-25T11:38:12.365+0800 E STORAGE [initandlisten] WiredTiger (0) [1472096292:365648][36410:0x7f3b0fdb4cc0], file:new_koala/collection-2-5201603023682654486.wt, WT_CURSOR.insert: read checksum error for 4096B block at offset 4121976832: block header checksum of 181112124 doesn't match expected checksum of 3456616363
2016-08-25T11:38:12.365+0800 E STORAGE [initandlisten] WiredTiger (0) [1472096292:365718][36410:0x7f3b0fdb4cc0], file:new_koala/collection-2-5201603023682654486.wt, WT_CURSOR.insert: new_koala/collection-2-5201603023682654486.wt: encountered an illegal file format or internal value
2016-08-25T11:38:12.365+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1472096292:365731][36410:0x7f3b0fdb4cc0], file:new_koala/collection-2-5201603023682654486.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-08-25T11:38:12.365+0800 I - [initandlisten] Fatal Assertion 28558
2016-08-25T11:38:12.365+0800 I - [initandlisten]

***aborting after fassert() failure

And i use ./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2-5201603023682654486.wt,
wt's info:

lt-wt: salvage(file:collection-2-5201603023682654486.wt): No such file or directory

i sure that the collection-2-5201603023682654486.wt fle is in ../mongo-bak directory.And i also try ../mongo-bak/collection-2-5201603023682654486.wt ,not work ,error same as before.
Some advice to me?

Comment by kuririn [ 25/Aug/16 ]

Thanks anonymous.user. i got that and try to start,but have a new error. One collection corrupted too, i'm try to recovery it.
Please accept my knees! Thanks again.

Comment by Kelsey Schubert [ 24/Aug/16 ]

Hi gaoxianrui52,

I have attached the repair attempts for the files you have uploaded. Please move these files to your $dbpath and restart the mongod.

Kind regards,
Thomas

Comment by Ramon Fernandez Marina [ 24/Aug/16 ]

Sorry you run into this gaoxianrui52, we'll examine your files and get back to you with more information.

Comment by kuririn [ 24/Aug/16 ]

I have 3 dbs need recovery,somebody help me?

Generated at Thu Feb 08 04:10:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.