[SERVER-29358] Another WiredTiger.wt: encountered an illegal file format or internal value Created: 24/May/17  Updated: 27/Jul/18  Resolved: 09/Jun/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tanner Ezell Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair-SERVER-29358.tar.gz     File sizeStorer.wt     File storage.bson    
Operating System: ALL
Participants:

 Description   

Running version:

$ mongod --version
db version v3.2.10
git version: 79d9b3ab5ce20f51c272b4411202710a082d0317
OpenSSL version: OpenSSL 1.0.2j 26 Sep 2016
allocator: system
modules: none
build environment:
distarch: x86_64
target_arch: x86_64

The server experienced an unexpected shutdown and since I cannot get mongo to run, I get the following output:

2017-05-24T10:58:34.183-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=37G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-05-24T10:58:34.184-0700 E STORAGE [initandlisten] WiredTiger (0) [1495648714:184033][44975:0x7fff75d69000], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 3070115156 doesn't match expected checksum of 2589322872
2017-05-24T10:58:34.184-0700 E STORAGE [initandlisten] WiredTiger (0) [1495648714:184064][44975:0x7fff75d69000], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-05-24T10:58:34.184-0700 E STORAGE [initandlisten] WiredTiger (-31804) [1495648714:184081][44975:0x7fff75d69000], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-05-24T10:58:34.184-0700 I - [initandlisten] Fatal Assertion 28558

I have tried with --repair to no avail and I could not find any other instructions on how to recover this collection. Please advise.



 Comments   
Comment by Kelsey Schubert [ 09/Jun/17 ]

Hi tannerezell,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Thomas

Comment by Mark Agarunov [ 24/May/17 ]

Hello tannerezell,

Thank you for providing this information. If possible, could you provide the complete logs from MongoDB when this issue occurs? The additional information may help shed some light on the best approach for this issue.

Thanks,
Mark

Comment by Tanner Ezell [ 24/May/17 ]

Hello Mark,

Thank you for your help, I have loaded the files and was able to run a repair. However, one of the collection is missing, filename collection-6--7108862593159262418.wt. Is there a way to restore this collection? I have both the file and the index of it.

1. Local storage, SSD,
3. Yes
4. No
6. No backups or restores of this database have happened yet

Regards

Comment by Mark Agarunov [ 24/May/17 ]

Hello tannerezell,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:20:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.