[SERVER-29415] Unable to start WiredTiger service due to corruption in WiredTiger.wt Created: 01/Jun/17  Updated: 14/Aug/18  Resolved: 03/Jun/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Eran Yaffe Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envm, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows server 2012 R2 + VMware Horizon Mirage 5.4


Attachments: Text File WT_salvage_log.txt     File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Operating System: Windows
Participants:

 Description   

2017-05-29T07:48:10.150+0000 I CONTROL  ***** SERVER RESTARTED *****2017-05-29T07:48:10.693+0000 I CONTROL  Trying to start Windows service 'MongoDB'2017-05-29T07:48:10.696+0000 I STORAGE  Service running2017-05-29T07:48:10.738+0000 W -        [initandlisten] Detected unclean shutdown - \\AZ-MRGSTR-PAC01\M$\MirageStorage\NonSis\MongoData\mongod.lock is not empty.2017-05-29T07:48:10.741+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.2017-05-29T07:48:10.742+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),2017-05-29T07:48:10.775+0000 E STORAGE  [initandlisten] WiredTiger (0) [1496044090:775502][5668:140711126897472], file:WiredTiger.wt, connection: read checksum error [4096B @ 28672, 336979136 != 806227696]2017-05-29T07:48:10.775+0000 E STORAGE  [initandlisten] WiredTiger (0) [1496044090:775502][5668:140711126897472], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value2017-05-29T07:48:10.775+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1496044090:775502][5668:140711126897472], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic2017-05-29T07:48:10.775+0000 I -        [initandlisten] Fatal Assertion 285582017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x141cf32017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0xfaf172017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0xed7c02017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x149952017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x41923c2017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x418da22017-05-29T07:48:10.943+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x4193ee2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3a89972017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3a8bde2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3b4a5f2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3b39f32017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3b38a32017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3d02fb2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3d0cbf2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x41832f2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3fc8052017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x3ce65a2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0xc832017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x13f2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      mongo::parseNumberFromStringWithBase<unsigned __int64>+0x1f2abf2017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x43bde82017-05-29T07:48:10.944+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x43d5672017-05-29T07:48:10.945+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x43d6252017-05-29T07:48:10.945+0000 I CONTROL  [initandlisten] mongod.exe      index_collator_extension+0x11c9802017-05-29T07:48:10.945+0000 I CONTROL  [initandlisten] sechost.dll     GetIdentityProviderInfoByGUID+0x2312017-05-29T07:48:10.945+0000 I CONTROL  [initandlisten] KERNEL32.DLL    BaseThreadInitThunk+0xd2017-05-29T07:48:10.945+0000 I CONTROL  [initandlisten] 2017-05-29T07:48:10.945+0000 I -        [initandlisten] ***aborting after fassert() failure



 Comments   
Comment by Ramon Fernandez Marina [ 03/Jun/17 ]

Thanks for letting us know, and glad to hear you're up and running again. I'd recommend using replication and keeping backups of your data.

Regards,
Ramón.

Comment by Eran Yaffe [ 02/Jun/17 ]

Hi Thomas,
After replacing the files the service started running and it looks like the system is working.
Thank you for your quick assistance.
If you still require answers to your questions please let me know.
Regards,
Eran.

Comment by Kelsey Schubert [ 01/Jun/17 ]

Hello fezzik12,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thank you,
Thomas

Comment by Eran Yaffe [ 01/Jun/17 ]

Hi Thomas,
thank you for handling my case.
just to do some correction in my subject.
the "VMware Mirage MongoDB" fails to start due to the problem with the file.
this is a production system that doesn't work and holds hundreds of images so the situation is very critical.
please update me if you need any more information or logs.
Thanks,
Eran Yaffe

Generated at Thu Feb 08 04:20:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.