[SERVER-29323] file:Collection.wt read checksum error, mongodb won't start Created: 23/May/17  Updated: 12/Jul/17  Resolved: 09/Jun/17

Status: Closed
Project: Core Server
Component/s: Querying, WiredTiger
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: DHARANIDHARAN [X] Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair-SERVER-29323.tar.gz    
Operating System: ALL
Participants:

 Description   

Hi,
I tried to start mongoDB server using services. I wasn't able to start it.
Then i checked the logs. Its showing the read checksum error with some file named as collection.
Below is the mongoDB log

2017-05-23T02:31:07.518-0400 W -        [initandlisten] Detected unclean shutdown - E:\Apps\MongoDB\MongoDB\mongoDB\Data\mongod.lock is not empty.
2017-05-23T02:31:07.518-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-05-23T02:31:07.518-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),direct_io=(data),
2017-05-23T02:31:08.846-0400 E STORAGE  [initandlisten] WiredTiger (0) [1495521068:845181][2580:8766840902224], file:collection-8--4378964801540422943.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 344616960: block header checksum of 1316359805 doesn't match expected checksum of 3610363250
2017-05-23T02:31:08.846-0400 E STORAGE  [initandlisten] WiredTiger (0) [1495521068:846228][2580:8766840902224], file:collection-8--4378964801540422943.wt, WT_SESSION.open_cursor: collection-8--4378964801540422943.wt: encountered an illegal file format or internal value
2017-05-23T02:31:08.846-0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1495521068:846228][2580:8766840902224], file:collection-8--4378964801540422943.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-05-23T02:31:08.846-0400 I -        [initandlisten] Fatal Assertion 28558
2017-05-23T02:31:08.846-0400 I -        [initandlisten] 
 
***aborting after fassert() failure

After that, I tried to repair the monogo server using --repair command. Its also Failed with error.
Below is the repair log.

2017-05-23T06:48:51.314-0400 E STORAGE  [initandlisten] WiredTiger (0) [14955365
31:314231][3284:8766840902224], file:collection-22--4378964801540422943.wt, WT_C
URSOR.prev: read checksum error for 8192B block at offset 128155648: block heade
r checksum of 0 doesn't match expected checksum of 4290245144
2017-05-23T06:48:51.318-0400 E STORAGE  [initandlisten] WiredTiger (0) [14955365
31:318313][3284:8766840902224], file:collection-22--4378964801540422943.wt, WT_C
URSOR.prev: collection-22--4378964801540422943.wt: encountered an illegal file f
ormat or internal value
2017-05-23T06:48:51.323-0400 E STORAGE  [initandlisten] WiredTiger (-31804) [149
5536531:323335][3284:8766840902224], file:collection-22--4378964801540422943.wt,
 WT_CURSOR.prev: the process must exit and restart: WT_PANIC: WiredTiger library
 panic
2017-05-23T06:48:51.328-0400 I -        [initandlisten] Fatal Assertion 28558
2017-05-23T06:48:51.330-0400 I -        [initandlisten]
***aborting after fassert() failure



 Comments   
Comment by Kelsey Schubert [ 09/Jun/17 ]

Hi DHARAN993,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Thomas

Comment by Mark Agarunov [ 24/May/17 ]

Hello DHARAN993,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Comment by DHARANIDHARAN [X] [ 24/May/17 ]

Hi,
I have attached the required files.

Comment by Mark Agarunov [ 23/May/17 ]

Hello DHARAN993,

Thank you for the report. If you can provide the WiredTiger.wt and WiredTiger.turtle files we can attempt a repair of the database, but please keep in mind that this is not a guaranteed fix.
Thanks,
Mark

Comment by DHARANIDHARAN [X] [ 23/May/17 ]


I don't know the required files to repair the server.If any files you want, i will attach the corresponding files.

Generated at Thu Feb 08 04:20:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.