[SERVER-26361] WiredTiger.wt read checksum error Created: 27/Sep/16  Updated: 13/Aug/18  Resolved: 28/Sep/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.0.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ricardo Ferreira Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rns, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     Text File mongod.log     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi,
I've a problem with mongodb. I stop the service and I wasn't able to start it again.

When I try to start I've the following errors:

2016-09-28T01:23:07.264+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),statistics=(fast),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-09-28T01:23:07.276+0800 E STORAGE  [initandlisten] WiredTiger (0) [1474996987:276311][18976:0x7fbc41f32c20], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 16384: block header checksum of 2753424562 doesn't match expected checksum of 4062939573
2016-09-28T01:23:07.276+0800 E STORAGE  [initandlisten] WiredTiger (0) [1474996987:276409][18976:0x7fbc41f32c20], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-09-28T01:23:07.276+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1474996987:276443][18976:0x7fbc41f32c20], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-09-28T01:23:07.276+0800 I -        [initandlisten] Fatal Assertion 28558
2016-09-28T01:23:07.300+0800 I CONTROL  [initandlisten]
 0xf75549 0xf12271 0xef60f1 0xd93f8a 0x13ab369 0x13ab525 0x13ab9c4 0x12fe3d2 0x131811c 0x1315ef8 0x1316d52 0x1340b4b 0x13aa50b 0x13780ab 0x133e207 0xd7e61c 0xd7c3a8 0xa7dc5d 0x7f57a2 0x7fa739 0x3d2f41ed1d 0x7f3549

Could you please help to recover my database, it's very important.
I attached two file where I think is the problem.



 Comments   
Comment by Kelsey Schubert [ 28/Sep/16 ]

Hi rjferreira,

Thanks for the update, I'm glad the repair attempt was successful.

Regards,
Thomas

Comment by Ricardo Ferreira [ 28/Sep/16 ]

Hi Thomas,

It solved the problem.
Thanks a lot.

Regards,
Ricardo

Comment by Kelsey Schubert [ 27/Sep/16 ]

Hi rjferreira,

Thank you for the additional information! I've attached repair_attempt.tar.gz. Please extract it and replace the repaired files in your $dbpath and attempt to restart the mongod.

Kind regards,
Thomas

Comment by Ricardo Ferreira [ 27/Sep/16 ]

Here the answers:
1. Yes,The journaling was enable.
2. I'm not able to answer this question. I just know we are using HDD's on a remote storage. I think we have RAID but I don't know the configuration.
3. I stopped the server with command db.shutdownServer();
4. I attached the logs. This happened during an upgrade from version 3.0.6 to version 3.2.8. After I updated the mongodb binaries with rpm process it changed permissions/owner dbpath directory. So I fixed the permissions problem but mongodb never stated again. I've already tried the repair without success.

Comment by Kelsey Schubert [ 27/Sep/16 ]

Hi rjferreira,

I'm working on a repair attempt, in the meantime, would you please answer the following questions so we get a better idea of what happened here?

  1. Are you using journaling?
  2. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  3. Would you please clarify how you stopped the mongod?
  4. Would you please upload the complete log, showing the shutdown that preceded the corruption?

Thank you,
Thomas

Generated at Thu Feb 08 04:11:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.