[SERVER-24973] WiredTiger library panic Created: 11/Jul/16  Updated: 14/Aug/18  Resolved: 11/Jul/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Stefan Kutzke Assignee: Unassigned
Resolution: Done Votes: 0
Labels: envns, rdi, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File crash.log     Text File mongod.conf     HTML File stats_affected_collection    
Operating System: Linux
Steps To Reproduce:

unknown

Participants:

 Description   

Primary mongod crashed with a "WiredTiger library panic" error.

Setup consist of 3 nodes (1x Primary, 2x Secondary). Operating system is CentOS 6.7 (x86_64). Machines have 16 CPU cores and 60 GB RAM, WiredTiger cache size is 32 GB.

Attachments: crash log, mongod config and stats of affected collection.



 Comments   
Comment by Kelsey Schubert [ 11/Jul/16 ]

Hi skutzke,

Thank you for clarifying that there was a disk failure in your storage layer. Since you have been able to successfully restore your database, I will close this ticket.

Kind regards,
Thomas

Comment by Stefan Kutzke [ 11/Jul/16 ]

Hi Ramon,
you are right. In the meantime I have found out that there was a disk failure in the underlaying RAID5 storage.
I restored the complete database and therefore did not try running validate().
Thank you for your prompt response.
Stefan

Comment by Ramon Fernandez Marina [ 11/Jul/16 ]

Below are the first lines for the log file above for convenience:

2016-07-09T05:05:54.765+0200 E STORAGE  [conn120402] WiredTiger (0) [1468033554:765166][6134:0x7f7b360de700], file:backend/collection/49--8667964377448587020.wt, cursor.search: read checksum error for 12288B block at offset 60156526592: calculated block checksum of 1384899899 doesn't match expected checksum of 3706134454
2016-07-09T05:05:54.765+0200 E STORAGE  [conn120402] WiredTiger (0) [1468033554:765280][6134:0x7f7b360de700], file:backend/collection/49--8667964377448587020.wt, cursor.search: backend/collection/49--8667964377448587020.wt: encountered an illegal file format or internal value
2016-07-09T05:05:54.765+0200 E STORAGE  [conn120402] WiredTiger (-31804) [1468033554:765314][6134:0x7f7b360de700], file:backend/collection/49--8667964377448587020.wt, cursor.search: the process must exit and restart: WT_PANIC: WiredTiger library panic

skutzke, these errors indicate that MongoDB has found data corruption on your files, and is terminating to avoid any other damage. This type of corruption is often cause by a faulty storage layer, so I'd recommend you check the integrity of your disks.

Have you tried running validate() on this collection?

Generated at Thu Feb 08 04:07:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.