[SERVER-25998] journal/WiredTigerLog.0000000222 corrupted Created: 07/Sep/16  Updated: 14/Aug/18  Resolved: 08/Sep/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ashley Mort Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Steps To Reproduce:

happens whenever I try to start mongodb

Participants:

 Description   

Power failure on server now getting this error when I try to start. Repair operation fails.

2016-09-07T15:19:25.542+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1473261565:542419][19988:0x7fba68a4fd40], txn-recover: log file journal/WiredTigerLog.0000000222 corrupted: Bad magic number 740455270: WT_ERROR: non-specific WiredTiger error
2016-09-07T15:19:25.542+0000 E STORAGE [initandlisten] WiredTiger (-31804) [1473261565:542655][19988:0x7fba68a4fd40], txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-09-07T15:19:25.542+0000 I - [initandlisten] Fatal Assertion 28558



 Comments   
Comment by Kelsey Schubert [ 08/Sep/16 ]

Hi mortac8,

I'm glad that deleting the subsequent log files resolved the issue. Since this issue appears to be the result of a filesystem bug, I am going to close this ticket.

It is worth noting that the data in the corrupted file is not compressed. If you are using the default compression settings, WiredTiger did not write the contents of this file.

Non-default compression settings would be most likely specified in your configuration file.

Kind regards,
Thomas

Comment by Ashley Mort [ 08/Sep/16 ]

Deleting all log files except 0021 seems to have resolved my issue. Thanks!

Comment by Ashley Mort [ 07/Sep/16 ]

I believe I am using the default compression, snappy. I'm not positive how to confirm this.
File system is ext4.
I'm not sure if fsck ran on reboot, this system has several users/admins.
I'm not sure how many restarts...I probably attempted 8-10 restarts of mongod.

[root@sandbox journal]# ls -l
total 1331336
rw-rr- 1 mongod mongod 104857728 2016-09-07 09:12 WiredTigerLog.0000000221
rw-rr- 1 mongod mongod 104857728 2016-09-06 13:56 WiredTigerLog.0000000222
rw-rr- 1 mongod mongod 104857728 2016-09-07 13:36 WiredTigerLog.0000000223
rw-rr- 1 mongod mongod 104857728 2016-09-07 13:55 WiredTigerLog.0000000224
rw-rr- 1 mongod mongod 104857728 2016-09-07 14:01 WiredTigerLog.0000000225
rw-rr- 1 mongod mongod 104857728 2016-09-07 14:09 WiredTigerLog.0000000226
rw-rr- 1 mongod mongod 104857728 2016-09-07 14:09 WiredTigerLog.0000000227
rw-rr- 1 root root 104857728 2016-09-07 14:28 WiredTigerLog.0000000228
rw-rr- 1 root root 104857728 2016-09-07 14:29 WiredTigerLog.0000000229
rw-rr- 1 root root 104857728 2016-09-07 14:29 WiredTigerLog.0000000230
rw-rr- 1 root root 104857728 2016-09-07 14:29 WiredTigerLog.0000000231
rw-rr- 1 root root 104857728 2016-09-07 14:39 WiredTigerLog.0000000232
rw-rr- 1 mongod mongod 104857728 2016-09-07 15:19 WiredTigerLog.0000000233

Comment by Kelsey Schubert [ 07/Sep/16 ]

Hi mortac8,

Thank you for uploading the corrupted file. Please note that I have removed the attachment from this ticket in case it contains sensitive data.

Since the uploaded file contains data that is not written to WiredTiger log files, this issue appears to be the result of a filesystem bug following the power failure.

To continue to investigate your system, please answer the following questions:

  • What file system are you running?
  • Did you execute fsck or its equivalent on reboot?

To understand whether this log file was active when the power failure occurred, please provide the following information:

  • The output of ls -l of the journal directory
  • The number of restarts have you attempted since the power failure (each restart creates the next log file)

My recommendation would be to create a copy of your $dbpath and attempt to restart after deleting the corrupted file and all subsequent log files. Please note that if 222 was not the last active log file, removing these files will lose some data, so I would suggest that you first test this procedure on a copy.

Kind regards,
Thomas

Generated at Thu Feb 08 04:10:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.