[SERVER-56177] WiredTiger.wt: encountered an illegal file format or internal value Created: 19/Apr/21  Updated: 28/Apr/21  Resolved: 28/Apr/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Flavio Silveira Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive mongo_logs.zip     Zip Archive mongodb.zip    
Operating System: ALL
Participants:

 Description   

Hello, so after a bad server/mongod shutdown, I am unable to restart Mongod. I'm 99% confident that my issue is that same as This one.

Like the above thread, I've uploaded my WiredTiger.* files in the hopes that someone can respond with fixed WiredTiger.* files. I would be extremely grateful...thank you!

If anyone has any further questions, feel free to ask. Thank you!!

 

https://jira.mongodb.org/browse/SERVER-42601



 Comments   
Comment by Edwin Zhou [ 28/Apr/21 ]

Hi flavio.br@gmail.com,

2021-04-15T11:08:06.659-0300 I NETWORK  [conn77] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47198] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn153] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47350] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn50] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47140] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn131] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47306] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn121] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47286] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn75] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47194] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn145] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47334] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn126] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47296] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn95] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47234] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn43] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47126] 
2021-04-15T11:08:06.659-0300 I NETWORK  [conn89] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:47222] 
2021-04-15T11:20:35.878-0300 I NETWORK  [conn167] end connection 127.0.0.1:47418 (15 connections now open)
2021-04-15T11:20:35.881-0300 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:47422 #169 (16 connections now open)
2021-04-15T13:00:17.779-0300 I COMMAND  [PeriodicTaskRunner] task: UnusedLockCleaner took: 232817ms
2021-04-15T15:04:07.830-0300 W NETWORK  [HostnameCanonicalizationWorker] Failed to obtain name info for: [ (10.2.97.51, "Name or service not known"), (10.2.97.51, "Name or service not known"), (10.2.97.51, "Name or service not known") ]
2021-04-15T16:05:48.932-0300 E STORAGE  [thread1] WiredTiger (0) [1618513548:904348][29594:0x7f782c290700], file:index-9--7114367227520222323.wt, WT_SESSION.checkpoint: read checksum error for 4096B block at offset 147456: block header checksum of 0 doesn't match expected checksum of 2317450009
2021-04-15T18:01:51.506-0300 E STORAGE  [thread1] WiredTiger (0) [1618520511:506061][29594:0x7f782c290700], file:index-9--7114367227520222323.wt, WT_SESSION.checkpoint: index-9--7114367227520222323.wt: encountered an illegal file format or internal value
2021-04-15T18:01:51.506-0300 E STORAGE  [thread1] WiredTiger (-31804) [1618520511:506117][29594:0x7f782c290700], file:index-9--7114367227520222323.wt, WT_SESSION.checkpoint: the process must exit and restart: WT_PANIC: WiredTiger library panic
2021-04-15T18:01:51.506-0300 I -        [thread1] Fatal Assertion 28558
2021-04-15T18:01:51.506-0300 I -        [thread1] 

The error messages in the logs you provided us leads us to believe you're running into some physical corruption.

Unfortunately, --repair was unable to repair this type of corruption.

To avoid a problem like this in the future, it is our strong recommendation to:

Best,
Edwin

Comment by Flavio Silveira [ 28/Apr/21 ]

Hi Edwin, thank you for your support!

I upgraded mongo to version 4.4.5, and tried to repair, with no success.

I'm sending the requested logs attached.

Thanks again,

Flávio

mongo_logs.zip

Comment by Edwin Zhou [ 26/Apr/21 ]

Hi flavio.br@gmail.com,

Please make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath.

Since MongoDB v4.0.3, running -repair will also perform the same WiredTiger metadata repair that was used in SERVER-42601. We recommend upgrading your MongoDB version to 4.0.3 or later and attempting -repair.

In the event that a --repair operation is unsuccessful, then please also provide:

  • The logs leading up to the first occurrence of any issue
  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Best,
Edwin

Generated at Thu Feb 08 05:38:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.