[SERVER-26103] WiredTiger.wt: encountered an illegal file format or internal value Created: 14/Sep/16  Updated: 14/Oct/21  Resolved: 15/Sep/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Matt Price Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log     File repair.tar     File repair_attempt.tar.gz     Zip Archive wiredtiger.zip    
Operating System: Linux
Participants:
Case:

 Description   

Hello, so after a bad server/mongod shutdown, I am unable to restart Mongod. I'm 99% confident that my issue is that same as This one.

Like the above thread, I've uploaded my WiredTiger.* files in the hopes that someone can respond with fixed WiredTiger.* files. I would be extremely grateful...thank you!

If anyone has any further questions, feel free to ask. Thank you!!



 Comments   
Comment by Edwin Zhou [ 14/Oct/21 ]

Hi pinoycao1@gmail.com,

Can you please file a new SERVER ticket with your issue so we can better track your particular case?

Thanks!
Edwin

Comment by pinoy cao [ 11/Oct/21 ]

Hi Thomas @kelsey.schubert,

I have the same problem with mongo v3.4 now,I have uploaded  the complete logs of the affected mongod mongod.logand the WiredTiger.wt/WiredTiger.turtle filesrepair.tar。Can u help me to remove the corruption?I look forward to your reply。

Kind regards,

Pinoy

Comment by Kelsey Schubert [ 15/Sep/16 ]

Hi mattprice09,

I'm glad the repair was successful!

Unfortunately, I cannot share the procedure to repair these files at this time, but I can point you to an open ticket to make the --repair operation more robust with WiredTiger. Please feel free to vote for SERVER-19815 and watch it for updates.

Kind regards,
Thomas

Comment by Matt Price [ 15/Sep/16 ]

That worked! Everything is all up and running now. Thanks a ton!!

I'm not sure if this is something you can share or not, but what did you do to the files to remove the corruption?

Have a good one,
Matt

Comment by Kelsey Schubert [ 15/Sep/16 ]

Hi mattprice09,

I've attached repair_attempt.tar.gz containing my repair attempt of the WiredTiger.wt and WiredTiger.turtle files. Please replace these files in your $dbpath and attempt to restart. If there is no additional corruption present your mongod instance should start successfully.

Thank you,
Thomas

Comment by Kelsey Schubert [ 15/Sep/16 ]

Thanks for the additional information, mattprice09. I'm reopening to take another look.

Comment by Matt Price [ 15/Sep/16 ]

Hi Thomas,

After double-checking permissions of my .sock file in /tmp/.sock and running `sudo mongod`, I still get the following error:

ubuntu@ip-172-31-13-38:~$ sudo mongod
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] MongoDB starting : pid=1877 port=27017 dbpath=/data/db 64-bit host=ip-172-31-13-38
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] db version v3.2.7
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] allocator: tcmalloc
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] modules: none
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] build environment:
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] distmod: ubuntu1404
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] distarch: x86_64
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] target_arch: x86_64
2016-09-14T17:43:27.843-1000 I CONTROL [initandlisten] options: {}
2016-09-14T17:43:27.861-1000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-09-14T17:43:27.861-1000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=34G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-09-14T17:43:27.869-1000 E STORAGE [initandlisten] WiredTiger (0) [1473911007:868991][1877:0x7fa91b535cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 106496: block header checksum of 1848638174 doesn't match expected checksum of 2868547657
2016-09-14T17:43:27.869-1000 E STORAGE [initandlisten] WiredTiger (0) [1473911007:869041][1877:0x7fa91b535cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-09-14T17:43:27.869-1000 E STORAGE [initandlisten] WiredTiger (-31804) [1473911007:869057][1877:0x7fa91b535cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-09-14T17:43:27.869-1000 I - [initandlisten] Fatal Assertion 28558
2016-09-14T17:43:27.869-1000 I - [initandlisten]

***aborting after fassert() failure

2016-09-14T17:43:27.883-1000 F - [initandlisten] Got signal: 6 (Aborted).

Thanks Thomas,
Matt

Comment by Kelsey Schubert [ 15/Sep/16 ]

Hi mattprice09,

Thank you for uploading the logs. I have examined them and do not see any evidence of WiredTiger corruption. This issue is unrelated to SERVER-18448, which was fixed in MongoDB 3.0.4 (SERVER-18316).

The fatal assertion you are seeing, 28578, instead indicates that the mongod is unable to unlink the socket file. I would recommend ensuring that the permissions on the socket file are correct to resolve this issue.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Matt Price [ 14/Sep/16 ]

Thanks for the response, Thomas.

I just uploaded the mongod.log file to the portal you linked.

-Matt

Comment by Kelsey Schubert [ 14/Sep/16 ]

Hi mattprice09,

Thanks for reporting this issue. Would you please upload the complete logs of the affected mongod so we can investigate this issue?

Please be sure to include the events preceding the unclean shutdown and subsequent attempts to restart. I have created a secure upload portal here for you to use. Files uploaded to this portal will only be visible to MongoDB employees and are routinely deleted after some time.

Thank you,
Thomas

Generated at Thu Feb 08 04:11:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.