[SERVER-37044] unable to start mongodb server, mongod --repair fails Created: 07/Sep/18  Updated: 06/Dec/22  Resolved: 25/Oct/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sarah jones Assignee: Backlog - Triage Team
Resolution: Done Votes: 0
Labels: envm, rpo, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTigerFiles.tar.gz     Text File mongod.log    
Assigned Teams:
Server Triage
Operating System: Linux
Participants:

 Description   

Last night our mongodb server was crashed with the error message

" E STORAGE [WTCheckpointThread] WiredTiger error (0) [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293) Raw: [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293)"

Then I tried to restart the server and also run the mongod --repair command but I always get the same error. The log file and wire tiger files are attached. Thanks in advance for any help



 Comments   
Comment by Nick Brewer [ 01/Oct/18 ]

afshan886@gmail.com Have you had a chance to look into whether another process may be manipulating dbpath files while the mongod is running?

Thanks,
-Nick

Comment by Nick Brewer [ 14/Sep/18 ]

afshan886@gmail.com Glad to hear you were able to get it working from a backup. Looking at your logs, it appears that you've seen a few different errors in the past related to the WiredTiger.turtle file, some of them permissions-based, and some of them most likely due to corruption:

2018-08-27T21:58:52.084+0200 E STORAGE  [WTCheckpointThread] WiredTiger error (103) [1535399932:84754][463:0x7fe33e567700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle.set: handle-open: open: Software caused connection abort Raw: [1535399932:84754][463:0x7fe33e567700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle.set: handle-open: open: Software caused connection abort

2018-09-06T23:56:50.760+0200 E STORAGE  [WTCheckpointThread] WiredTiger error (0) [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 293)

2018-08-16T12:23:46.219+0200 E STORAGE  [initandlisten] WiredTiger error (13) [1534415026:219001][4273:0x7f33e59029c0], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied

I'm curious if there is another process that may be attempting to manipulate these files while the mongod is running?

Thanks,
-Nick

Comment by sarah jones [ 14/Sep/18 ]

nick.brewer Thanks for your reply. Okay I restored the database from a backup. The answers to your questions are:

  • the cause of the issue was unexpected shutdown
  • the underlying platform is virtual machine

Thanks again

Comment by Nick Brewer [ 11/Sep/18 ]

afshan886@gmail.com It appears that your WiredTiger.turtle file is badly corrupted - as this file contains metadata that is used to interpret other WiredTiger files, it will not be possible to proceed with a repair using the file in this state. If you have a backup of your dbpath available I would recommend that you restore from it to get you database into working condition.

Some other things I'd like to confirm:

  • What was the cause of this issue (power outage, unexpected shutdown, filesystem issues, etc)?
  • What is the underlying platform (virtual machine, container, native hardware, etc)?

Thanks,
-Nick

Generated at Thu Feb 08 04:44:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.