[SERVER-41415] mongodb start failed,encountered an illegal file format or internal value. Created: 31/May/19  Updated: 04/Jun/19  Resolved: 04/Jun/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.6.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: lyswhut Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Cent OS 6


Attachments: File SERVER-41415_repair_attempt.tgz     HTML File WiredTiger     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File _mdb_catalog.wt     Text File mongod.log    
Operating System: ALL
Participants:

 Description   

Hi,My MongoDB can't start, I tried to fix it with this solution: https://superuser.com/questions/914215/mongod-start-failed-monogdb-3-0-2-database-storageengine-wiredtiger, but still failed.

2019-05-31T12:04:16.689+0800 I CONTROL [initandlisten] MongoDB starting : pid=2225 port=27017 dbpath=/var/lib/mongo 64-bit host=172.96.228.50.16clouds.com
2019-05-31T12:04:16.689+0800 I CONTROL [initandlisten] db version v3.6.8
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] git version: 6bc9ed599c3fa164703346a22bad17e33fa913e4
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] modules: none
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] build environment:
2019-05-31T12:04:16.690+0800 I CONTROL [initandlisten] distmod: rhel62
2019-05-31T12:04:16.691+0800 I CONTROL [initandlisten] distarch: x86_64
2019-05-31T12:04:16.691+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-05-31T12:04:16.691+0800 I CONTROL [initandlisten] options: \{ repair: true, storage: { dbPath: "/var/lib/mongo", engine: "wiredTiger" } }
2019-05-31T12:04:16.691+0800 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2019-05-31T12:04:16.692+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-05-31T12:04:16.692+0800 I STORAGE [initandlisten] 
2019-05-31T12:04:16.692+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-05-31T12:04:16.692+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-05-31T12:04:16.693+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2019-05-31T12:04:16.693+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-05-31T12:04:17.468+0800 E STORAGE [initandlisten] WiredTiger error (0) [1559275457:468398][2225:0x7f02b81ada80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 333)
2019-05-31T12:04:17.468+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1559275457:468767][2225:0x7f02b81ada80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-05-31T12:04:17.468+0800 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2019-05-31T12:04:17.469+0800 F - [initandlisten]
 
***aborting after fassert() failure

I attached the wiredtiger files.

If some could help me I would be extremely grateful.
Thank you.



 Comments   
Comment by lyswhut [ 04/Jun/19 ]

Roger that! Thank you for your advice!

Comment by Eric Sedor [ 04/Jun/19 ]

Glad to hear! Be sure to check out this backup documentation.

Other recommendations to avoid a problem like in the future are:

Comment by lyswhut [ 04/Jun/19 ]

@eric.sedor, thank you very much for your help, the repair file you provided successfully repaired it, it is now working again.

My server is a VPS. I have this problem after I use the snapshot to restore the VPS. The server is running when the snapshot is taken.

Thanks again   !

Comment by Eric Sedor [ 03/Jun/19 ]

lyshwut, this error message leads us to suspect corruption in the wiredTiger.wt file. Please make a complete copy of the database's $dbpath directory to work off of and safeguard the current $dbpath.

Our ability to determine the source of this corruption depends greatly on your ability to provide:

  1. The logs for the affected node, including before, leading up to, and after the first sign of corruption.
  2. A description of the underlying storage mechanism in use, including details like:
    1. What file system and/or volume management system is in use?
    2. Is data storage locally attached or network-attached?
    3. Are disks RAIDed and if so how?
    4. Are disks SSDs or HDDs?
  3. A description of your backup method, if any.
  4. A description of your disks have been recently checked for integrity?
  5. A history of the deployment, including:
    1. a timeline of version changes
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes
    3. a timeline of disaster recovery or backup restoration activities
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation.
      The ideal resolution is to perform a clean resync from an unaffected node.

That said, I've attached a repair attempt of the files you provided as SERVER-41415_repair_attempt.tgz. Please extract these files, replace them in your $dbpath and let us know if it resolves your issue. Please keep in mind that this repair effort may not be successful.

Generated at Thu Feb 08 04:57:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.