[SERVER-53300] WiredTiger error on Startup Created: 09/Dec/20  Updated: 16/Mar/21  Resolved: 16/Mar/21

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rodrigo Cadena Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello! After a power shortage we are not able to get our mongoDB database online anymore. 

It's huge database with .wt file of 200MB. 

 

Is there any chance to recover this DB?

 

2020-12-09T10:50:15.628-0300 I CONTROL  [main] ***** SERVER RESTARTED *****

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] db version v3.4.24

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] git version: 865b4f6a96d0f5425e39a18337105f33e8db504d

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] allocator: tcmalloc

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] modules: none

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] build environment:

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten]     distmod: ubuntu1404

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten]     distarch: x86_64

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten]     target_arch: x86_64

2020-12-09T10:50:15.634-0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net:

{ port: 27017 }

, storage: { dbPath: "*****", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongod.log" } }

2020-12-09T10:50:15.653-0300 I -        [initandlisten] Detected data files in /home/seahorse/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2020-12-09T10:50:15.653-0300 I STORAGE  [initandlisten] 

2020-12-09T10:50:15.654-0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine

2020-12-09T10:50:15.654-0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem

2020-12-09T10:50:15.654-0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15571M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),

2020-12-09T10:50:15.670-0300 E STORAGE  [initandlisten] WiredTiger error (0) [1607521815:670795][42233:0x7f171e2f8d80], file:WiredTiger.wt, connection: read checksum error for 8192B block at offset 2236416: block header checksum of 1886085989 doesn't match expected checksum of 190355549

2020-12-09T10:50:15.670-0300 E STORAGE  [initandlisten] WiredTiger error (0) [1607521815:670892][42233:0x7f171e2f8d80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value

2020-12-09T10:50:15.670-0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1607521815:670933][42233:0x7f171e2f8d80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic

2020-12-09T10:50:15.670-0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365

2020-12-09T10:50:15.671-0300 I -        [initandlisten] 

 

***aborting after fassert() failure

 

 

2020-12-09T10:50:15.700-0300 F -        [initandlisten] Got signal: 6 (Aborted).



 Comments   
Comment by Eric Sedor [ 16/Mar/21 ]

Understood rodrigo.cadena@eletromidia.com.br, thank you!

Comment by Rodrigo Cadena [ 16/Mar/21 ]

Thanks, Eric! The --repair did not work but we were able to restore the database from a older backup and it's working fine now. 

Best,

Comment by Eric Sedor [ 15/Mar/21 ]

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please let us know the results of --repair?

Thanks,
Eric

Comment by Eric Sedor [ 16/Dec/20 ]

Hi rodrigo.cadena@eletromidia.com.br,

MongoDB 3.4 reached end of life in January of 2020. But we can provide limited guidance on this issue. The ideal resolution is to perform a clean resync from an unaffected node.

First, make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath. Then, try mongod --repair using the latest version of MongoDB.

Sincerely,
Eric

Generated at Thu Feb 08 05:30:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.