[SERVER-33536] Corrupted WiredTigerLog after server crash Created: 28/Feb/18  Updated: 02/Apr/18  Resolved: 06/Mar/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: afi Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I can't start the mongo sever after crash.
Whan I deleted all files from journal directory and run
{{ mongod --dbpath /var/lib/mongodb --repair}} it started but it seems to some data lost

is there a way to fix journal/WiredTigerLog.0000000311?
Thanx

{{----- END BACKTRACE -----
2018-02-28T08:17:53.586+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] MongoDB starting : pid=500 port=27017 dbpath=/var/lib/mongodb 64-bit host=DB-i-4c9a67c1
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] db version v3.4.6
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] git version: c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] allocator: tcmalloc
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] modules: none
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] build environment:
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] distmod: debian81
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] distarch: x86_64
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] target_arch: x86_64
2018-02-28T08:17:53.624+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ port: 27017 }

, storage: { dbPath: "/var/lib/mongodb", journal:

{ enabled: true }

}, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }

}
2018-02-28T08:17:53.631+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-02-28T08:17:53.684+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-02-28T08:17:53.684+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2018-02-28T08:17:53.684+0300 I STORAGE [initandlisten]
2018-02-28T08:17:53.684+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-02-28T08:17:53.684+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-02-28T08:17:53.684+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3490M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-02-28T08:17:53.798+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1519795073:798660][500:0x7f854899cd00], txn-recover: log file journal/WiredTigerLog.0000000311 corrupted: Bad magic number 0: WT_ERROR: non-specific WiredTiger error
2018-02-28T08:17:53.798+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1519795073:798763][500:0x7f854899cd00], txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-02-28T08:17:53.798+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-02-28T08:17:53.798+0300 I - [initandlisten]

***aborting after fassert() failure
}}



 Comments   
Comment by afi [ 12/Mar/18 ]

Thanks for help

Comment by Kelsey Schubert [ 06/Mar/18 ]

Hi afi,

Unfortunately, repair of the journal file is not feasible. If you're unable to resync from another node, my advice would be to proceed with starting mongod without the journal files numbered 0000000311 or above. This should reduce the amount of data lost.

Please note the following recommendations to help mitigate any issues related to unreliable storage layers or server failures.

Kind regards,
Kelsey

Generated at Thu Feb 08 04:33:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.