[SERVER-36480] WiredTiger error Created: 07/Aug/18  Updated: 04/Nov/18  Resolved: 01/Oct/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zhang Meng Assignee: Nick Brewer
Resolution: Incomplete Votes: 0
Labels: envns, rge, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.tar.gz    
Operating System: Linux
Participants:

 Description   

my mongo DB crashed suddenly and I have no idea how, when I tried to restart it, the error appears:

 

2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] db version v3.6.0
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] git version: a57d8e71e6998a2d0afde7edc11bd23e5661c915
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] allocator: tcmalloc
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] modules: none
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] build environment:
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] distmod: rhel70
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] distarch: x86_64
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] target_arch: x86_64
2018-08-06T22:28:32.699-0700 I CONTROL [initandlisten] options: { net:

{ bindIp: "127.0.0.1,10.196.182.32", port: 27018 }

, storage: { dbPath: "./db" } }
2018-08-06T22:28:32.721-0700 W - [initandlisten] Detected unclean shutdown - ./db/mongod.lock is not empty.
2018-08-06T22:28:32.722-0700 I - [initandlisten] Detected data files in ./db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-08-06T22:28:32.722-0700 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2018-08-06T22:28:32.722-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=128318M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-08-06T22:28:32.736-0700 E STORAGE [initandlisten] WiredTiger error (0) [1533619712:736794][22136:0x7f6707bd5b00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 291)
2018-08-06T22:28:32.736-0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1533619712:736824][22136:0x7f6707bd5b00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-08-06T22:28:32.736-0700 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-08-06T22:28:32.736-0700 F - [initandlisten]

***aborting after fassert() failure

 

the attached is the WiredTiger related files, I tried to repair them with: 

mongod --storageEngine wiredTiger --dbpath ./db --repair --port 27018

but it did not work, could you please help to take a look?

Thanks!

 



 Comments   
Comment by Nick Brewer [ 01/Oct/18 ]

mzhang11 Since there's been no activity on this ticket in some time, I'm going to close it. If you have a chance to update this ticket with the requested platform information, it would greatly assist in our tracking of this issue.

Thanks,
-Nick

Comment by Nick Brewer [ 14/Sep/18 ]

mzhang11 So that we can better categorize this issue, can you confirm if this machine is a VM, container, native hardware, etc?

Thanks,
-Nick

Comment by Nick Brewer [ 08/Aug/18 ]

mzhang11 There's an internal tool that we use to fix specific instances of WiredTiger metadata file corruption - it's not something we make publicly available at the moment as it can easily be misused.

Is this a VM, container, etc? 

Thanks,

Nick

Comment by Zhang Meng [ 08/Aug/18 ]

Thanks for your reply, Nick.

the power and the storage was fine(another mongo process on the same server was fine), and the log was overwritten when I tried to restart it, so I'm not sure about the root cause...

in fact, I have deleted all the DB files, and recovered the DB with the latest backup.

could you please share something about how to repair the wierdTiger files, maybe next time I can try to repair it myself.

Comment by Nick Brewer [ 07/Aug/18 ]

mzhang11 I can attempt to repair the files - before I do, I'd like to confirm:

  • The underlying cause of the crash (power failure, disk corruption, etc)
  • The environment (native, virtualized, container)

Thanks,
Nick

Generated at Thu Feb 08 04:43:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.