[SERVER-27416] Fatal Assertion 34433 Created: 14/Dec/16  Updated: 13/Aug/18  Resolved: 20/Jan/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: lakshitha weerakkody Assignee: Michael Cahill (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to WT-3114 Avoid archiving log files immediately... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Storage 2017-01-23, Storage 2017-02-13
Participants:

 Description   

I had this error this morning on my local machine when I trying to start the DB. It says it's because of an unclean shutdown. but i'm unable to recover the DB. --repair also didn't work.

2016-12-14T11:50:42.278+0530 I CONTROL  [initandlisten] MongoDB starting : pid=7
256 port=27017 dbpath=c:\mongodb 64-bit host=EYEPAXLT-57
2016-12-14T11:50:42.279+0530 I CONTROL  [initandlisten] targetMinOS: Windows 7/W
indows Server 2008 R2
2016-12-14T11:50:42.279+0530 I CONTROL  [initandlisten] db version v3.2.10
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] git version: 79d9b3ab5ce
20f51c272b4411202710a082d0317
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL
 1.0.1t-fips  3 May 2016
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] modules: none
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] build environment:
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten]     distmod: 2008plus-ss
l
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten]     distarch: x86_64
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2016-12-14T11:50:42.280+0530 I CONTROL  [initandlisten] options: { storage: { db
Path: "c:\mongodb" } }
2016-12-14T11:50:42.281+0530 I -        [initandlisten] Detected data files in c
:\mongodb created by the 'wiredTiger' storage engine, so setting the active stor
age engine to 'wiredTiger'.
2016-12-14T11:50:42.281+0530 W -        [initandlisten] Detected unclean shutdow
n - c:\mongodb\mongod.lock is not empty.
2016-12-14T11:50:42.282+0530 W STORAGE  [initandlisten] Recovering data from the
 last clean checkpoint.
2016-12-14T11:50:42.282+0530 I STORAGE  [initandlisten] wiredtiger_open config:
create,cache_size=8G,session_max=20000,eviction=(threads_max=4),config_base=fals
e,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snapp
y),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),stati
stics_log=(wait=0),
2016-12-14T11:50:42.555+0530 I -        [initandlisten] Fatal Assertion 34433
2016-12-14T11:50:42.555+0530 I -        [initandlisten]
 
***aborting after fassert() failure



 Comments   
Comment by Michael Cahill (Inactive) [ 10/Feb/17 ]

YouriT, the original assertion reported in this ticket indicates metadata corruption. There can be many causes of such corruption: for example, the filesystem may have been left in an inconsistent state after the failed container shutdown.

Please open a new ticket to request help getting to the bottom of this issue and we can create an upload portal for your data. In the meantime, if the node is part of a replicaset, I recommend resyncing it. If not, it is probably best to restore from a backup.

Comment by Youri Tolstoy [ 09/Feb/17 ]

Hi,

This happened to me today as well.

To give a bit more background to how it happened:

  • Standalone mongodb running in a container
  • Container stop failed somehow leading to a forced (unclean) shutdown
  • The lock file was still there
  • Removed it
  • Still getting this fatal assertion

If it helps, I can provide logs and tarball but I would also need to send it through a private push.

Haven't seen the issue on 3.4 so far.

Comment by Michael Cahill (Inactive) [ 20/Jan/17 ]

lakikeanu, our team has done a forensic analysis of the data you provided, thank you.

We have carefully reviewed the code responsible for creating and dropping collections and indexes with WiredTiger to try to figure out how your database could have ended up in the state it is in. We have not seen these symptoms elsewhere, even with users with unreliable storage systems: our best explanation is that the repair process corrupted some of the metadata.

Given that we can't reproduce this situation and it was a test project, I am going to close this ticket now. If you encounter any further problems using MongoDB, please open a new ticket with the details.

Comment by lakshitha weerakkody [ 22/Dec/16 ]

Hi Thomas,

1. How can I get the logs? As I remember machine got automatically shutdown (for a windows update) while mongodb was running. Robomongo app was also opened and connected to the DB at that time.
2. It Didn't happen again.

Thanks,
Lakshitha.

Comment by Kelsey Schubert [ 19/Dec/16 ]

Hi lakikeanu,

Thanks for the providing the files, they are very helpful. I have a couple of follow up questions as we continue our investigation of this issue.

  1. Would you please provide the logs preceding this issue (including the events prior to the unclean shutdown)?
  2. Are you able to reproduce this behavior in your test project? Has this event only occured once?

Thanks again for your help,
Thomas

Comment by lakshitha weerakkody [ 15/Dec/16 ]

Hi Thomas,

It was a test project so I have uploaded all the files there (https://10gen-httpsupload.s3.amazonaws.com/upload_forms/fadb4f0e-2c7e-4865-9f03-5759067bafc5.html) . so you can take a good look in to that.
I'm bit doubtful whether I can move forward with MongoDB (WiredTiger storage engine) if this kind of issues come across. I hope there will be a solutions or future release will handle such issues.

Let me know if you find anything.

Thanks,
Lakshitha.

Comment by Kelsey Schubert [ 14/Dec/16 ]

Hi lakikeanu,

Thank you for reporting this issue. To help us investigate the root cause of this corruption, please upload the following:

  1. tarball of the WiredTiger files (_mdb_catalog.wt, sizeStorer.wt, WiredTiger* files)
  2. output of ls -l of the database directory
  3. WiredTigerLog.<long number>

Finally, please consider creating a backup of the current $dbpath of the affected node as we may need to examine additional files as our ing.

For your convenience, I have have created a secure upload portal for you to use. Files uploaded to this portal only be visible to MongoDB employees investigating this issue and are routinely deleted after some time.

My understanding is that is node is a standalone. In this case, my only reccomendation to resolve this issue would to restore from a backup.

Kind regards,
Thomas

Generated at Thu Feb 08 04:15:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.