[SERVER-32069] Fatal Assertion 28558 Created: 22/Nov/17  Updated: 05/May/18  Resolved: 29/Nov/17

Status: Closed
Project: Core Server
Component/s: Admin, Diagnostics, WiredTiger
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Moskalenko Alexey [X] Assignee: Josef Ahmad
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File MongoDB.log     File MongoDB.log.2017-10-26T01-29-33     File WiredTiger.turtle     File WiredTiger.wt     Text File command.txt     File repair-SERVER-32069.tar.gz    
Operating System: ALL
Steps To Reproduce:

Repair or strart -

Participants:

 Description   

Hi! After some failure i tried to restore my MATA server( Advanced Threat Analytics) from snapshot,but MOngoDB service wont start ar all, in services - 1067 error appears, and in cmd it cannot be repaired - it fails with this:

C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin>mongod -
-dbpath "C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin
\data" --repair
2017-11-22T17:59:46.325+0300 I CONTROL  [initandlisten] MongoDB starting : pid=3
860 port=27017 dbpath=C:\Program Files\Microsoft Advanced Threat Analytics\Cente
r\MongoDB\bin\data 64-bit host=MSK02MATA01
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] targetMinOS: Windows 7/W
indows Server 2008 R2
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] db version v3.4.2
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] git version: 3f76e40c105
fc223b3e5aac3e20dcd026b83b38b
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] modules: none
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] build environment:
2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten]     distmod: 2008plus
2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten]     distarch: x86_64
2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten] options: { repair: true,
 storage: { dbPath: "C:\Program Files\Microsoft Advanced Threat Analytics\Center
\MongoDB\bin\data" } }
2017-11-22T17:59:46.327+0300 I -        [initandlisten] Detected data files in C
:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin\data crea
ted by the 'wiredTiger' storage engine, so setting the active storage engine to
'wiredTiger'.
2017-11-22T17:59:46.328+0300 I STORAGE  [initandlisten] Detected WT journal file
s.  Running recovery from last checkpoint.
2017-11-22T17:59:46.329+0300 I STORAGE  [initandlisten] journal to nojournal tra
nsition config: create,cache_size=24063M,session_max=20000,eviction=(threads_max
=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=jour
nal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60
,log_size=2GB),statistics_log=(wait=0),
2017-11-22T17:59:46.344+0300 E STORAGE  [initandlisten] WiredTiger error (0) [15
11362786:343919][3860:140721647718528], file:WiredTiger.wt, connection: WiredTig
er.turtle: encountered an illegal file format or internal value
2017-11-22T17:59:46.345+0300 E STORAGE  [initandlisten] WiredTiger error (-31804
) [1511362786:344920][3860:140721647718528], file:WiredTiger.wt, connection: the
 process must exit and restart: WT_PANIC: WiredTiger library panic
2017-11-22T17:59:46.345+0300 I -        [initandlisten] Fatal Assertion 28558 at
 src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2017-11-22T17:59:46.346+0300 I -        [initandlisten]
 
***aborting after fassert() failure

Any suggestoin how i can fix it? I already know that some topics here have similar issue. What files do i need to upload to fix?



 Comments   
Comment by Moskalenko Alexey [X] [ 30/Nov/17 ]

How can it be? After this snapshot was taken - MATA server and MOngoDB were working fine. Issue with server happend about 2 weeks after this backup. And in this 2 weeks all things works whithout problems. And i ve got older backup,can i upload files from there and you pls check it?

Comment by Mark Agarunov [ 29/Nov/17 ]

Hello Lomachet,

Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to resync the affected node or restore from a backup if possible.

Thanks,
Mark

Comment by Moskalenko Alexey [X] [ 29/Nov/17 ]

I've replaced files but error is the same. I've attached a MongoDB file from a log directory,last log file + command output command.txt . About questions
1. Its a VM, so the are vhdd
2. Storage is fine
3. Its a pre-installed DB in microsoft advanced threat analytics server,so i dont know if there are any upgrades
4. Nope, error appears after restore from snapshot
5. First try, never been restored before
6. VM snapshots
7. Just before snapshot

Comment by Mark Agarunov [ 28/Nov/17 ]

Hello Lomachet,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Comment by Moskalenko Alexey [X] [ 28/Nov/17 ]

Is it safe to upgrade mongo in MATA server?

Comment by Ramon Fernandez Marina [ 27/Nov/17 ]

Can you please upload the WiredTiger.wt and WiredTiger.turtle files? We can attempt to repair them. Also, please consider upgrading to MongoDB 3.4.10.

Thanks,
Ramón.

Generated at Thu Feb 08 04:29:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.