[SERVER-32181] Mongod Failed Starting after shutdown Created: 06/Dec/17  Updated: 07/Jan/18  Resolved: 07/Dec/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Miguel Gomes [X] Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     Text File log.txt     File repair-SERVER-32181.tar.gz    
Operating System: ALL
Participants:

 Description   

2017-12-06T09:23:20.819+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7680M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-12-06T09:23:20.867+0000 E STORAGE [initandlisten] WiredTiger error (0) [1512552200:867442][47422:0x7fff9a34d340], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 32768B block at offset 61440: calculated block checksum of 563799302 doesn't match expected checksum of 55106942
2017-12-06T09:23:20.867+0000 E STORAGE [initandlisten] WiredTiger error (0) [1512552200:867531][47422:0x7fff9a34d340], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
2017-12-06T09:23:20.867+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1512552200:867559][47422:0x7fff9a34d340], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-12-06T09:23:20.867+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2017-12-06T09:23:20.867+0000 I - [initandlisten]



 Comments   
Comment by Mark Agarunov [ 07/Dec/17 ]

Hello MiguelGomes,

Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to resync the affected node or restore from a backup if possible.

Thanks,
Mark

Comment by Miguel Gomes [X] [ 07/Dec/17 ]

After running --repair with these new files I have a different error:

2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] MongoDB starting : pid=57272 port=27017 dbpath=mongoBot 64-bit host=Miguels-MacBook-Pro.local
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2l  25 May 2017
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] allocator: system
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] modules: none
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten] build environment:
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-12-07T09:21:48.891+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-12-07T09:21:48.892+0000 I CONTROL  [initandlisten] options: { storage: { dbPath: "mongoBot" } }
2017-12-07T09:21:48.892+0000 I -        [initandlisten] Detected data files in mongoBot created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-12-07T09:21:48.892+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7680M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-12-07T09:21:49.011+0000 I -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 796
2017-12-07T09:21:49.011+0000 I -        [initandlisten]
 
***aborting after fassert() failure

Comment by Miguel Gomes [X] [ 07/Dec/17 ]

Hi Mark,

After replace the files I have the same issue, I already attached the logs.

About your questions:
1 - Locally and the disk is an SSD
2 - I already checked and it is fine
3 - Yes, it was always in the same version
4 - I copied but mongo was stopped before (although the system kill the application without properly stopped it)
5 - No
6 - Never created on this database
7 - Don't have this information

Comment by Mark Agarunov [ 06/Dec/17 ]

Hello MiguelGomes,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Comment by Miguel Gomes [X] [ 06/Dec/17 ]

Mark, attached the files. Let me know if you need something more from me.

Thanks,
Miguel

Comment by Mark Agarunov [ 06/Dec/17 ]

Hello MiguelGomes,

Thank you for the report. If you can provide the WiredTiger.wt and WiredTiger.turtle files we can attempt a repair of the database, but please keep in mind that this is not a guaranteed fix.

Thanks,
Mark

Comment by Miguel Gomes [X] [ 06/Dec/17 ]

Any idea about this problem?

Generated at Thu Feb 08 04:29:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.