[SERVER-53095] MongoDB 3.6.14 crashing on FreeBSD12.1 when unclean shutdown/reboot Created: 27/Nov/20  Updated: 27/Oct/23  Resolved: 13/Jan/21

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.6.14
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Akash Talole Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

FreeBSD12.1 UNIX


Attachments: File aftercrash_WiredTiger.wt     File aftercrash_WiredTigerLAS.wt     File aftercrash_sizeStorer.wt     File beforecrash_WiredTiger.turtle     File beforecrash_WiredTiger.wt     File beforecrash_WiredTigerLAS.wt     File beforecrash_sizeStorer.wt     Text File crashed_mongodb_logs.txt     Text File mongo_recover_script.txt     PNG File mongodb_crash.png     Text File working_mongodb_logs.txt    
Operating System: ALL
Steps To Reproduce:
  1. Run mongodb on FreeBSD12.1 VM
  2. Hard reboot or poweroff VM from host
  3. start vm and check mongodb service
  4. try to repair
  5. WiredTiger.turtle not found error.
Participants:

 Description   

We have mongodb installed on Unix (FreeBSD12.1) VM. When hard reboot occurs due to vSphere HA or host issue, mongodb is crashing. When we are doing repair it gives attached error. It is found that WiredTiger.turtle file is empty after crash, which causing problem in repair. How to solve this issue?

 

Thanks and regards,

Akash T



 Comments   
Comment by Dmitry Agranat [ 13/Jan/21 ]

Hi taloleakash@gmail.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 29/Dec/20 ]

taloleakash@gmail.com, can you try using --repair with the latest MongoDB version, which is 4.4.3 after significantly increasing your node memory? In the backtrace I've noticed this:

_ZN5mongo29reportOutOfMemoryErrorAndExitEv

Comment by Akash Talole [ 28/Dec/20 ]

Hi @dmitry.agranat 

MongoDB server version is 3.6.14.

check mongo_recover_script.txtwhich i used for restoring backup.

beforecrash_WiredTiger.turtleand after crashed it is becaming empty so unable to upload.

crashed_mongodb_logs.txtsame logs are coming when i ran mongo repair command 

mongod --repair --dbpath /var/db/mongodb --port 9091

 

Comment by Dmitry Agranat [ 28/Dec/20 ]

Thanks taloleakash@gmail.com, did you try to --repair it with the latest MongoDB version? The attached screenshot does not show what version was used and the --repair log lines & output.

In my previous comment, I've requested:

The wiredTiger.wt and wiredTiger.turtle files (before a repair on these files was executed)

But I could not find the wiredTiger.turtle after the crash. Please note that both files are needed after the crash but before your repair attempt.

Comment by Akash Talole [ 23/Dec/20 ]

Hi Dmitry,

This case is for standalone mongodb deployment.
We have hourly backup of application configuration not whole database backup of high size.

This is UFS filesystem. We are using local datastore of VMFS5 on vsphere.

Check attached log files and mongodb files for before and after crash of mongodb. WiredTiger.turtle is empty after crash.

aftercrash_sizeStorer.wtbeforecrash_sizeStorer.wtworking_mongodb_logs.txtcrashed_mongodb_logs.txtaftercrash_WiredTiger.wtbeforecrash_WiredTiger.wtaftercrash_WiredTigerLAS.wtbeforecrash_WiredTigerLAS.wt beforecrash_WiredTiger.turtle

 

Comment by Akash Talole [ 21/Dec/20 ]

Hi Dmitry,

I will share additional information today.

Comment by Dmitry Agranat [ 20/Dec/20 ]

Hi taloleakash@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the data mentioned in my last comment?

Thanks,
Dima

Comment by Dmitry Agranat [ 08/Dec/20 ]

Hi taloleakash@gmail.com,

The scenario you've described and the outcome should not have happened but we do not have enough information for the event. For example, you have not mentioned what filesystem is being used. MongoDB requires a filesystem that supports fsync() on directories.

You can also try mongod --repair using the latest version of MongoDB. The ideal resolution is to perform a clean resync from an unaffected node.

If the above fails, please provide:

  1. The logs for the affected node, including before, leading up to, and after the first sign of corruption.
  2. As much of syslog and dmesg content leading up to the first sign of corruption as possible.
  3. A description of the underlying storage mechanism in use, including details like:
    1. What file system and/or volume management system is in use?
    2. Is data storage locally attached or network-attached?
    3. Are disks RAIDed and if so how?
    4. Are disks SSDs or HDDs?
  4. A description of your backup method, if any.
  5. A description of your disks have been recently checked for integrity?
  6. A history of the deployment, including:
    1. a timeline of version changes
    2. a timeline of hardware upgrade/downgrade cycles or configuration changes
    3. a timeline of disaster recovery or backup restoration activities
    4. a timeline of any manipulations of the underlying database files, including copies or moves, and information about whether mongod was running during each manipulation.
  7. The wiredTiger.wt and wiredTiger.turtle files (before a repair on these files was executed)

Thanks,
Dima

Generated at Thu Feb 08 05:29:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.