[SERVER-36341] MongoDB does not start after restoring whole of dbPath directory Created: 28/Jul/18  Updated: 04/Sep/18  Resolved: 28/Jul/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: saeed Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: envm, rfi, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File err-after-replacing-1.log     File err.log     Text File mongo.conf     File repair-attempt.tgz    
Operating System: Linux
Participants:

 Description   

Hello there.

 

Unfortunately our server raid failed today. right now we have a zip file of mongo dbPath from last night. after extracting it, we are not able to start again mongo db.

 

Please help us to solve this big problem.



 Comments   
Comment by Ramon Fernandez Marina [ 28/Jul/18 ]

Unfortunately this error indicates that your backups are in a state beyond repair, which is not unexpected since they were taken while the mongod process was running. In this situation, our best recommendation would be to resync the affected node if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Regards,
Ramón.

Comment by saeed [ 28/Jul/18 ]

I extract again the zip file of dbPath, and replace WiredTiger.wt and WiredTiger.turtle with repair-attempt.tgz

After starting mongod via this command, i got error which its log is uploaded. please check my mongo.conf and error log.

Thanks

/usr/bin/mongod --quiet --config /etc/mongod.conf

Comment by Ramon Fernandez Marina [ 28/Jul/18 ]

I've attached repair-attempt.tgz; please extract it in your dbpath and try to start mongod again.

As for additional information, I think the key issue here is that you're not following the documented procedure:

If your storage system does not support snapshots, you can copy the files directly using cp, rsync, or a similar tool. Since copying multiple files is not an atomic operation, you must stop all writes to the mongod before copying the files. Otherwise, you will copy the files in an invalid state.

Comment by saeed [ 28/Jul/18 ]

Thanks for your comment, both required files uploaded.

Our machine is a vm, its not native.

please consider that we sync production machine mongo dbPath into other machine in Digital-Ocean via rsync.
Every time rsync script executed, mongo in source node is running, and we archive dbPath from secondary machine.

what kind of other information is helpful?

 

Thanks again.

Comment by Ramon Fernandez Marina [ 28/Jul/18 ]

Please upload the WiredTiger.wt and WiredTiger.turtle files so we can try to repair the metadata. I see the logs mention you're running Ubuntu 16.04, but can you please specify if this is a native installation, a VM or a container? If not native, any further details you can provide may be useful.

Note that there's no guarantee the repair attempt will succeed, especially if the zip file was created while mongod was running.

Thanks,
Ramón.

Generated at Thu Feb 08 04:42:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.