[SERVER-37356] Unable to start mongo server due to corrupted admin database Created: 27/Sep/18  Updated: 06/Dec/22  Resolved: 25/Oct/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Karthik Anant Assignee: Backlog - Triage Team
Resolution: Done Votes: 0
Labels: nrp, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod-cleanedup.txt     Text File mongod.log    
Assigned Teams:
Server Triage
Operating System: ALL
Participants:

 Description   

Hello,

We have an instance of mongo which does not start up anymore due to corrupted data, the root cause seems to be unclean shutdown. I tried using the --repair option but it did not help. Here is the error I see in the log files:

 

Fatal assertion 28548 NoSuchKey Unable to find metadata for table:admin/collection-184-1679902527159703400

 

I did find a related issue https://jira.mongodb.org/browse/SERVER-28364. The final comment seem to indicate there is nothing that could be done. But I just wanted to see if anything has changed since then. Is there anything that could done to recover this database (admin). Our other databases contain sensitive data, but since this is just the admin database, I was wondering if I upload this securely so that somebody from mongo can take a look (don't want to just attach it here)?

Please let me know your thoughts!

 



 Comments   
Comment by Nick Brewer [ 03/Oct/18 ]

kanant Happy to help. If you could confirm for us the underlying platform of this machine (virtual machine, container, native hardware, etc) fit would be useful for us in tracking this issue.

Thanks,
-Nick

Comment by Karthik Anant [ 02/Oct/18 ]

Hello Nick,

Unfortunately I cannot since it contains sensitive data. We will try to see if there is a newer backup (we do have an old one). Thanks for all your help with this issue!

Comment by Nick Brewer [ 02/Oct/18 ]

kanant I suspect the error may be related to the difference in versions. If you'd like, I can try performing a manual repair - I'll need the WiredTiger.wt and WiredTiger.turtle files, from a copy of the dbpath before the repair was performed.

Thanks,
-Nick

Comment by Karthik Anant [ 02/Oct/18 ]

Sorry, my mistake - I did try to start the server even though there were errors during repair. It did not start though. I have attached the log file - mongod.log. Please let me know if you have any questions.

mongod.log

Comment by Nick Brewer [ 02/Oct/18 ]

kanant Sorry if my instructions were unclear - please boot your 3.2.1 mongod installation using the dbpath that you repaired with the 4.0 build. If you encounter any errors, please include them here.

Thanks,
-Nick

Comment by Karthik Anant [ 02/Oct/18 ]

Hello Nick,

Sorry for the late response. Lost track of this. Is that not what I am supposed to use? I used it since your initial reply indicated 4.0 nightly. If you could point me to the right one I can try with that.

I have already attached the log file - Please check the mongod-cleanedup.txt attachment. This line caught my attention, got me wondering if a repair is going to be possible since our version of mongo is 3.2.1

2018-09-27T16:07:38.691-0500 F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.6 before attempting an upgrade to 4.0; see http://dochub.mongodb.org/core/4.0-upgrade-fcv for more details.

Thank you!

 

Comment by Nick Brewer [ 28/Sep/18 ]

kanant The attached file appears to be from the 4.0 nightly - do you have the startup logs from the 3.2 mongod when it was booted with the repaired dbpath?

-Nick

Comment by Karthik Anant [ 28/Sep/18 ]

Hello Nick,

Yes, that's what I eventually did, but did not help. The file I attached in my previous comment, that has the details. Please let me know if you have any thoughts on whether anything could be done. Thanks!

 

Comment by Nick Brewer [ 28/Sep/18 ]

kanant Sorry for the confusion - I meant that you should start our 3.2 installation using the same dbpath on which the 4.0 nightly --repair was run.

-Nick

Comment by Karthik Anant [ 27/Sep/18 ]

Hello Nick,

Thank you for the quick response. I backed up the files in my db path and tried repairing with your instructions (using the 4.0 nightly build - http://downloads.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-v4.0-latest.zip).

No luck. I just see errors in the log file - I have attached the log file (removed rows containing our product's collection names etc). One thing I wanted to reiterate is our mongo version is 3.2.1 (your last point refers to 3.4).

Especially this line kind of supports my point - am I out of luck since the data files were created using 3.2.1? Or am I missing something?

2018-09-27T16:07:38.691-0500 F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.6 before attempting an upgrade to 4.0; see http://dochub.mongodb.org/core/4.0-upgrade-fcv for more details.

Please let me know if you have any questions.

Thanks!

mongod-cleanedup.txt

 

Comment by Nick Brewer [ 27/Sep/18 ]

kanant There's a repair process you can try that makes use of some recently introduced updates to the mongod --repair option:

Important: Before you start this process, take a backup of your dbpath via a manual copy of the directory, or a filesystem snapshot.

  • Download and untar the latest 4.0 Nightly release from the MongoDB Download Center
  • From the unpacked directory, run mongod --repair, and point it to your dbpath:

      bin/mongod --repair --dbpath /path/to/corrupted/dbpath
      

  • Once the repair completes, stop the 4.0 nightly mongod, and then start your regular 3.4 installation as you normally would, using the same dbpath that you previously ran the --repair on.

Please let us know how it works for you - we'd be happy to help if you run into any issues.

-Nick

Generated at Thu Feb 08 04:45:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.