[SERVER-30165] MongoDB Service doesn't start (Error 1067); Stored data for our application not accessible Created: 17/Jul/17  Updated: 27/Oct/23  Resolved: 07/Aug/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.0.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Clemens Schadner [X] Assignee: Mark Agarunov
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-30165-repair.tar.gz     File for Mongo Support.rar     File logs complete.rar    
Operating System: ALL
Participants:

 Description   

Dear Support-Team,

due to an software update for our data historian and visualisation application, we faced the problem that the installation process crashed and the update gone wrong. Since then we are unable to access the stored data and also can't write new data into database (of course because the service isn't running).
I think we may restarted the server not properly (which also holds the MongoDB service) and so the mongod.lock file remained NOT empty.
Deleting it didn't solve the problem.

What can be done to get the MongoDB service running again?

Attached please find all WiredTiger.* files such as sizeStorer.wt and *.lock files.

Thanks in advance!

Yours hopefully,

Clemens Schadner
cts GmbH



 Comments   
Comment by Clemens Schadner [X] [ 04/Aug/17 ]

Hello Mark,

later I started the command prompt as admin and could do a mongodump, but the data is somehow gone.
Since this is not in a productive environment we don't essentially need the data. You can close the ticket.

So thanks a lot for your time Mark and have a nice day!

Mit freundlichen Grüßen / Best regards

Clemens Schadner
Software Engineer | Advanced Solutions

cts GmbH
Fuhrmannstraße 10
D-84508 Burgkirchen

Tel. .+49 (0)8679 91689-202
Fax. .+49 (0)8679 91689-120
clemens.schadner@cts-gmbh.de
www.group-cts.de

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Burgkirchen a. d. Alz
Handelsregister: Amtsgericht Traunstein, HRB16883
Geschäftsführer: Hans Gehringer, Robert Schüller

Comment by Mark Agarunov [ 02/Aug/17 ]

Hello ctsClemens,

Thank you for the information. Looking at the error output you provided, this looks like it may be a permissions issue. Are you starting mongodb as an administrator? The specific error is Access Denied, which would indicate that the user you are trying to start mongod as is not the same user, and does not have the same access as the user it was previously running as. Please try executing this command as an administrator and let me know if this fixes the issue.

Thanks,
Mark

Comment by Clemens Schadner [X] [ 18/Jul/17 ]

Hey Mark,

thanks for your quick response! The files provided by you didn't solve the problem - the damn service throws the same error 1067.

When I try:
C:\inmation.root\MongoDB\bin>mongod --repair --repairpath C:\inmation.root\MongoDB\data --dbpath C:\inmation.root\MongoDB\data --storageEngine wiredTiger --noauth
2017-07-18T07:05:32.850+0200 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: C:\inmation.root\MongoDB\data\mongod.lock errno:5
Access denied. Is a mongod instance already running?, terminating
2017-07-18T07:05:32.850+0200 I CONTROL [initandlisten] dbexit: rc: 100

But I'm pretty sure that no mongod is running.

Please find the complete logs in the attachment.

  1. underlying storage mechanism: wiredTiger; The MongoDB runs on a VM with a local storage, its backup runs over the network to our NAS. Disk typ: SAS HDD; RAID: 5
  2. Integrity: The VM containing the MongoDB runs in an productive environment. The IT guy told me he doesn't has enough resources to move all VMs to another server and smash those check commands into the console to check the disks. But the server is quite knew so everything should be fine (hopefully).
  3. Yes, the version didn't changed.
  4. I didn't manipulated the underlying db files, just made a software update for our application which uses MongoDB. I guess mongod was running, but in previous updates that wasn't a problem.
  5. No I never restored this instance from backups.
  6. We use Veeam Backup & Replication 9.5 to backup our VMs.
  7. What you are especially looking for doesn't really exist at our company (according to the IT guy). But our ESXi shows no errors and runs without any issues.

Thanks for your time!

Best regards,
Clemens

Comment by Mark Agarunov [ 17/Jul/17 ]

Hello ctsClemens,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Would you please check the integrity of your disks?
  3. Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
  4. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  5. Have you ever restored this instance from backups?
  6. What method do you use to create backups?
  7. When was the underlying filesystem last checked and is it currently marked clean?

Thanks,
Mark

Generated at Thu Feb 08 04:22:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.