[SERVER-36905] Database corrupt Created: 28/Aug/18  Updated: 23/Sep/18  Resolved: 30/Aug/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: it@iidf.ru Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File WiredTiger.wt.1     Text File log.txt    
Participants:

 Description   

Hello!

Our MongoDB base is corrupted by problems with hard drive. We recover our disk with database and MongoDB server

We check config and MongoDB database folder. Config ok and have a right dbpath.

In dbpath directory we have many files like this:

total 2.5G
rwxrwxr- 1 mongodb mongodb 21 Apr 16 2017 WiredTiger.lock
rwxrwxr- 1 mongodb mongodb 95 Apr 16 2017 storage.bson
rwxrwxr- 1 mongodb mongodb 16K Apr 16 2017 index-4--2017253496148277652.wt
rwxrwxr- 1 mongodb mongodb 52K Oct 10 2017 index-15-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 36K May 20 02:33 _mdb_catalog.wt.1
rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 collection-2--2017253496148277652.wt
rwxrwxr- 1 mongodb mongodb 36K May 20 02:33 collection-2--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 72K May 20 02:33 collection-18-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 52K May 20 02:33 collection-14-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 100K May 20 02:33 collection-0--4753795739824518231.wt
rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 index-3--2017253496148277652.wt
rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 index-3--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 36K May 20 02:34 index-1--2017253496148277652.wt
rwxrwxr- 1 mongodb mongodb 40K May 20 02:34 collection-0--2017253496148277652.wt
rwxrwxr- 1 mongodb mongodb 44K May 20 02:38 index-19-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 16K May 20 02:38 index-1--4753795739824518231.wt
rwxrwxr- 1 mongodb mongodb 40K May 20 02:51 index-11-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 16K May 20 06:49 index-1-7644913279549684036.wt
rwxrwxr- 1 mongodb mongodb 32K Jun 5 11:53 index-3-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 36K Jun 5 11:53 collection-2-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 52K Jun 17 08:26 index-13-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 132K Jun 22 10:14 collection-12-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 68K Aug 8 03:35 index-25-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 322M Aug 8 03:35 collection-24-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 36K Aug 13 15:24 index-9--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 60K Aug 13 15:24 collection-8--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 368K Aug 15 10:06 index-21-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 1.2M Aug 15 10:06 collection-20-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 84K Aug 16 10:23 index-7-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 192K Aug 16 10:23 collection-6-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 772K Aug 16 14:49 index-5-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 148K Aug 16 14:49 index-23-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 6.1M Aug 16 14:49 collection-4-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 1.1M Aug 16 14:49 collection-22-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 324K Aug 16 14:49 collection-10-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 36K Aug 16 14:49 collection-0-7644913279549684036.wt
rwxrwxr- 1 mongodb mongodb 2.2M Aug 17 11:50 index-17-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 16M Aug 17 11:50 collection-16-6056211751025971878.wt
rwxrwxr- 1 mongodb mongodb 468K Aug 17 16:59 index-7--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 864K Aug 17 16:59 collection-6--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 23M Aug 18 00:16 index-1--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 2.1G Aug 18 00:16 collection-0--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 968K Aug 18 00:17 index-5--3777467847017729897.wt
rwxrwxr- 1 mongodb mongodb 124K Aug 18 00:59 WiredTiger.wt.1
rwxrwxr- 1 mongodb mongodb 49 Aug 28 09:12 WiredTiger
rwxrwxr- 1 mongodb mongodb 1.6M Aug 28 09:12 collection-4--3777467847017729897.wt
rw-rr- 1 mongodb nogroup 16K Aug 28 09:37 index-1-7056954381577752709.wt
rwxrwxr- 1 mongodb mongodb 6 Aug 28 09:38 mongod.lock
drwxrwxr-- 2 mongodb mongodb 4.0K Aug 28 09:39 journal
rw-rr- 1 mongodb nogroup 1000 Aug 28 09:40 WiredTiger.turtle
drwxrwxr-- 2 mongodb mongodb 4.0K Aug 28 10:54 diagnostic.data
rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 _mdb_catalog.wt
rw-rr- 1 mongodb nogroup 32K Aug 28 10:55 index-2-7056954381577752709.wt
rw-rr- 1 mongodb nogroup 16K Aug 28 10:55 index-0-7056954381577752709.wt
rwxrwxr- 1 mongodb mongodb 16K Aug 28 10:55 collection-2-3814419529578137210.wt
rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 collection-0-3814419529578137210.wt
rwxrwxr- 1 mongodb mongodb 92K Aug 28 10:55 WiredTiger.wt
rw-rr- 1 mongodb nogroup 4.0K Aug 28 10:55 WiredTigerLAS.wt
rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 sizeStorer.wt

Its looks like a database!

Our mongod daemon started and we can enter in it by Robo3t, but we see only admin and local databases.

We try to repair it by "mongod --repair --dbpath (path)" key and its nothing gave to as

We try "./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2657-1723320556100349955.wt" on all collections in dbpath directory (by WiredTiger utility) and its nothing gave to as again

Can you help as with this problem? Thanks!

Have a good day!



 Comments   
Comment by it@iidf.ru [ 31/Aug/18 ]

Ok! Thank you for your help, Nick!

Have a good day! =)

Comment by Nick Brewer [ 30/Aug/18 ]

it@iidf.ru This still appears to be a permissions issue - I would suggest ensuring that all of the files within /var/lib/mongodb are owned by the mongodb user and group, and then removing the mongod.lock file, and rebooting the mongod. 

Since this doesn't demonstrate an underlying bug in MongoDB, I'm going to close this ticket. For MongoDB-related support discussion I suggest posting on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

-Nick

Comment by it@iidf.ru [ 30/Aug/18 ]

Hello!

When we try to "systemctl  start mongod" we get this message:

root@db01:~# systemctl start mongod
root@db01:~# systemctl status mongod
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2018-08-30 02:50:08 EDT; 4s ago
Process: 4126 ExecStart=/usr/bin/mongod --verbose --config /etc/mongod.conf (code=exited, status=1/FAILURE)
Main PID: 4126 (code=exited, status=1/FAILURE)

Aug 30 02:50:08 db01 systemd[1]: Started High-performance, schema-free document-oriented database.
Aug 30 02:50:08 db01 systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
Aug 30 02:50:08 db01 systemd[1]: mongod.service: Unit entered failed state.
Aug 30 02:50:08 db01 systemd[1]: mongod.service: Failed with result 'exit-code'.

 

In log we get:

2018-08-30T02:53:01.977-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] MongoDB starting : pid=4196 port=27017 dbpath=/var/lib/mongodb 64-bit host=db01
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] db version v3.4.3
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] git version: f07437fb5a6cca07c10bafa78365456eb1d6d5e1
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] allocator: tcmalloc
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] modules: none
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] build environment:
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] distmod: debian81
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] distarch: x86_64
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] target_arch: x86_64
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "127.0.0.1,192.168.70.14", port: 27017 }

, storage: { dbPath: "/var/lib/mongodb", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", verbosity: 1 } }
2018-08-30T02:53:01.981-0400 D - [initandlisten] User Assertion: 28596:Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied: "/var/lib/mongodb/mongod.lock" src/mongo/db/service_context_d.cpp 86
2018-08-30T02:53:01.981-0400 I STORAGE [initandlisten] exception in initAndListen: 28596 Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied: "/var/lib/mongodb/mongod.lock", terminating
2018-08-30T02:53:01.981-0400 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-08-30T02:53:01.981-0400 I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] now exiting
2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] shutting down with code:100

 

I see "Permission denied" message, but I check permissions of the dbpath directory, it's ok =(

Comment by Nick Brewer [ 30/Aug/18 ]

it@iidf.ru Are you starting the mongod in a different way than you normally would? For example, do you usually start mongod via the init system (something like sudo systemctl start mongod or sudo service mongod start), and are you instead starting it manually via mongod --repair --dbpath (path)?

If so, I'd like to know what happens when you try starting the mongod as you normally would. 

Thanks, 

Nick 

Comment by it@iidf.ru [ 29/Aug/18 ]

Hello!

Yes, we make a "chmod R 664 /var/lib/mongodb" and "chown -R mongodb:mongodb /var/lib/mongodb" before start the daemon

I make new logs for you

And I can't find the "src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp" directory, where it must been?

 

Comment by Nick Brewer [ 28/Aug/18 ]

it@iidf.ru From the log file you've provided, it appears that you're running into a permissions issue after the repair you attempted:

Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267

Have you compared the permissions of the files in your dbpath after the attempt to repair with the wt binary against the permissions of the files in the backup taken last week?

-Nick

Comment by it@iidf.ru [ 28/Aug/18 ]

From the MongoDB shell we see 2 standart databases (admin and local) and a standart collections in it.

In sum we see with shell the some thing like a with Robo3T

Comment by it@iidf.ru [ 28/Aug/18 ]

MongoDB version is 3.4.3

Linux db01 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux

Server working at VMvare as virtual machine

We have a backup by 1 week ago, we try to put it instead of original database folder and result not change =(

We have a WiredTiger.wt and WiredTiger.wt.1, I upload both to this issue

Also I upload a log at disk breakdown moment. 

Thank you for your help!

Comment by Nick Brewer [ 28/Aug/18 ]

it@iidf.ru Can you include the log entries from the time you start the mongod? Additionally, have you checked what collections are available via the mongo shell directly, as a user with appropriate access?

I should note that we don't recommend attempting to recover files via the wt binary directly. If you have any backups available from before the hard drive issue, this would likely be the fastest way to get your database back into a functional state. That said, if you can upload your WiredTiger.wt and WiredTiger.turtle files, I would be happy to attempt a repair; in addition to the requested log entry, I'd need to confirm:

  • The MongoDB version
  • The operating system and version
  • Platform (virtual machine, container, native hardware, etc)

Thanks,
Nick

Generated at Thu Feb 08 04:44:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.