[SERVER-25794] Can't start MongoDB 3.2.8 server with WiredTiger Created: 25/Aug/16  Updated: 13/Aug/18  Resolved: 06/Sep/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dong Koo Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File WiredTigerLog.0000000094     File WiredTigerLog.0000000095     File _mdb_catalog.wt     Text File mongo.log     File repair_attempt.tar.gz     File sizeStorer.wt     File storage.bson    
Operating System: ALL
Participants:

 Description   

Hi.

This is for installation on my Mac. I was running a mongo via homebrew and recently also ran another installation via docker with local db path mounted to it. Without realizing both server process were on, I ran a mongorestore for one collection I got off from production and did some rename collection and created a new collection, etc. After that, I restarted servers via homebrew and now server exits with checksum error.

Please see attached files for more info. Need it fixed asap. Thanks!



 Comments   
Comment by Kelsey Schubert [ 08/Sep/16 ]

Hi dongkoo,

We do not support the steps you have referenced. If your database cannot be recovered using the built-in repair or the attempted manual repair effort, I unfortunately don't have any further suggestions to offer.

I would recommend extreme caution if you are attempting to recover data using the wt binary, as it is not aware of the data structures used by MongoDB and may be using a different version of the WiredTiger engine.

Finally, please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. As such, I'm unable to provide additional support for data recovery on this forum beyond the initial repair attempt which was not successful.

Kind regards,
Thomas

Comment by Dong Koo [ 06/Sep/16 ]

One of the collection failed to dump using wt cmd tool. I guess it is unrecoverable at this point.

[1473190736:9334][7411:0x7fff7a974000], WT_SESSION.open_cursor: 'table:collection-40-5575108824741027845' cannot be used until all column groups are created: Invalid argument
wt: cursor open(table:collection-40-5575108824741027845) failed: Invalid argument

Comment by Dong Koo [ 06/Sep/16 ]

Hi,

Just one more thing. I found this http://www.alexbevi.com/blog/2016/02/10/recovering-a-wiredtiger-collection-from-a-corrupt-mongodb-installation/ and trying it manually. It seems your repair attempt files allowed me to run wt cmd tool. While running salvage cmd, I found some files return file not found error:
wt: salvage(file:collection-116-5575108824741027845.wt): No such file or directory

And trying wt dump on these files return this error:
wt: metadata:: cursor.search: WT_NOTFOUND: item not found

Was wondering if you had any insight into this.

Comment by Kelsey Schubert [ 06/Sep/16 ]

Hi dongkoo,

Thank you for the update. As you identified, this error indicates that the repair attempt was unsuccessful. Unfortunately, I cannot point you to any tools that would recover data from this situation.

Kind regards,
Thomas

Comment by Dong Koo [ 06/Sep/16 ]

Hi,

That didn't work. Error log is below. Any way to salvage db data such as export from directory?

mongod(_ZN5mongo13initAndListenEi+0x13) [0x10ca5ba33]
mongod(main+0x3FC) [0x10ca5ff8c]
libdyld.dylib(start+0x1) [0x7fff923155ad]
??? [0x3]
----- END BACKTRACE -----
2016-09-06T11:55:02.395-0400 I - [initandlisten]

***aborting after invariant() failure

Comment by Kelsey Schubert [ 06/Sep/16 ]

Hi dongkoo,

Sorry for the delay getting back to you. Unfortunately, from your description I think recovery from this situation is unlikely.

I have attempted a repair of WiredTiger files and attached the new files to this ticket. Please replace these files in your $dbpath and attempt to restart the mongod instance.

Thank you,
Thomas

Comment by Dong Koo [ 06/Sep/16 ]

Hi,

Anything on this yet?

Generated at Thu Feb 08 04:10:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.