[SERVER-2676] Exception during Repair--Ramifications? Created: 04/Mar/11  Updated: 30/Mar/12  Resolved: 04/Mar/11

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 1.6.5
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: Jamie Turner Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

While trying to run a repair on our production database, the following is printed to the log:

0x540c7e 0x4eb09c 0x71c222 0x7325a8 0x5fdfd0 0x706cd6 0x70ab0d 0x70b5e1 0x56ffbd 0x570528 0x6d306b 0x6d4a48 0x6d61ad 0x73071b 0x824ca8 0x827e20 0x828936 0x829518 0x8317a5 0x7fd8da101c4d
mongod(_ZN5mongo11msgassertedEiPKc+0x1de) [0x540c7e]
mongod(_ZN5mongo7BSONObj4initEPKcb+0x4ec) [0x4eb09c]
mongod(_ZN5mongo7BSONObjC1EPKNS_6RecordE+0x32) [0x71c222]
mongod(_ZN5mongo11BasicCursor7currentEv+0x18) [0x7325a8]
mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x350) [0x5fdfd0]
mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x256) [0x706cd6]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x14ed) [0x70ab0d]
mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_b+0x81) [0x70b5e1]
mongod(_ZN5mongo14DBClientCursor11requestMoreEv+0x2fd) [0x56ffbd]
mongod(_ZN5mongo14DBClientCursor4moreEv+0x58) [0x570528]
mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x5fb) [0x6d306b]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x1018) [0x6d4a48]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x6d61ad]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x3eb) [0x73071b]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x824ca8]
mongod(_ZN5mongo15repairDatabasesEv+0x280) [0x827e20]
mongod(_ZN5mongo14_initAndListenEiPKc+0x436) [0x828936]
mongod(_ZN5mongo13initAndListenEiPKc+0x18) [0x829518]
mongod(main+0x6f75) [0x8317a5]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7fd8da101c4d]

What are the ramifications of this? Have we lost data?



 Comments   
Comment by Will Moss [ 04/Mar/11 ]

Okay cool. Well our general record counts looks right and we're okay with loosing a few records.

Comment by Eliot Horowitz (Inactive) [ 04/Mar/11 ]

Just skips that record.

Comment by Will Moss [ 04/Mar/11 ]

Just to clarify the specifics, when the repair process hits a bad record, does it skip that record and continue on or does it stop repairing that collection?

Comment by Eliot Horowitz (Inactive) [ 04/Mar/11 ]

Ok.
Worse case, one collection is missing data.
Most likely its only 1 document, but its a bit hard to tell without the whole log.

Comment by Will Moss [ 04/Mar/11 ]

Sorry, but I inadvertently didn't keep the full log--I haven't gotten much sleep since our production Mongo instance went down on Wednesday night.

Comment by Eliot Horowitz (Inactive) [ 04/Mar/11 ]

Can you attach the full log

Comment by Jamie Turner [ 04/Mar/11 ]

To clarify, we're referring to the database post-restoration, and we're inquiring about the presence (or not) in that restored database of records that existed pre-crash. The actual corrupted files, we have a copy of, so we're not worried about them.

Comment by Jamie Turner [ 04/Mar/11 ]

So our repair finished on that database.

If we saw this appear exactly once, does it indicate that a record (or a few records) were skipped, or that the rest of the repair process for that collection had to be aborted, e.g., many records are potentially missing?

We're just trying to assess the ceiling on the damage so that we can weigh our options on our side before proceeding--thanks.

Comment by Eliot Horowitz (Inactive) [ 04/Mar/11 ]

That doesn't change the original file, but it does mean its hard to recover whats in there.
Do you have a backup or is that your only copy?
If its your only copy, you should download the latest nightly, and use mongodump repair to recover

mongodumo --dbpath <path to data> --repair

Generated at Thu Feb 08 03:00:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.