[SERVER-14529] assertion src/mongo/db/structure/btree/key.cpp:433 Created: 11/Jul/14  Updated: 08/Feb/23  Resolved: 25/Aug/14

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.6.1
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Dan Maas Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-15071 assertion src/mongo/db/structure/btre... Closed
Operating System: ALL
Steps To Reproduce:

This happens very rarely on a tiny fraction of similar queries. The query that produced it was of the form:

db.mycollection.find({ $and: [ { base_map_loc: [ 182, 156 ] }, { _id: { $ne: "s1588142_270" } } ] })

where there is a geo2D index on base_map_loc with parameters {min:0, max:1024, bits:1}

Participants:

 Description   

Got this assertion error, here is the log (not sure if the very first line is related to it or not):

2014-07-11T06:00:22.319+0000 [conn491470] key seems to have moved in the index, refinding. 0:28b8000
2014-07-11T06:00:22.330+0000 [conn452054] trprod_region_sector205.tr_region_sector205_map Assertion failure false src/mongo/db/structure/btree/key.cpp 433
2014-07-11T06:00:22.531+0000 [conn452054] trprod_region_sector205.tr_region_sector205_map 0x11c0e91 0x1163109 0x1146f4e 0xf0bc8b 0xb681da 0xb6482d 0xa9dd4e 0xa9abbd 0xaa169c 0xaa1fab 0xd67e9f 0xd4c138 0xb97322 0xb99902 0x76b6af 0x117720b 0x7f29f53def18 0x7f29f46f0e0d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1163109]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x1146f4e]
 /usr/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x19b) [0xf0bc8b]
 /usr/bin/mongod(_ZNK5mongo18BtreeInterfaceImplINS_12BtreeData_V1EE5keyAtEPKNS_17IndexCatalogEntryENS_7DiskLocEi+0x9a) [0xb681da]
 /usr/bin/mongod(_ZNK5mongo16BtreeIndexCursor6getKeyEv+0x3d) [0xb6482d]
 /usr/bin/mongod(_ZN5mongo9IndexScan4workEPm+0x6e) [0xa9dd4e]
 /usr/bin/mongod(_ZN5mongo10FetchStage4workEPm+0xad) [0xa9abbd]
 /usr/bin/mongod(_ZN5mongo18KeepMutationsStage4workEPm+0x3c) [0xaa169c]
 /usr/bin/mongod(_ZN5mongo10LimitStage4workEPm+0x5b) [0xaa1fab]
 /usr/bin/mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_7DiskLocE+0xef) [0xd67e9f]
 /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x958) [0xd4c138]
 /usr/bin/mongod() [0xb97322]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb99902]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b6af]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x117720b]
 /lib64/libpthread.so.0(+0x7f18) [0x7f29f53def18]
 /lib64/libc.so.6(clone+0x6d) [0x7f29f46f0e0d]
2014-07-11T06:00:22.566+0000 [conn452054] assertion 0 assertion src/mongo/db/structure/btree/key.cpp:433 ns:trprod_region_sector205.tr_region_sector205_map query:{ $and: [ { base_map_loc: [ 182, 156 ] }, { _id: { $ne: "s1588142_270" } } ] }
2014-07-11T06:00:22.566+0000 [conn452054]  ntoskip:0 ntoreturn:-1



 Comments   
Comment by Ramon Fernandez Marina [ 25/Sep/14 ]

Hi fraccy, without access to the malformed index data there was no further investigation that could be done. If running validate() shows some errors there may be enough hints for us to continue the investigation. If the issue is a corrupt index key you may want to try rebuilding your indexes to see if the problem is corrected. Note that both operations may be resource-intensive and have a performance impact on your MongoDB instance.

Regards,
Ramón.

Comment by Tim Putnam [ 25/Sep/14 ]

Not sure that this should be closed, this just happened to me too. It wasn't on an update it was just on a find. Don't know what to describe really, am executing this query (its finding individual elements using indexes) a vast number of times on a collection with about 4 million docs in it, and have seen it happen just once (on 2.6.4). Can't share the data.

Comment by Ramon Fernandez Marina [ 25/Aug/14 ]

dmaas, thanks for running validate(). Unfortunately if it didn't find any errors there's not a lot of information that can help us investigate further. Looks like there was a transient index corruption, but the constant adding/removing documents in this collection seems to have taken care of it.

I'm going to close this ticket. If this happens again it would be of great help to get a copy of your data to investigate (we can provide a private, secure way to upload it – let us know).

Regards,
Ramón.

Comment by Dan Maas [ 26/Jul/14 ]

Thanks for the update Ramon! I have run the validate() command against all our collections. Is there anything specific I should look for? Nothing shows up in the "errors" part of the result. The collection that caused this exception has documents added and removed frequently, so the error might be transient. Next time I see the exception, I will try to run the validate command before the affected document disappears.

Comment by Ramon Fernandez Marina [ 25/Jul/14 ]

dmaas, the assertion happens when inspecting an index key, and finding that it doesn't contain a valid BSON type, so this seems to indicate you have a data corruption issue. Can you check the integrity of your data using validate? You can do so by running this command:

db.collection.validate(true)

Note that this is a resource-intensive operation and may have an impact on the performance of your MongoDB instance.

Generated at Thu Feb 08 03:35:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.