[SERVER-14093] Assertion failure !e.eoo() in bsonobjbuilder.h when sorting on an empty field name Created: 30/May/14  Updated: 10/Dec/14  Resolved: 07/Jul/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jiangcheng Wu Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Participants:

 Description   

Got "Assertion failure !e.eoo() src/mongo/db/../bson/bsonobjbuilder.h" when queried at SECONDARY
and

$readPreference: { mode: "secondaryPreferred" }

Logs:
SECONDARY3:

Fri May 30 15:15:22.551 [conn1649793]  ez3hl2bej7d81o4qrqigbusft3u7qgm5xahbcbkiat5pjn3f.XiaoHuaImage Assertion failure !e.eoo() src/mongo/db/../bson/bsonobjbuilder.h 92
0xde8c31 0xda9c9d 0x9f4960 0x9f110b 0xb725a5 0xc45f61 0xa8a7a0 0xa8ab4a 0xa86a61 0xa8c0be 0xa8ef28 0xa0182e 0xa02b73 0x6eb838 0xdd51fe 0x7f15e88a9e9a 0x7f15e7bbc3fd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde8c31]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda9c9d]
 /usr/bin/mongod(_ZNK5mongo14KeyGeneratorV18_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_10BSONObjCmpESaIS9_EEiSB_+0x6c0) [0x9f4960]
 /usr/bin/mongod(_ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_10BSONObjCmpESaIS1_EE+0x38b) [0x9f110b]
 /usr/bin/mongod(_ZNK5mongo16FieldRangeVector10firstMatchERKNS_7BSONObjE+0xd5) [0xb725a5]
 /usr/bin/mongod(_ZN5mongo12ScanAndOrder3addERKNS_7BSONObjEPKNS_7DiskLocE+0x51) [0xc45f61]
 /usr/bin/mongod(_ZN5mongo20ReorderBuildStrategy19_handleMatchNoDedupEPNS_13ResultDetailsE+0x70) [0xa8a7a0]
 /usr/bin/mongod(_ZN5mongo20ReorderBuildStrategy11handleMatchEPNS_13ResultDetailsE+0x5a) [0xa8ab4a]
 /usr/bin/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0xb1) [0xa86a61]
 /usr/bin/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_12ChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE+0x3be) [0xa8c0be]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1ac8) [0xa8ef28]
 /usr/bin/mongod() [0xa0182e]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x393) [0xa02b73]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6eb838]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdd51fe]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f15e88a9e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f15e7bbc3fd]

SECONDARY1:

Fri May 30 15:15:01.711 [conn1638743]  ez3hl2bej7d81o4qrqigbusft3u7qgm5xahbcbkiat5pjn3f.XiaoHuaImage Assertion failure !e.eoo() src/mongo/db/../bson/bsonobjbuilder.h 92
0xde8c31 0xda9c9d 0x9f4960 0x9f110b 0xb725a5 0xc45f61 0xa8a7a0 0xa8ab4a 0xa86a61 0xa8c0be 0xa8ef28 0xa0182e 0xa02b73 0x6eb838 0xdd51fe 0x7fc428052e9a 0x7fc427365ccd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde8c31]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda9c9d]
 /usr/bin/mongod(_ZNK5mongo14KeyGeneratorV18_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_10BSONObjCmpESaIS9_EEiSB_+0x6c0) [0x9f4960]
 /usr/bin/mongod(_ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_10BSONObjCmpESaIS1_EE+0x38b) [0x9f110b]
 /usr/bin/mongod(_ZNK5mongo16FieldRangeVector10firstMatchERKNS_7BSONObjE+0xd5) [0xb725a5]
 /usr/bin/mongod(_ZN5mongo12ScanAndOrder3addERKNS_7BSONObjEPKNS_7DiskLocE+0x51) [0xc45f61]
 /usr/bin/mongod(_ZN5mongo20ReorderBuildStrategy19_handleMatchNoDedupEPNS_13ResultDetailsE+0x70) [0xa8a7a0]
 /usr/bin/mongod(_ZN5mongo20ReorderBuildStrategy11handleMatchEPNS_13ResultDetailsE+0x5a) [0xa8ab4a]
 /usr/bin/mongod(_ZN5mongo20QueryResponseBuilder8addMatchEv+0xb1) [0xa86a61]
 /usr/bin/mongod(_ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_12ChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE+0x3be) [0xa8c0be]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1ac8) [0xa8ef28]
 /usr/bin/mongod() [0xa0182e]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x393) [0xa02b73]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6eb838]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdd51fe]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fc428052e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fc427365ccd]



 Comments   
Comment by Ramon Fernandez Marina [ 07/Jul/14 ]

Marking ticket as resolved, as this issue has been fixed in 2.6.

Comment by Jason McCay [ 03/Jul/14 ]

Thomas ... thanks for the update on this. One of the customers that we support was seeing this, so I am not exactly sure what they were doing to cause it. However, they were using 2.4.x versions of MongoDB as well as Mongoid.

Thanks for verifying this and showing how it is improved in 2.6. Grateful for your efforts.

Comment by Thomas Rueckstiess [ 30/Jun/14 ]

Okay I was able to reproduce this with an empty field name (""), I think that's what you meant.

Mon Jun 30 18:07:15.891 [conn7] assertion 0 assertion src/mongo/db/../bson/bsonobjbuilder.h:92 ns:test.docs query:{ query: {}, orderby: { : 1.0 } }
Mon Jun 30 18:07:40.598 [conn7]  test.docs Assertion failure !e.eoo() src/mongo/db/../bson/bsonobjbuilder.h 92
0x1004508ab 0x10042a4fc 0x100034ed5 0x1001e7280 0x1001e77f4 0x1001e4072 0x1002d5fd6 0x10034bb3e 0x100240c00 0x10023e9cd 0x100241a40 0x100241fb7 0x100244fb2 0x1001f0b2f 0x10000b8b2 0x100446279 0x100483445 0x7fff8c7e2899 0x7fff8c7e272a 0x7fff8c7e6fc9
 0   mongod                              0x00000001004508ab _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010042a4fc _ZN5mongo12verifyFailedEPKcS1_j + 284
 2   mongod                              0x0000000100034ed5 _ZN5mongo14BSONObjBuilder8appendAsERKNS_11BSONElementERKNS_10StringDataE + 325
 3   mongod                              0x00000001001e7280 _ZNK5mongo14KeyGeneratorV18_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_10BSONObjCmpESaIS9_EEiSB_ + 658
 4   mongod                              0x00000001001e77f4 _ZNK5mongo14KeyGeneratorV17getKeysERKNS_7BSONObjERSt3setIS1_NS_10BSONObjCmpESaIS1_EE + 396
 5   mongod                              0x00000001001e4072 _ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_10BSONObjCmpESaIS1_EE + 132
 6   mongod                              0x00000001002d5fd6 _ZNK5mongo16FieldRangeVector10firstMatchERKNS_7BSONObjE + 204
 7   mongod                              0x000000010034bb3e _ZN5mongo12ScanAndOrder3addERKNS_7BSONObjEPKNS_7DiskLocE + 78
 8   mongod                              0x0000000100240c00 _ZN5mongo20ReorderBuildStrategy19_handleMatchNoDedupEPNS_13ResultDetailsE + 136
 9   mongod                              0x000000010023e9cd _ZN5mongo20ReorderBuildStrategy11handleMatchEPNS_13ResultDetailsE + 79
 10  mongod                              0x0000000100241a40 _ZN5mongo20QueryResponseBuilder8addMatchEv + 204
 11  mongod                              0x0000000100241fb7 _ZN5mongo23queryWithQueryOptimizerEiRKSsRKNS_7BSONObjERNS_5CurOpES4_S4_RKN5boost10shared_ptrINS_11ParsedQueryEEES4_RKNS_12ChunkVersionERNS7_10scoped_ptrINS_25PageFaultRetryableSectionEEERNSG_INS_19NoPageFaultsAllowedEEERNS_7MessageE + 1287
 12  mongod                              0x0000000100244fb2 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 5346
 13  mongod                              0x00000001001f0b2f _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1519
 14  mongod                              0x000000010000b8b2 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 198
 15  mongod                              0x0000000100446279 _ZN5mongo17PortMessageServer17handleIncomingMsgEPv + 1657
 16  mongod                              0x0000000100483445 thread_proxy + 229
 17  libsystem_pthread.dylib             0x00007fff8c7e2899 _pthread_body + 138
 18  libsystem_pthread.dylib             0x00007fff8c7e272a _pthread_struct_init + 0
 19  libsystem_pthread.dylib             0x00007fff8c7e6fc9 thread_start + 13
Mon Jun 30 18:07:40.602 [conn7] assertion 0 assertion src/mongo/db/../bson/bsonobjbuilder.h:92 ns:test.docs query:{ query: {}, orderby: { : 1.0 } }
Mon Jun 30 18:07:41.416 [conn7] end connection 127.0.0.1:55819 (6 connections now open)

This also happens when querying on the primary node.

In 2.6.x this is caught and returns an error:

replset:PRIMARY> db.docs.find().sort({"": 1})
error: {
	"$err" : "Can't canonicalize query: BadValue bad sort specification",
	"code" : 17287
}

Comment by Thomas Rueckstiess [ 30/Jun/14 ]

Hi jasonmccay,

How did you construct a document that is missing a field name? What driver do you use?

Regards,
Thomas

Comment by Jason McCay [ 14/Jun/14 ]

Just an observation ... but we have seen this error in the past (MongoDB 2.4.8) and it was caused by a malformed query. In our case, it was an $orderby that that was missing a field argument ... so it looked like: $orderby: { : 1 } }.

Comment by Ramon Fernandez Marina [ 03/Jun/14 ]

Hi wujiangcheng,

we'll need more information to try to diagnose this problem. Can you please send us the logs from the primary and the secondaries covering the last few hours leading up to this issue? Also, can you send us the full query that's triggering the issue?

You may want to validate your data; be aware that this operation is resource-intensive and may have an impact on the performance of your MongoDB instance, so you may want to perform it one secondary at a time, during off-peak hours, and/or on a maintenance window.

Regards,
Ramón.

Generated at Thu Feb 08 03:33:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.