[SERVER-14123] some operations can create BSON object larger than the 16MB limit Created: 31/May/14  Updated: 09/Mar/21  Resolved: 09/Jun/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 2.6.4, 2.7.4

Type: Bug Priority: Major - P3
Reporter: Kevin Locke Assignee: David Storch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-14314 writeConcern complains about too larg... Closed
Related
related to SERVER-54710 Large number of $or clauses can creat... Closed
Tested
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Install 2.6.1 and run the following script:

db.createCollection("orders");
 
var ids = [];
// On my system 215295 is the highest number that works
for (var i = 0; i < 215500; ++i) {
  ids.push(new ObjectId());
}
db.orders.find({_id: {$in: ids}}).toArray();

Participants:
Linked BF Score: 0

 Description   
Issue Status as of Jul 22, 2014

ISSUE SUMMARY
MongoDB collects statistics about every operation. These statistics are converted into a BSON format for presentation to the user in the following scenarios:

In these scenarios, MongoDB fails to check that the BSON format doesn't grow beyond the 16MB limit. This can be the case for large query predicates, such as queries that contain an $in with thousands of elements.

USER IMPACT
When statistics grow beyond 16MB, queries fail with an error message and an assertion error is printed on the logs.

WORKAROUNDS
N/A

AFFECTED VERSIONS
MongoDB 2.6 production releases up to 2.6.3 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
Keep the amount of statistics returned to the user within the 16MB limit for BSON objects, and add a warning message when these statistics are truncated.

Original description

Running a query for a large number of _ids results in error code 10334 due to a BSON object larger than the maximum size. The error is not in the query object itself (which is ~4MB in the example) but internally from what appears to be the query planner. Such a query generates the following message in the log:

2014-05-30T22:01:54.013-0600 [conn11] Assertion: 10334:BSONObj size: 17805128 (0x10FAF48) is invalid. Size must be between 0 and 16793600(16MB) First element: type: "FETCH"
2014-05-30T22:01:54.026-0600 [conn11] test.trades 0x11c0e91 0x1163109 0x11477e6 0x1147d3c 0x76d23b 0xd16f1a 0xd18092 0xda2043 0xd4cb1c 0xb97322 0xb99902 0x76b6af 0x117720b 0x7f7e2bc4f062 0x7f7e2af56c1d 
 ./mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91]
 ./mongod(_ZN5mongo10logContextEPKc+0x159) [0x1163109]
 ./mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11477e6]
 ./mongod() [0x1147d3c]
 ./mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x41b) [0x76d23b]
 ./mongod() [0xd16f1a]
 ./mongod(_ZN5mongo11explainPlanERKNS_14PlanStageStatsEPPNS_11TypeExplainEb+0x12) [0xd18092]
 ./mongod(_ZNK5mongo20SingleSolutionRunner7getInfoEPPNS_11TypeExplainEPPNS_8PlanInfoE+0x53) [0xda2043]
 ./mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x133c) [0xd4cb1c]
 ./mongod() [0xb97322]
 ./mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb99902]
 ./mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b6af]
 ./mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x117720b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8062) [0x7f7e2bc4f062]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7e2af56c1d]

The issue is partially that a valid query object significantly below the 16MB limit is rejected, but to me the issue is primarily that this appears to be an arbitrary query limit which can't be determined a priori and is likely dependent on implementation details that may change between versions.

Is there a way to determine if such queries will run (for particular server versions or for all versions), or Is the only solution for large queries to try them and cut them in half (whatever that means for queries on multiple keys) whenever they don't work?

Updating the documentation to make a note of this limitation could also be useful to others in a similar situation in the future.

Thanks for considering,
Kevin



 Comments   
Comment by Githook User [ 25/Jul/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14123 Delete jstests/sharding/explain_large.js

explain_large.js spawns 18 shards and runs a query that allocates
300MB+ of memory on each. This causes OOM conditions on MCI's 32-bit
buildvariants (limited to 4GB of addressable memory, no swap).

(cherry picked from commit 4dd1c9e70cc4a17d233c32180933da24b4c57ef5)
Branch: v2.6
https://github.com/mongodb/mongo/commit/3a38fc173f1ebd74ce5ed5b28e1459ac1154adba

Comment by Githook User [ 25/Jul/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14123 Delete jstests/sharding/explain_large.js

explain_large.js spawns 18 shards and runs a query that allocates
300MB+ of memory on each. This causes OOM conditions on MCI's 32-bit
buildvariants (limited to 4GB of addressable memory, no swap).
Branch: master
https://github.com/mongodb/mongo/commit/4dd1c9e70cc4a17d233c32180933da24b4c57ef5

Comment by Githook User [ 24/Jul/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14123 prevent explain BSON from growing too large on mongos

(cherry picked from commit ffc53f46956eb52a86efd51620dede98e1b48444)
Branch: v2.6
https://github.com/mongodb/mongo/commit/4f284938fea696e1d1c5cf6c48642ee227c51fbb

Comment by Githook User [ 24/Jul/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14123 prevent explain BSON from growing too large on mongos
Branch: master
https://github.com/mongodb/mongo/commit/ffc53f46956eb52a86efd51620dede98e1b48444

Comment by Githook User [ 23/Jul/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14123 prevent the explain BSONObj from growing too large

(cherry picked from commit 906cfb13a337fdbdf1b94b773390c4230b059fec)
Branch: v2.6
https://github.com/mongodb/mongo/commit/b89d01e649241f932fafb0bb3777bea455ead397

Comment by Daniel Pasette (Inactive) [ 23/Jul/14 ]

Dave will backport the current fix to 2.6 as well as work on a separate patch for mongos to ensure a similar thing can't happen there.

Comment by Kevin Locke [ 09/Jun/14 ]

It's great to see a fix so quickly. Thanks for working on it!

Comment by Githook User [ 09/Jun/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14123 prevent the explain BSONObj from growing too large
Branch: master
https://github.com/mongodb/mongo/commit/906cfb13a337fdbdf1b94b773390c4230b059fec

Comment by Asya Kamsky [ 31/May/14 ]

Thank you for reporting this bug.

EDIT

Just noticed the reproducing script, so please disregard the previous request. I was able to reproduce the assert.

Asya

Generated at Thu Feb 08 03:33:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.