[SERVER-64597] We should produce SBE Object instead of BSON in $lookup and $group plans Created: 17/Mar/22  Updated: 29/Oct/23  Resolved: 13/Apr/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: Bug Priority: Blocker - P1
Reporter: Yoon Soo Kim Assignee: Nikita Lapkov (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-64877 $addToSet and $push fail in pushed do... Closed
Related
is related to SERVER-65265 Implement memory tracking for accumul... Closed
Backwards Compatibility: Fully Compatible
Sprint: QE 2022-04-04, QE 2022-04-18
Participants:

 Description   

$lookup and $group stages may produce an object which exceeds BSON size limit. In case of $lookup, this happens when sufficient number of documents from the foreign collection is pushed into the result array. In case of $group, $push and $addToSet accumulators can create large arrays as well.

Instead of returning BSON, these stages return Document, which does not have a size limitation. Current implementation of $lookup and $group in SBE use mkbson stage, which produces BSON. This means that some queries will successfully produce results in classic engine, but fail with "BSON size limit exceeded" exceptions in SBE.

Here are examples of such queries:

$lookup

for(let i = 0; i < 8; ++i) { db.b.insert({foreign: 1, bigField: 'x'.repeat(10 * 1024 * 1024)}) }
db.a.insert({local: 1})
db.a.aggregate([{$lookup: {from: 'b', localField: 'local', foreignField: 'foreign', as: 'out'}}, {$project: {foo: {$add: ["$local", 2]}}}]) 

$group

for(let i = 0; i < 8; ++i) { db.b.insert({foreign: 1, bigField: 'x'.repeat(10 * 1024 * 1024)}) }
db.b.aggregate([{$group: {_id: "$foreign", out: {$push: "$bigField"}}}, {$project: {a: {$add: [1, "$foreign"]}}}])

To mitigate this issue, we should:

  • Make SBE plans for $lookup and $group produce SBE Object instead of BSON
  • Implement translation from SBE Object to Document in PlanExecutor. This is to make sure that even if only $lookup or $group are pushed into SBE and the rest of the plan is executed by classic engine, we can still return larger-than-BSON-limit objects from SBE.

Separate ticket SERVER-65265 is created to ensure that arrays produced by $lookup and $group do not exceed maximum size limit specified by corresponding query knobs. This ticket is only about lifting BSON size limitation.



 Comments   
Comment by Githook User [ 12/Apr/22 ]

Author:

{'name': 'Nikita Lapkov', 'email': 'nikita.lapkov@mongodb.com', 'username': 'laplab'}

Message: SERVER-64597 Produce SBE Object instead of BSON in $lookup and $group
Branch: master
https://github.com/mongodb/mongo/commit/3780d57e1e378b55a887a43115c8d303e8962a6b

Generated at Thu Feb 08 06:00:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.