[SERVER-53074] [SBE] Add support for $map aggregation pipeline operator Created: 25/Nov/20  Updated: 24/Apr/21  Resolved: 24/Apr/21

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Drew Paroski Assignee: Drew Paroski
Resolution: Duplicate Votes: 0
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-51823 Use classic engine to evaluate querie... Closed
duplicates SERVER-53036 [SBE] Investigate why several sys-per... Closed
Related
is related to SERVER-51655 Investigate sys-perf benchmark perfor... Closed
Participants:

 Description   

When I tried to run the "word_count.js" benchmark (from https://github.com/10gen/workloads) and with SBE mode enabled, mongod crashed due to an invariant failure at line 1539 at "src/mongo/db/query/sbe_stage_builder_expression.cpp": 

  invariant(it != _context->environment.end());

 

I attached gdb to mongod (debug build) and then I ran the "work_count.js" benchmark with SBE mode enabled to reproduce the crash, and here was the stack trace I got: 

#0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
..
#3 0x0000564379faf11d in mongo::invariantWithLocation<bool> (testOK=@0x7fca449670e8: false,
 expr=0x56437e6eedc8 "it != _context->environment.end()",
 file=0x56437e6ee668 "src/mongo/db/query/sbe_stage_builder_expression.cpp", line=1539)
 at src/mongo/util/invariant.h:69
#4 0x000056437b83ecea in mongo::stage_builder::(anonymous namespace)::ExpressionPostVisitor::visit (
 this=0x7fca44967370, expr=0x7fca15da4f20)
 at src/mongo/db/query/sbe_stage_builder_expression.cpp:1539
#5 0x000056437c328da0 in mongo::ExpressionFieldPath::acceptVisitor (this=0x7fca15da4f20,
 visitor=0x7fca44967370) at src/mongo/db/pipeline/expression.h:1441
#6 0x000056437b84a45e in mongo::stage_builder::(anonymous namespace)::ExpressionWalker::postVisit (
 this=0x7fca449673a0, expr=0x7fca15da4f20)
 at src/mongo/db/query/sbe_stage_builder_expression.cpp:2728
#7 0x000056437b84bd84 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca15da4f20)
 at src/mongo/db/pipeline/expression_walker.h:62
#8 0x000056437b84bd63 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca3d11dd80)
 at src/mongo/db/pipeline/expression_walker.h:59
#9 0x000056437b84bd63 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca15da5520)
 at src/mongo/db/pipeline/expression_walker.h:59
#10 0x000056437b84aaf6 in mongo::stage_builder::generateExpression (opCtx=0x7fca2586f720,
 expr=0x7fca15da5520, stage=..., slotIdGenerator=0x7fc9d50b6f48, frameIdGenerator=0x7fc9d50b6f58,
 rootSlot=2, env=0x7fc9fe7a9120, planNodeId=2, relevantSlots=0x7fca44967848)
 at src/mongo/db/query/sbe_stage_builder_expression.cpp:2797
#11 0x000056437b87d8f3 in mongo::stage_builder::(anonymous namespace)::ProjectionTraversalPostVisitor::visit (this=0x7fca44967750, node=0x7fc9cc5010e0)
 at src/mongo/db/query/sbe_stage_builder_projection.cpp:243
#12 0x000056437c0feb18 in mongo::projection_ast::ExpressionASTNode::acceptVisitor (this=0x7fc9cc5010e0,
 visitor=0x7fca44967750) at src/mongo/db/query/projection_ast.h:293
#13 0x000056437b87f298 in mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker::postVisit (this=0x7fca44967760, node=0x7fc9cc5010e0) at src/mongo/db/query/sbe_stage_builder_projection.cpp:382
#14 0x000056437b87fb20 in mongo::tree_walker::walk<true, mongo::projection_ast::ASTNode, mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker> (node=0x7fc9cc5010e0, walker=0x7fca44967760)
 at src/mongo/db/query/tree_walker.h:70
#15 0x000056437b87faff in mongo::tree_walker::walk<true, mongo::projection_ast::ASTNode, mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker> (node=0x7fc9eb38d358, walker=0x7fca44967760)
 at src/mongo/db/query/tree_walker.h:67
#16 0x000056437b87f4c7 in mongo::stage_builder::generateProjection (opCtx=0x7fca2586f720,
 projection=0x7fc9eb38d358, stage=..., slotIdGenerator=0x7fc9d50b6f48,
 frameIdGenerator=0x7fc9d50b6f58, inputVar=2, env=0x7fc9fe7a9120, planNodeId=2)
 at src/mongo/db/query/sbe_stage_builder_projection.cpp:413
#17 0x000056437b8138ef in mongo::stage_builder::SlotBasedStageBuilder::buildProjectionDefault (
 this=0x7fc9d50b6f20, root=0x7fc9eb38d2e0, reqs=...) at src/mongo/db/query/sbe_stage_builder.cpp:708
..
#25 0x000056437b816af6 in mongo::stage_builder::SlotBasedStageBuilder::build (this=0x7fc9d50b6f20,
 root=0x7fc9eb38d2e0, reqs=...) at src/mongo/db/query/sbe_stage_builder.cpp:1052
#26 0x000056437b80ea57 in mongo::stage_builder::SlotBasedStageBuilder::build (this=0x7fc9d50b6f20,
 root=0x7fc9eb38d2e0) at src/mongo/db/query/sbe_stage_builder.cpp:159
#27 0x000056437b88a4c3 in mongo::stage_builder::buildSlotBasedExecutableTree (opCtx=0x7fca2586f720,
 collection=..., cq=..., solution=..., yieldPolicy=0x7fca2445f160,
 needsTrialRunProgressTracker=false) at src/mongo/db/query/stage_builder_util.cpp:74
#28 0x000056437b7b8421 in mongo::(anonymous namespace)::SlotBasedPrepareExecutionHelper::buildExecutableTree (this=0x7fca449688f0, solution=..., needsTrialRunProgressTracker=false)
 at src/mongo/db/query/get_executor.cpp:956
#29 0x000056437b7b7d86 in mongo::(anonymous namespace)::SlotBasedPrepareExecutionHelper::buildExecutableTree (this=0x7fca449688f0, solution=...) at src/mongo/db/query/get_executor.cpp:904
#30 0x000056437b7c4e87 in mongo::(anonymous namespace)::PrepareExecutionHelper<std::pair<std::unique_ptr<mongo::sbe::PlanStage, std::default_delete<mongo::sbe::PlanStage> >, mongo::stage_builder::PlanStageData>, mongo::(anonymous namespace)::SlotBasedPrepareExecutionResult>::prepare (this=0x7fca449688f0)
 at src/mongo/db/query/get_executor.cpp:672
#31 0x000056437b7b8ebb in mongo::(anonymous namespace)::getSlotBasedExecutor (opCtx=0x7fca2586f720,
 collection=0x7fca44969938, cq=...,
 requestedYieldPolicy=mongo::PlanYieldPolicy::YieldPolicy::YIELD_AUTO, plannerOptions=0)
 at src/mongo/db/query/get_executor.cpp:1068
#32 0x000056437b7b9660 in mongo::getExecutor (opCtx=0x7fca2586f720, collection=0x7fca44969938,
 canonicalQuery=..., yieldPolicy=mongo::PlanYieldPolicy::YieldPolicy::YIELD_AUTO, plannerOptions=0)
 at src/mongo/db/query/get_executor.cpp:1116
..
#36 0x000056437b791def in mongo::PipelineD::prepareExecutor (expCtx=..., collection=..., nss=...,
 pipeline=0x7fc9f55427c0, sortStage=..., rewrittenGroupStage=..., unavailableMetadata=...,
 queryObj=..., skipThenLimit=..., aggRequest=0x7fc96b543058, matcherFeatures=@0x56437e6a8198: 57,
 hasNoRequirements=0x7fca44969042) at src/mongo/db/pipeline/pipeline_d.cpp:788
..

I came up with a small repro for this crash - see the "Steps To Reproduce" section.

I also tried out running a modified version of the small repro where I replaced "$$word" with 1, like so:

db.adminCommand({setParameter:1, internalQueryEnableSlotBasedExecutionEngine:true});
db.foo.insert({ "t" : "blah" });
db.foo.aggregate([{$project: {emits: {$map: {input: {$split: ["$t", " "]}, as: "word", in: {k: 1, v: 1}}}}}]);

When I ran the commands above, mongod did not crash, but I got the following error message: 

"Expression is not supported in SBE: $map"

Thus it looks like even if the crash were fixed, the "word_count.js" benchmark would still fail because SBE doesn't support the $map aggregation pipeline operator yet.

The goal of this task is: (1) to implement SBE support for the $map aggregation pipeline operator; and (2) to fix the "invariant(it != _context->environment.end())" crash.



 Comments   
Comment by Drew Paroski [ 24/Apr/21 ]

SERVER-51823 fixed things so that queries that contain the $map operator will fall back to the classic execution engine when SBE is enabled (because SBE does not yet support the $map operator as of 4/23/2021).

I ran the simplified "repro" steps from the description of this task with SBE enabled, and I verified that "db.foo.aggregate([{$project: {emits: {$map: {input: {$split: ["$t", " "]}, as: "word", in: {k: 1, v: 1}}}]);}}" no longer fails when SBE is enabled (it correctly falls back to the classic execution engine).

The SERVER-51019 task exists to track what is needed to eventually add SBE support for the $map operator in the future. The SERVER-53036 task exists to make sure we re-run the bestbuy benchmarks to see if there are any functional failures that are still happening when SBE is enabled. Thus, we no longer need this task.

Closing as a duplicate of SERVER-51823 and SERVER-53036.

Generated at Thu Feb 08 05:29:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.