-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Query Optimization
-
None
-
None
-
None
-
None
-
None
-
None
-
None
SERVER-127538 made the pipeline dependency graph soft-delete its nodes on truncation so that a rebuild can reuse each node's container allocations. That reuse currently only benefits containers that are populated in place on the node, namely base fields of dotted paths and small inherited scope maps.
The dependency sets of leaf Field nodes and of Stage nodes are still built in local FieldDependencies temporaries (processStageDependencies, processExpressionDependencies, processPathDependencies) and then moved into the node. The local allocates a fresh flat_hash_set on every rebuild (flat_hash_set has no SOO, so even a one-element set heap-allocates), and move-assigning it into a recycled node discards the node's retained buffer. The per-rebuild allocation traffic for these sets is therefore unchanged by SERVER-127538, and they are the most numerous moved-in containers in the graph.
Extending the reuse requires the builder to populate the destination node's set directly instead of going through a temporary, which likely means appending or recycling the node before its dependencies are computed and exposing an insert-style population path in declareField and processStage. Note the ordering constraints: processScope reads the stage-level dependencies before the Stage node is appended, and declareField computes leaf dependencies before the leaf node exists. For scale, in SERVER-127538 benchmarking a flat-path rebuild workload (all dependency sets moved in) showed no improvement from node recycling for exactly this reason, while BM_RebuildDependencyGraphWideStages improved ~8-9% mostly via the in-place-populated containers.
Optionally in the same change: remove the dormant NodeContainer::operator== (it has no callers, and instantiating it requires operator== on the node types).
- depends on
-
SERVER-127538 Keep memory allocations for containers in the pipeline dependency graph
-
- In Code Review
-