Overview

In the SBE hash-join lowering of $lookup (localField/foreignField equality lookup), the outer (probe) key is converted to an ArraySet once per outer document via the VM builtin arrayToSet. This per-row set construction is redundant and can be skipped, giving a measurable throughput improvement on the $lookup hot path.

Background

While investigating the Aggregation.Lookup.LocalArrayOfObject microbenchmark (task aggregation_read_commands, variant "Standalone inMemory ARM"), an on-CPU flamegraph of the SBE path (server default internalQueryFrameworkControl=trySbeRestricted, which lowers this $lookup to EQ_LOOKUP with the HashJoin strategy) showed ~96.7% of time in real runtime execution and ~27-33% of total CPU in tcmalloc allocation/free. A large share of that allocation churn is the per-outer-row construction of an ArraySet (an absl::flat_hash_set) for the probe key: bucket allocation + per-element hashing + value cloning (push_back_clone/copyValue), for a key array that in this workload has only ~3 elements.

The stage builder emits this in buildKeySetForLocal() via sbe::EFn::kArrayToSet (src/mongo/db/query/stage_builder/sbe/gen_lookup.cpp), and the resulting ProjectStage runs ByteCode::builtinArrayToSet() (src/mongo/db/exec/sbe/vm/vm_builtin_array.cpp) for every outer document.

Why it is redundant

The purpose of the outer-key arrayToSet is to de-duplicate the outer key values. But LookupHashTableIter::initSearchArray() (src/mongo/db/exec/sbe/stages/lookup_hash_table.cpp) already de-duplicates the matched buffer indices while probing (it iterates the outer key elements one by one and collects matches into a de-duplicating container). Pre-deduping the outer key values therefore only saves a few redundant hash-table find() calls (negligible for small key arrays) while costing a full ArraySet allocation every row.

Proposed change

Add a convertArrayToSet parameter to buildKeySetForLocal() (default true) and pass false from buildHashJoinLookupStage() so the outer/probe key is left as a plain array. Only the hash-join path is changed; the nested-loop-join and index-join paths keep their existing set conversion (the scalar-outer probe path, initSearchScalar(), returns the per-key match vector without index-dedup, so the inner/foreign-side arrayToSet in buildKeySetForForeign() is retained).

Measured impact

Local prototype, ARM, inMemory, trySbeRestricted, 5 trials x 20s, Aggregation.Lookup.LocalArrayOfObject:
* baseline: ops_per_sec = 123.0 (stdev 2.3)
* with change (A): ops_per_sec = 130.7 (stdev 0.9) -- +6.3%, and lower run-to-run variance

Correctness verified against a harness that exercises interior/boundary docs, a duplicate-local-key doc (foreignKey.x=[10,10,11] must match {{_id.x in

}} exactly once), and a no-match doc (empty as array): PASS.

Files

* src/mongo/db/query/stage_builder/sbe/gen_lookup.cpp -- buildKeySetForLocal() + hash-join call site

Notes / follow-up

* Prototype patch attached (changeA_skip_outer_arraytoset.patch).
* This touches the shared hash-join key path used by $lookup and $lookup+$unwind, including collation cases -- run the full SBE $lookup correctness suite before merging.
* Independent of, and additive with, the companion ticket (replacing the std::set match collector with a reused sorted vector).