[SERVER-75927] Shards see different values for 'let' parameter when using $rand Created: 10/Apr/23  Updated: 29/Oct/23  Resolved: 10/Oct/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: David Storch Assignee: David Percy
Resolution: Fixed Votes: 0
Labels: query-director-triage, query-product-scope-1, query-product-urgency-1, query-product-value-2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-75356 explain command for a find with $expr... Closed
is related to SERVER-82042 Sharded write commands should evaluat... Closed
is related to SERVER-48905 Allow $rand in command-level 'let' Closed
Assigned Teams:
Query Optimization
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

I wrote the following script to demonstrate the issue:

(function() {
"use strict";
 
const dbName = "test";
 
const st = new ShardingTest({shards: 2});
const db = st.s.getDB(dbName);
 
const coll = db.coll;
coll.drop();
 
assert.commandWorked(coll.insert([{x: -2}, {x: 2}]));
assert.commandWorked(coll.createIndex({x: 1}));
 
st.shardColl(coll, {x: 1}, {x: 0}, {x: 1});
 
const findCmd = {
    find: coll.getName(),
    projection: {_id: 0, rand: "$$randVal"},
    let : {randVal: {$rand: {}}}
};
let result = assert.commandWorked(db.runCommand(findCmd));
let batch = result.cursor.firstBatch;
assert.eq(batch.length, 2, result);
 
// Assert that both shards see the same pseudo-random value. This assertion fails.
assert.eq(batch[0].rand, batch[1].rand, result);
 
st.stop();
}());

I'm running it locally with a resmoke.py invocation like this:

python3 buildscripts/resmoke.py run --suites=sharding repro_let_rand.js

Sprint: QO 2023-06-26, QO 2023-07-10, QO 2023-07-24, QO 2023-08-07, QO 2023-08-21, QO 2023-09-04, QO 2023-09-18, QO 2023-10-02, QO 2023-10-16
Participants:

 Description   

In SERVER-48905 we extended the let parameters mechanism so that it works in combination with the $rand expression. For example, the following query was made legal by SERVER-48905:

const findCmd = {
    find: coll.getName(),
    projection: {_id: 0, rand: "$$randVal"},
    let : {randVal: {$rand: {}}}
};

The intended semantics is that the $rand expression is evaluated just once up front and the result is treated as a constant for the remainder of the execution of the query.

In sharded contexts, mongos should generate the random number and pass it to the shards so that all shards see the same random number. This is not working as expected. Glancing at the code, it looks like we are sending the full $rand expression to each shard. As a result, each shard generates its own random number. My script in "steps to reproduce" has a full example of the problem.



 Comments   
Comment by Githook User [ 10/Oct/23 ]

Author:

{'name': 'David Percy', 'email': 'david.percy@mongodb.com', 'username': 'dpercy'}

Message: SERVER-75927 Evaluate command-level let only once, even when sharded

Command-level let bindings are evaluated once for the whole command. For
most expressions this is not observable, but for $rand it is. If you
need a random value drawn once and then used consistently within the
query, you use a command-level binding.

Previously this was only working correctly for sharded aggregate queries:
we were evaluating the let bindings on each shard separately. This commit
fixes the find command: it ensures we evaluate the expressions once on
mongos and then serialize the resulting values to the shards.

This commit also includes test cases to demonstrate the same bug in
write commands, to be fixed separately (SERVER-82042).
Branch: master
https://github.com/mongodb/mongo/commit/7533908c28d85da5356b2b9c49ff29d026cda2f1

Generated at Thu Feb 08 06:31:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.