Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-93463

Update $stdDev implementation to make it compatible with SBE window stage algorithm

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • Fully Compatible
    • ALL
    • Hide 
      const doc = [
{ _id: 0, num: NaN},
{ _id: 1, num: 1},
{ _id: 2, num: 2},
];

db.coll.drop();
assert.commandWorked(db.coll.insertMany(doc));

const pipeline = [
{$setWindowFields: {partitionBy: null, sortBy: {_id: 1}, output: {abc: {$stdDevSamp: "$num", window: {documents: [1, 2]}}}}},
{$limit: 1},
];

assert.commandWorked(db.adminCommand({
    setParameter: 1,
    internalQueryFrameworkControl: "forceClassicEngine",
}));

const classic = db.coll.aggregate(pipeline).toArray();
jsTestLog("classic");
jsTestLog(classic[0]);

assert.commandWorked(db.adminCommand({
    setParameter: 1,
    internalQueryFrameworkControl: "trySbeEngine",
}));

const sbe = db.coll.aggregate(pipeline).toArray();
jsTestLog("SBE");
jsTestLog(sbe[0]);

const explain = db.coll.explain().aggregate(pipeline);
const stages = explain.queryPlanner.winningPlan.slotBasedPlan.stages;
jsTestLog("SBE plan");
print(stages);
      Show
      const doc = [ { _id: 0, num: NaN}, { _id: 1, num: 1}, { _id: 2, num: 2}, ]; db.coll.drop(); assert .commandWorked(db.coll.insertMany(doc)); const pipeline = [ {$setWindowFields: {partitionBy: null , sortBy: {_id: 1}, output: {abc: {$stdDevSamp: "$num" , window: {documents: [1, 2]}}}}}, {$limit: 1}, ]; assert .commandWorked(db.adminCommand({ setParameter: 1, internalQueryFrameworkControl: "forceClassicEngine" , })); const classic = db.coll.aggregate(pipeline).toArray(); jsTestLog( "classic" ); jsTestLog(classic[0]); assert .commandWorked(db.adminCommand({ setParameter: 1, internalQueryFrameworkControl: "trySbeEngine" , })); const sbe = db.coll.aggregate(pipeline).toArray(); jsTestLog( "SBE" ); jsTestLog(sbe[0]); const explain = db.coll.explain().aggregate(pipeline); const stages = explain.queryPlanner.winningPlan.slotBasedPlan.stages; jsTestLog( "SBE plan" ); print(stages);
    • 0

      Currently, for a document window like [1, 1], the SBE window stage will add document 0, 1, then remove document 0. This algorithm doesn't work well with the particular implementation of $stdDev on some corner cases. The removable stddev keeps track of the total count and non-finite count in the window frame, but the numeric state is calculated based on the total count. If a non-finite number (NaN, Inf) is added and then removed from the window frame, the total count includes those non-finite numbers and it affects the final numeric result.

      The simplest way is to update the $stdDev implementation, so that we do not count non-finite numbers in the window frame during numeric calculation. The result is still correct if there are non-finite numbers in the window frame, because the non-finite count is still tracked and we should just return null. This keeps the stddev result correct yet makes it compatible with the SBE window stage algorithm.

            Assignee:
            rui.liu@mongodb.com Rui Liu
            Reporter:
            rui.liu@mongodb.com Rui Liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: