Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63018

Have ExpressionAdd::evaluate() return intermediate result for pipeline optimization constant folding

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 5.2.0, 4.2.17, 5.0.5, 5.1.1, 4.4.12
    • Component/s: None
    • None
    • ALL
    • Hide

      Starts mongod with --setParameter enableTestCommands=1

      // Inserts any decimal value
      db.opt.insert({n: NumberDecimal("0")})
      
      // Turns off pipeline optimization
      db.adminCommand({configureFailPoint: "disablePipelineOptimization", mode: "alwaysOn"});
      
      // In mixed type $add, if there's any decimal input ($n), the final result is promoted to decimal.
      // While doing addition, the double double sum algorithm accumulates non-decimal values into a separate nonDecimalTotal state.
      // So, we don't lose precision here.
      // Note that int64 can't be exactly expressed.
      db.opt.aggregate({$project: {_id: 0, o: {$add: ["$n", 0, NumberLong("111111111111111111")]}}});
      { "o" : NumberDecimal("111111111111111111") }
      
      // Enables pipeline optimization
      db.adminCommand({configureFailPoint: "disablePipelineOptimization", mode: "off"});
      
      // Here an interesting thing happens when a pipeline is optimized. This is the SBE result.
      // While optimizing the pipeline, double 0 and NumberLong("111111111111111111") is optimized into a single double value.
      // And the SBE binary $add algorithm promotes the double to a decimal with 15 digit precision and so we lose 3 digit precision here.
      // This issue happens because the optimization passes the final result instead of the intermediate results in the form of
      // decimalTotal and nonDecimalTotal.
      db.adminCommand({getParameter: 1, internalQueryForceClassicEngine: 1});
      { "internalQueryForceClassicEngine" : false, "ok" : 1 }
      db.opt.aggregate({$project: {_id: 0, o: {$add: ["$n", 0, NumberLong("111111111111111111")]}}});
      { "o" : NumberDecimal("111111111111111000") }
      
      // This is the classic engine's result.
      // The classic engine's double double sum algorithm promotes the optimization result double value to a decimal with 34 digit 
      // precision. So, we keep the double value as close as possible to the addition result which is 111111111111111104.0.
      // But the real issue here is that the optimization passes the final result instead of the intermediate results in the form of
      // decimalTotal and nonDecimalTotal.
      db.adminCommand({setParameter: 1, internalQueryForceClassicEngine: true});
      { "was" : false, "ok" : 1 }
      db.opt.aggregate({$project: {_id: 0, o: {$add: ["$n", 0, NumberLong("111111111111111111")]}}});
      { "o" : NumberDecimal("111111111111111104") }
      
      Show
      Starts mongod with --setParameter enableTestCommands=1 // Inserts any decimal value db.opt.insert({n: NumberDecimal( "0" )}) // Turns off pipeline optimization db.adminCommand({configureFailPoint: "disablePipelineOptimization" , mode: "alwaysOn" }); // In mixed type $add, if there's any decimal input ($n), the final result is promoted to decimal. // While doing addition, the double double sum algorithm accumulates non-decimal values into a separate nonDecimalTotal state. // So, we don't lose precision here. // Note that int64 can't be exactly expressed. db.opt.aggregate({$project: {_id: 0, o: {$add: [ "$n" , 0, NumberLong( "111111111111111111" )]}}}); { "o" : NumberDecimal( "111111111111111111" ) } // Enables pipeline optimization db.adminCommand({configureFailPoint: "disablePipelineOptimization" , mode: "off" }); // Here an interesting thing happens when a pipeline is optimized. This is the SBE result. // While optimizing the pipeline, double 0 and NumberLong( "111111111111111111" ) is optimized into a single double value. // And the SBE binary $add algorithm promotes the double to a decimal with 15 digit precision and so we lose 3 digit precision here. // This issue happens because the optimization passes the final result instead of the intermediate results in the form of // decimalTotal and nonDecimalTotal. db.adminCommand({getParameter: 1, internalQueryForceClassicEngine: 1}); { "internalQueryForceClassicEngine" : false , "ok" : 1 } db.opt.aggregate({$project: {_id: 0, o: {$add: [ "$n" , 0, NumberLong( "111111111111111111" )]}}}); { "o" : NumberDecimal( "111111111111111000" ) } // This is the classic engine's result. // The classic engine's double double sum algorithm promotes the optimization result double value to a decimal with 34 digit // precision. So, we keep the double value as close as possible to the addition result which is 111111111111111104.0. // But the real issue here is that the optimization passes the final result instead of the intermediate results in the form of // decimalTotal and nonDecimalTotal. db.adminCommand({setParameter: 1, internalQueryForceClassicEngine: true }); { "was" : false , "ok" : 1 } db.opt.aggregate({$project: {_id: 0, o: {$add: [ "$n" , 0, NumberLong( "111111111111111111" )]}}}); { "o" : NumberDecimal( "111111111111111104" ) }

      The proposed fix is to make ExpressionAdd::evaluate() return intermediate result in the form of decimalTotal and nonDecimalTotal for pipeline optimization constant folding. For example,

      {decimalTotal: val1, nonDecimalTotal: val2}

            Assignee:
            davis.haupt@mongodb.com Davis Haupt (Inactive)
            Reporter:
            yoonsoo.kim@mongodb.com Yoon Soo Kim (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: