[SERVER-62485] Match the SBE $add behavior to the classic engine's or the other way around Created: 10/Jan/22  Updated: 06/Dec/22  Resolved: 01/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Yoon Soo Kim Assignee: Backlog - Query Execution
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-67282 $add expression should not use Double... Closed
is duplicated by SERVER-58978 Determine narrowest type before compu... Closed
is duplicated by SERVER-61679 $sum accumulator fails in aggregation... Closed
is duplicated by SERVER-63518 Equivalent find and agg queries produ... Closed
Related
Assigned Teams:
Query Execution
Operating System: ALL
Steps To Reproduce:

See description.

Sprint: QE 2022-04-04, QE 2022-02-07, QE 2022-02-21, QE 2022-03-07, QE 2022-03-21, QE 2022-01-24, QE 2022-04-18
Participants:
Linked BF Score: 146

 Description   

1st repro

> db.t.insert({a: NumberDecimal("0.0")});
> db.t.find({}, {_id: 0, o: {$add: [{$exp: NumberLong("4")}, "$a"]}});
{ "o" : NumberDecimal("54.5981500331442") }
> db.adminCommand({setParameter: 1, internalQueryForceClassicEngine: true});
> db.t.find({}, {_id: 0, o: {$add: [{$exp: NumberLong("4")}, "$a"]}});
{ "o" : NumberDecimal("54.59815003314423620395245961844921") }

2nd repro:

> db.t3.find();
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cb"), "a" : NumberDecimal("0.0"), "b" : 1e-15 }
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cc"), "a" : NumberDecimal("0.0"), "b" : 2e-15 }
> db.t3.find({}, {o: {$exp: NumberLong("4")}});
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cb"), "o" : 54.598150033144236 }
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cc"), "o" : 54.598150033144236 }
> db.t3.find({}, {o: {$add: [{$exp: NumberLong("4")}, "$a", "$b"]}});
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cb"), "o" : NumberDecimal("54.59815003314420100000000000000") }
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cc"), "o" : NumberDecimal("54.59815003314420200000000000000") }
> db.adminCommand({setParameter: 1, internalQueryForceClassicEngine: true});
{ "was" : false, "ok" : 1 }
> db.t3.find({}, {o: {$add: [{$exp: NumberLong("4")}, "$a", "$b"]}});
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cb"), "o" : NumberDecimal("54.59815003314423720395245961844929") }
{ "_id" : ObjectId("61ddd0403dea1cd22d1fb3cc"), "o" : NumberDecimal("54.59815003314423820395245961844937") }

Last repro:
Data:

> db.t1.find();
{ "_id" : ObjectId("61d7310c6fce2cdbb8498b8e"), "a" : 1e+34, "b" : NumberDecimal("0.1"), "c" : -1e+34, "d" : NumberDecimal("0.1") }
{ "_id" : ObjectId("61d732ce6fce2cdbb8498b8f"), "a" : 1e+34, "b" : 0.1, "c" : -1e+34, "d" : 0.1 }
{ "_id" : ObjectId("61d734236fce2cdbb8498b90"), "a" : NumberDecimal("1.0E+34"), "b" : NumberDecimal("0.1"), "c" : NumberDecimal("-1.0E+34"), "d" : NumberDecimal("0.1") }

SBE:

> db.t1.find({}, {o: {$add: ["$a", "$b", "$c", "$d"]}});
{ "_id" : ObjectId("61d7310c6fce2cdbb8498b8e"), "o" : NumberDecimal("0.1") }
{ "_id" : ObjectId("61d732ce6fce2cdbb8498b8f"), "o" : 0.2 }
{ "_id" : ObjectId("61d734236fce2cdbb8498b90"), "o" : NumberDecimal("0.1") }

Classic:

> db.t1.find({}, {o: {$add: ["$a", "$b", "$c", "$d"]}});
{ "_id" : ObjectId("61d7310c6fce2cdbb8498b8e"), "o" : NumberDecimal("0.2") }
{ "_id" : ObjectId("61d732ce6fce2cdbb8498b8f"), "o" : 0.2 }
{ "_id" : ObjectId("61d734236fce2cdbb8498b90"), "o" : NumberDecimal("0.1") }

This issue is caused by the difference of how we $add operations in classic engine and SBE.

There are two sources of differences:

  1. The SBE’s builtinDoubleDoubleSum algorithm is slightly different from the classic engine’s in the aspect that the SBE’s computes the expected result type first and then do real addition. If the expected result type is not Decimal128, it does not use DoubleDoubleSummation at all and all input values are converted first to Decimal128 and added to the partial Decimal128 result. But the classic engine’s always uses DoubleDoubleSummation for non-decimal values and add Decimal128 results and DoubleDoubleSummation result.
  2. As in 1., when the SBE’s builtinDoubleDoubleSum converts a double input value to Decimal128 or when the SBE's genericAdd for binary $add converts a double input value to Decimal128, it uses numericCast<Decimal128> and 15 digit rounding and the classic engine 34 digit rounding.
  3. The SBE genericAdd promotes overflowed long addition into Decimal128, which is very reasonable in that it can keep all digits but the SBE's builtinDoubleDoubleSum and the classical engine's ExpressionAdd::evaluate promote it to double.
  4. The $sum accumulator in both the SBE and the classical engine promote overflowed long into a double.

I think we can match the SBE behavior to the classic engine’s behavior.

  1. We always use DoubleDoubleSummation for non-decimal values just like the classic engine. For this, we need to update builtinDoubleDoubleSum() function.
  2. We use 34 digit rounding when converting double input into Decimal128. Otherwise, we may increase error. For this, we need to update numericCast<Decimal128> so that it can support rounding option and genericAdd calls numericCast<Decimal128> with the rounding option.
  3. Always promotes overflowed long to Decimal128 across the board: binary $add, 3 or more arg $add and $sum accumulator.

Update:

  • There was a typo in differences. Fixed it.
  • Added another difference that when the SBE's genericAdd for binary $add converts a double input value to Decimal128, it uses numericCast<Decimal128> and 15 digit rounding
  • Added the 2nd repro.
    1/12/2021
  • Updated differences and the proposed fix.


 Comments   
Comment by Ethan Zhang (Inactive) [ 25/Jan/22 ]

Waiting for Kyle's spec doc for this.

Comment by Yoon Soo Kim [ 10/Jan/22 ]

mihai.andrei, kyle.suarez, What do you think about the proposed fix?

Comment by Yoon Soo Kim [ 10/Jan/22 ]

I expect the proposed fix would fix SERVER-61679 and SERVER-58978’s original BF: BF-21976. Will update this ticket if the proposed fix also fixes the related issues.

Generated at Thu Feb 08 05:55:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.