We're doing a fuzzy <= comparison here before we subtract two values (<= operator implementation is here). If they are fuzzily but not exactly equal, subtraction can produce a negative value (depending on which is greater) and triggers the tassert in assertValid() in the subtraction operator. Use < instead of <= to avoid this issue.
The same issue exists in `computeAndSetNodeCost()` in cost_estimator.cpp, where using std::min results in use of the same fuzzy comparators. The toDouble() values should be used instead of comparing the CardinalityEstimate objects here to ensure that an exact comparison is used.
Both of these fixes are necessary to resolve AF-16798.
- related to
-
SERVER-125575 Inspect usage of the Estimates library and make it more resilient
-
- Closed
-
-
SERVER-125558 Use strict comparison to choose final CE result in estimate(OIL) in cardinality estimator
-
- Closed
-
-
SERVER-125525 Inspect usage of the Estimates library and make it more resilient
-
- In Progress
-