Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-91788

FLE2 range floating point insertion may cause invariant failure if using precision mode

    • Server Security
    • ALL
    • v8.0
    • Security 2024-07-08

      I have a FLE2 range field of type "double", with min = 0.0 , max = 2^53 , and precision = 0 , I expected that the ceil( log2((max - min + 1) * 10^prc )) calculation for determining precision mode will give me a value of 54 , however the result I get is 53 . I checked the floating-point arithmetic, and it was because (max - min + 1) = 2^53 + 1 and 2^53 + 1 is not exactly representable as a double value (not enough mantissa bits), so the actual value of (max - min + 1) is just 2^53.

      If I try to insert the value 2^53 for this field, that will run the getTypeInfoDouble function with a value 2^53 (i.e. the max value), and the code crashes due to this invariant. It's because the new max value is 2^53 - 1 instead of 2^54 - 1.

            Assignee:
            shreyas.kalyan@mongodb.com Shreyas Kalyan
            Reporter:
            erwin.pe@mongodb.com Erwin Pe
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: