Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48286

Handle accuracy issues of casting long long to double

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL
    • Sprint:
      Query 2020-06-15
    • Linked BF Score:
      18

      Description

      A double on 64-bit architectures uses 1 bit for sign, 52 bits for value, and 11 bits for exponent, thus, the maximum integer value that it can contain without losing precision is 0x1FFFFFFFFFFFFF (9,007,199,254,740,991). This is much smaller than LLONG_MAX (9,223,372,036,854,775,807) and the range long long covers.

      There are instances that we cast LLONG_MAX to double and do a comparison (see here. We should ensure the accuracy of such comparisons in presence of values that cause precision issues for double.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              justin.seyster Justin Seyster
              Reporter:
              amirsaman.memaripour Amirsaman Memaripour
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: