Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48286

Handle accuracy issues of casting long long to double

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None
    • ALL
    • Query 2020-06-15
    • 0

    Description

      A double on 64-bit architectures uses 1 bit for sign, 52 bits for value, and 11 bits for exponent, thus, the maximum integer value that it can contain without losing precision is 0x1FFFFFFFFFFFFF (9,007,199,254,740,991). This is much smaller than LLONG_MAX (9,223,372,036,854,775,807) and the range long long covers.

      There are instances that we cast LLONG_MAX to double and do a comparison (see here. We should ensure the accuracy of such comparisons in presence of values that cause precision issues for double.

      Attachments

        Issue Links

          Activity

            People

              justin.seyster@mongodb.com Justin Seyster
              amirsaman.memaripour@mongodb.com Amirsaman Memaripour
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: