Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48286

Handle accuracy issues of casting long long to double

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • ALL
    • Query 2020-06-15
    • 0

    Description

      A double on 64-bit architectures uses 1 bit for sign, 52 bits for value, and 11 bits for exponent, thus, the maximum integer value that it can contain without losing precision is 0x1FFFFFFFFFFFFF (9,007,199,254,740,991). This is much smaller than LLONG_MAX (9,223,372,036,854,775,807) and the range long long covers.

      There are instances that we cast LLONG_MAX to double and do a comparison (see here. We should ensure the accuracy of such comparisons in presence of values that cause precision issues for double.

      Attachments

        Activity

          People

            justin.seyster@mongodb.com Justin Seyster
            amirsaman.memaripour@mongodb.com Amirsaman Memaripour
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: