conversion from double to decimal rounds 0.999999999999999 incorrectly

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • 3.4.0-rc0
    • Affects Version/s: None
    • Component/s: Shell
    • None
    • Fully Compatible
    • ALL
    • Hide

      > x=NumberDecimal(0.999999999999999)
      NumberDecimal("1.00000000000000")

      Show
      > x=NumberDecimal(0.999999999999999) NumberDecimal("1.00000000000000")
    • Integration 2016-10-10
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When determining the number of digits to round to, the existing code first uses a possible overestimate of the decimal place to round at, and corrects it if the final number of digits is not 15. However, this is incorrect when the rounding causes the decade to change: 1.00000000000000 has the same number of digits as 0.999999999999999. So, first try rounding at the smaller decimal place instead.

              Assignee:
              Geert Bosch
              Reporter:
              Geert Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: