Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16007

Inconsistent rounding in float parser

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: JavaScript
    • Labels:
      None
    • ALL
    • Hide

      > var i = 1.2345
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.234 <- should be a '5'
      > parseFloat(i.toFixed(4))
      1.2345
      > parseFloat(i.toFixed(3))
      1.234 <- should be a '5'
      > var i = 1.234525
      > parseFloat(i.toFixed(5))
      1.23453 <- correct
      > parseFloat(i.toFixed(4))
      1.2345
      > parseFloat(i.toFixed(3))
      1.235 <- correct (mark as 'wtf')
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.235 <- correct
      > var i = 1.234525 (same variable, same value)
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.235 <- correct
      > var i = 1.22334455 (same variable, new value)
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.22
      > parseFloat(i.toFixed(3))
      1.223
      > parseFloat(i.toFixed(4))
      1.2233
      > parseFloat(i.toFixed(5))
      1.22334
      > parseFloat(i.toFixed(6))
      1.223345 <- correct
      > parseFloat(i.toFixed(7))
      1.2233445
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.22
      > parseFloat(i.toFixed(3))
      1.223
      > parseFloat(i.toFixed(4))
      1.2233
      > parseFloat(i.toFixed(5))
      1.22334
      > parseFloat(i.toFixed(6))
      1.223345 < -correct
      > parseFloat(i.toFixed(7))
      1.2233445
      > parseFloat(i.toFixed(6))
      1.223345 <-correct
      > db.version()
      2.6.5

      Show
      > var i = 1.2345 > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.234 <- should be a '5' > parseFloat(i.toFixed(4)) 1.2345 > parseFloat(i.toFixed(3)) 1.234 <- should be a '5' > var i = 1.234525 > parseFloat(i.toFixed(5)) 1.23453 <- correct > parseFloat(i.toFixed(4)) 1.2345 > parseFloat(i.toFixed(3)) 1.235 <- correct (mark as 'wtf') > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.235 <- correct > var i = 1.234525 (same variable, same value) > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.235 <- correct > var i = 1.22334455 (same variable, new value) > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.22 > parseFloat(i.toFixed(3)) 1.223 > parseFloat(i.toFixed(4)) 1.2233 > parseFloat(i.toFixed(5)) 1.22334 > parseFloat(i.toFixed(6)) 1.223345 <- correct > parseFloat(i.toFixed(7)) 1.2233445 > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.22 > parseFloat(i.toFixed(3)) 1.223 > parseFloat(i.toFixed(4)) 1.2233 > parseFloat(i.toFixed(5)) 1.22334 > parseFloat(i.toFixed(6)) 1.223345 < -correct > parseFloat(i.toFixed(7)) 1.2233445 > parseFloat(i.toFixed(6)) 1.223345 <-correct > db.version() 2.6.5

      I assume the server is handling EcmaScript parsing...

      This is capricious and I am not sure if it is happening in parseFloat or in toFixed()

      Some values are not being properly rounded. And sometimes the same value will be rounded differently on subsequent attemtps (see 'steps to reproduce').

      It appears that repeating the rounding calculation produced an accurate result: first time does not.

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            ccjon Jon Gorrono
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: