Inconsistent rounding in float parser

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: JavaScript
    • None
    • ALL
    • Hide

      > var i = 1.2345
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.234 <- should be a '5'
      > parseFloat(i.toFixed(4))
      1.2345
      > parseFloat(i.toFixed(3))
      1.234 <- should be a '5'
      > var i = 1.234525
      > parseFloat(i.toFixed(5))
      1.23453 <- correct
      > parseFloat(i.toFixed(4))
      1.2345
      > parseFloat(i.toFixed(3))
      1.235 <- correct (mark as 'wtf')
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.235 <- correct
      > var i = 1.234525 (same variable, same value)
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.23
      > parseFloat(i.toFixed(3))
      1.235 <- correct
      > var i = 1.22334455 (same variable, new value)
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.22
      > parseFloat(i.toFixed(3))
      1.223
      > parseFloat(i.toFixed(4))
      1.2233
      > parseFloat(i.toFixed(5))
      1.22334
      > parseFloat(i.toFixed(6))
      1.223345 <- correct
      > parseFloat(i.toFixed(7))
      1.2233445
      > parseFloat(i.toFixed(1))
      1.2
      > parseFloat(i.toFixed(2))
      1.22
      > parseFloat(i.toFixed(3))
      1.223
      > parseFloat(i.toFixed(4))
      1.2233
      > parseFloat(i.toFixed(5))
      1.22334
      > parseFloat(i.toFixed(6))
      1.223345 < -correct
      > parseFloat(i.toFixed(7))
      1.2233445
      > parseFloat(i.toFixed(6))
      1.223345 <-correct
      > db.version()
      2.6.5

      Show
      > var i = 1.2345 > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.234 <- should be a '5' > parseFloat(i.toFixed(4)) 1.2345 > parseFloat(i.toFixed(3)) 1.234 <- should be a '5' > var i = 1.234525 > parseFloat(i.toFixed(5)) 1.23453 <- correct > parseFloat(i.toFixed(4)) 1.2345 > parseFloat(i.toFixed(3)) 1.235 <- correct (mark as 'wtf') > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.235 <- correct > var i = 1.234525 (same variable, same value) > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.23 > parseFloat(i.toFixed(3)) 1.235 <- correct > var i = 1.22334455 (same variable, new value) > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.22 > parseFloat(i.toFixed(3)) 1.223 > parseFloat(i.toFixed(4)) 1.2233 > parseFloat(i.toFixed(5)) 1.22334 > parseFloat(i.toFixed(6)) 1.223345 <- correct > parseFloat(i.toFixed(7)) 1.2233445 > parseFloat(i.toFixed(1)) 1.2 > parseFloat(i.toFixed(2)) 1.22 > parseFloat(i.toFixed(3)) 1.223 > parseFloat(i.toFixed(4)) 1.2233 > parseFloat(i.toFixed(5)) 1.22334 > parseFloat(i.toFixed(6)) 1.223345 < -correct > parseFloat(i.toFixed(7)) 1.2233445 > parseFloat(i.toFixed(6)) 1.223345 <-correct > db.version() 2.6.5
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      I assume the server is handling EcmaScript parsing...

      This is capricious and I am not sure if it is happening in parseFloat or in toFixed()

      Some values are not being properly rounded. And sometimes the same value will be rounded differently on subsequent attemtps (see 'steps to reproduce').

      It appears that repeating the rounding calculation produced an accurate result: first time does not.

            Assignee:
            Ramon Fernandez
            Reporter:
            Jon Gorrono
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: