Learn More
Resend Email
11
GiddyNaya
6y

I have been debugging for like hours trying to figure out the cause an unknown bug spoiling my UI by making my elements overlap.
I'm working on a Unit Converter that takes kWh and then converts to mWh. (Logical Conversion: 1000 kWh = 1 mWh).
Just an easy shit i thought, using Javascript I just passed the dynamically generated kWh value to a function that takes maximum of 6 chars and multiply it by 0.001 to get the required result but this was where my problem started. All values came out as expected until my App hits a particular value (8575) and outputs a very long set of String (8.575000000000001), i couldn't figure the cause of this until i checked my console log and found the culprit value, and then i change the calculation logic from multiply by 0.001 to divide by 1000 and it came out as expected (8.575)

My question is that;
Is my math logically wrong or is this another Javascript Calculation setback?

question

javascript

calculation logic
Favorite
Ranter
Comments
  • 3
    6y
    I don't think the math is the problem, I found a similar issue in Python too (something like 50/2 being calculated to 24.9999999999999999). I think it has to do with how floats are stored in memory. Like, 0.1 is not exactly stored as 0.1 and things like that. You can play around with it here: https://h-schmidt.net/FloatConverte...
  • 3
    6y
    Math checks out. It’s gotta be JavaScript.
  • 6
    6y
    No. This is an issue with floating point numbers in computers which are difficult since it's all 0s and 1s. See:

    https://devrant.com/rants/1392187/...

    If you want the most precise calculation then use the base unit. (Or since this is engineering, just use .toFixed() :v)
  • 1
    6y
    This is most definitely a float issue. I think your code is treating the float as a double, therefore giving you a double precision result
  • 4
    6y
    You should give a read to:

    What Every Programmer Should Know About Floating Point Arithmetic
  • 1
    6y
    @byogdc This is Javascript though and in Javascript the only kinds of numbers there are are 64 bit IEEE754 floating point numbers
  • 0
    6y
    @inaba well that's dumb.
  • 1
    6y
    You know how in decimal you have infinite periods of numbers for certain numbers? 1/3 becomes 0.333...

    The same happens in binary (albeit for different numbers). You don't have infinite bits, so only an approximation gets stored in memory in any language. All that you can control is the level of precision.
  • 2
    6y
    By the way, you get the same effect when dividing by 1000.0 instead of 1000. The '.0' makes it a floating point type.
  • 1
    6y
    You actually meant MWh ("Mega") instead of mWh ("Milli"), right? ;)
    1000 kWh = 1 MWh, and
    1000000 mWh = 1000 Wh = 1 kWh
  • 1
    6y
    Do your homework, learn about IEEE754, specs for floating point arthimetic.
  • 0
    6y
    @Lisk ia right
    I was just about to say it myself.

    1Mega -> 10e6 ->1000kilo [M]
    1kilo -> 10e3 -> 1000 [k]
    1decy -> 10e-1 -> 0.1 [d]
    1centy -> 10e-2 -> 0.01 [c]
    1milli -> 10e-3 -> 0.001 [m]

    So 1kWh -> 1000000mWh
  • 0
    6y
    In binary you can't represent 0.1 exactly, just like in decimal you can't represent 1/3 or 1/7 exactly.

    If you need decimal precision, use decimal arithmetic.

    If you need arbitrary precision, use fractions (pairs of numerator, denomination integers).
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment