2

Any idea why if typed into spotlight search, the first formula results in 1E-29, while second formula result in 0?

Can a step-by-step explanation be had as to why this bug exists?

It's clear this calculation is performed with floating-point arithmetic, though I believe as there is no "floating-point" per-se in this calculation, the inequality is a bug, and not a result of floating-point arithmetic limitations. My intuition tells me this could be done with only integer arithmetic as there is no negative value and no decimal point.

To be clear, I don't rely on my intuition, and I'm happy to be told that it is wrong, if that is indeed the case. I'm just interested in the specific anomaly that leads to this inaccuracy.

(1024*64) - (2^16) = 1E-29

(1024*128) - (2^17) = 0

1E-29 0

  • Did you check this on your calculator? – Solar Mike Feb 9 at 23:35
  • calculator works properly – goofology Feb 10 at 3:53
2

Clearly it’s a bug if you believe the exact math should match. It’s probably also a feature since the math library chosen was likely chosen to be fast and small for the math it needs to do and not scientific accuracy for calculating a moon landing.

This sort of rounding errors happens all the time, especially if you’re sloppy or don’t implement a guard digit for “small numbers” like 65k. All math libraries make approximations when they encode values, so the bits sometimes matter when you are checking for something that’s very close to zero. In your example the two large numbers aren’t perfectly represented and you get a very insignificant rounding error that doesn’t cancel perfectly when the subtraction happens in one case but not the other. The 1e-29 should be rounded to zero IMO unless the goal is to introduce people to this concept.

When you need to be precise, use PCalc or another calculator that handles precision and rounding properly.

Also, if you take 1E-29 of the weight of the earth, that’s just under 60 mg. If you were estimating the mass of the earth, you’d be of by more than a snowflake and less than a grain of sand. This quite literally is the proverbial “making a mountain out of a mole hill”.

That being said, if you like math or computers, this is very cool, since it exposes the internal representation of a number and you see how an approximation can cause rounding errors for not that very large numbers in this case. There are legendary (to engineers and programmers at least) books devoted to numerical recipes and tricks to do quick and accurate calculations when you don’t bring along a perfect math library.

  • 1
    +1 for PCalc recommendation – dwightk Feb 9 at 17:34
  • I'm vaguely familiar with floating point rounding errors/limitations, but I assumed that it doesn't apply in this situation because 1 - this is not a floating point calculation, and 2 - the larger value/calculation results in zero correctly, there must be something else going on with the smaller value/calculation. I'll read up a bit. – goofology Feb 10 at 3:56
  • It is not a bug, just the way floating point works. stackoverflow.com/tags/floating-point/info – Mark Feb 12 at 12:54
  • Although I suppose it is a bug that this could be calculated exactly but when do you switch from floating point. I suspect using the exponation operator is the reason – Mark Feb 12 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .