Jump to content
315234

Loss of precision in calculator

Recommended Posts

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

Share this post


Link to post

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

 

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm going to move this to noted.

Share this post


Link to post

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product. :)

Share this post


Link to post

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product. :)

 

 

Different calculators take different approaches to floating point errors. Python exhibits the issue which Alfred works around by rounding to a certain number of decimal places:

 

 

Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 0.1+0.2
0.30000000000000004
 
Without this rounding, Alfred was giving unpredictable results to people who didn't understand floating point estimations.
 
Ultimately, you could actually use a live script filter workflow to bridge to python to create the calculator you are looking for, and fully disable Alfred's built in calculator.

Share this post


Link to post
×
×
  • Create New...