Loss of precision in calculator

[315234]

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

[Tyler Eich]

I can confirm this. Command line program 'bc' also loses precision when working with very small numbers.

[Andrew]

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

 

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm going to move this to noted.

[315234]

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product.

[Andrew]

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product.

 

 

Different calculators take different approaches to floating point errors. Python exhibits the issue which Alfred works around by rounding to a certain number of decimal places:

 

 

Python 2.7.2 (default, Oct 11 2012, 20:14:37) 

[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> 0.1+0.2

0.30000000000000004

 

Without this rounding, Alfred was giving unpredictable results to people who didn't understand floating point estimations.

 

Ultimately, you could actually use a live script filter workflow to bridge to python to create the calculator you are looking for, and fully disable Alfred's built in calculator.