Jump to content

Loss of precision in calculator


315234
 Share

Recommended Posts

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

Link to comment
Share on other sites

Hi!

 

There is a loss of precision with certain operations when using the calculator. For example, the following entry:

 

=1.1294776E-005 / 24.0

 

produces the following result:

 

.000000471

 

...which has only 3 significant figures. It should give something like this (this is from a python interpreter):

 

4.7061566666666667e-07

 

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm going to move this to noted.

Link to comment
Share on other sites

While it's a loss of precision, it's still a correct answer when it comes to how your Mac sees 'floating point' calculations. Alfred has some bounding on scientific notation numbers to prevent OS X from making a complete floating point pigs ear for casual calculations, so this isn't something I'm likely to change in Alfred.

 

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product. :)

Link to comment
Share on other sites

I'm afraid I have to disagree that this is a correct answer. IEEE floating point certainly allows for this level of precision, as demonstrated by the result python gives. A more extreme example:

 

=1.23456789/1000000000

 

gives the answer .000000001 which is clearly wrong. Floating point gives the result 1.23456789e-09 (no loss of precision). So it would seem to me that the output is just formatted wrong. I don't know how the internals work, but maybe just add an option to format all results in scientific format?

 

I know this all probably sounds like nit-picking, but the result is I can't trust Alfred to do quick calculations which is a shame. It is an otherwise really fantastic product. :)

 

 

Different calculators take different approaches to floating point errors. Python exhibits the issue which Alfred works around by rounding to a certain number of decimal places:

 

 

Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 0.1+0.2
0.30000000000000004
 
Without this rounding, Alfred was giving unpredictable results to people who didn't understand floating point estimations.
 
Ultimately, you could actually use a live script filter workflow to bridge to python to create the calculator you are looking for, and fully disable Alfred's built in calculator.
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...