315234 Posted April 11, 2013 Share Posted April 11, 2013 Hi! There is a loss of precision with certain operations when using the calculator. For example, the following entry: =1.1294776E-005 / 24.0 produces the following result: .000000471 ...which has only 3 significant figures. It should give something like this (this is from a python interpreter): 4.7061566666666667e-07 Link to comment Share on other sites More sharing options...
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!Register a new account
Already have an account? Sign in here.Sign In Now