[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#617894: [ARMel] tgammal isn't accurate enough

Le 13/03/2011 15:49, Aurelien Jarno a écrit :
On Sat, Mar 12, 2011 at 03:27:31PM +0100, Julien PUYDT wrote:
Isn't that a little short? If I read
http://en.wikipedia.org/wiki/IEEE_754-2008#Basic_formats well, even
simple precision boasts 23 digits.

23 binary digits, which means 7.22 decimal digit. For double, it lists
15.95 digit, so your example is in the expected precision range.

Oh, dear! Indeed, I was counting the base 10 digits and comparing with base 2 digits... that was a stupid and shameful mistake!

Snark on #sage-devel

Reply to: