Bug#617894: [ARMel] tgammal isn't accurate enough
Le 13/03/2011 15:49, Aurelien Jarno a écrit :
On Sat, Mar 12, 2011 at 03:27:31PM +0100, Julien PUYDT wrote:
Isn't that a little short? If I read
http://en.wikipedia.org/wiki/IEEE_754-2008#Basic_formats well, even
simple precision boasts 23 digits.
23 binary digits, which means 7.22 decimal digit. For double, it lists
15.95 digit, so your example is in the expected precision range.
Oh, dear! Indeed, I was counting the base 10 digits and comparing with
base 2 digits... that was a stupid and shameful mistake!
Snark on #sage-devel