Bug#860823: gcc-6: sqrt x64 intrinsic broken with -O3 and -ffast-math

[Catalin Ionescu <catalin.ionescu.ro@gmail.com>]

Package: gcc-6
Version: 6.3.0-12
Severity: normal

After being "hit" by unexplained division by 0 related crashes of an application, the
following simple test code shows the issue:

****
#include <cmath>
#include <cstdlib>

void TestSqrt(const float *Src,float *Dest,size_t Size,float &Gain)
{
  while (Size--)
  {
    float G,X;

    G=sqrt(X=*Src++);
    if (G)
    {
        Gain=Gain*0.9f+1/G*0.1f;
        X*=Gain;
    }
    *Dest++=X;
  }
}
****

The compile and listing generating where done using:

****
gcc -O3 -m64 -ffast-math -c -o sqrt.o sqrt.cpp
objdump -M intel -S -r -C sqrt.o > sqrt.lst
****

The "sqrt" coresponding piece of code looks like:

****
  30:	f3 0f 10 0c 87       	movss  xmm1,DWORD PTR [rdi+rax*4]
  35:	f3 0f 52 d1          	rsqrtss xmm2,xmm1
  39:	0f 28 c2             	movaps xmm0,xmm2
  3c:	0f 2f cb             	comiss xmm1,xmm3
  3f:	f3 0f 59 c1          	mulss  xmm0,xmm1
  43:	f3 0f 59 c2          	mulss  xmm0,xmm2
  47:	f3 0f 59 d4          	mulss  xmm2,xmm4
  4b:	f3 0f 58 c5          	addss  xmm0,xmm5
  4f:	f3 0f 59 c2          	mulss  xmm0,xmm2
  53:	74 18                	je     6d <TestSqrt(float const*, float*, unsigned long, float&)+0x6d>
  55:	f3 0f 10 11          	movss  xmm2,DWORD PTR [rcx]
  59:	f3 0f 59 c6          	mulss  xmm0,xmm6
  5d:	f3 0f 59 d7          	mulss  xmm2,xmm7
  61:	f3 0f 58 c2          	addss  xmm0,xmm2
  65:	f3 0f 59 c8          	mulss  xmm1,xmm0
  69:	f3 0f 11 01          	movss  DWORD PTR [rcx],xmm0
  6d:	f3 0f 11 0c 86       	movss  DWORD PTR [rsi+rax*4],xmm1
  72:	48 83 c0 01          	add    rax,0x1
  76:	48 39 c2             	cmp    rdx,rax
  79:	75 b5                	jne    30 <TestSqrt(float const*, float*, unsigned long, float&)+0x30>
****

The generated code is not handling in any way the case of 0 as input value. As per Intel spec,
rsqrtss has an +/-INF result when the input is 0 and the final results of the test code is
unpredictable, while 0 is a perfect valid input value for sqrt.

Regards,
Catalin


-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages gcc-6 depends on:
ii  binutils      2.28-3
ii  cpp-6         6.3.0-12
ii  gcc-6-base    6.3.0-12
ii  libc6         2.24-9
ii  libcc1-0      6.3.0-12
ii  libgcc-6-dev  6.3.0-12
ii  libgcc1       1:6.3.0-12
ii  libgmp10      2:6.1.2+dfsg-1
ii  libisl15      0.18-1
ii  libmpc3       1.0.3-1+b2
ii  libmpfr4      3.1.5-1
ii  libstdc++6    6.3.0-12
ii  zlib1g        1:1.2.8.dfsg-5

Versions of packages gcc-6 recommends:
ii  libc6-dev  2.24-9

Versions of packages gcc-6 suggests:
pn  gcc-6-doc         <none>
pn  gcc-6-locales     <none>
pn  gcc-6-multilib    <none>
pn  libasan3-dbg      <none>
pn  libatomic1-dbg    <none>
pn  libcilkrts5-dbg   <none>
pn  libgcc1-dbg       <none>
pn  libgomp1-dbg      <none>
pn  libitm1-dbg       <none>
pn  liblsan0-dbg      <none>
pn  libmpx2-dbg       <none>
pn  libquadmath0-dbg  <none>
pn  libtsan0-dbg      <none>
pn  libubsan0-dbg     <none>

-- no debconf information