Bug#746310: libc6: strcmp returns false equality between two strings

[Sebastien Bernard <seb@frankengul.org>]

Package: libc6
Version: 2.18-4
Severity: normal

Dear Maintainer,

While debugging a problem building the debian-installer,
we came across a case where strcmp reports 2 strings equality whereas they are not.
It's the comparaison between 2 translated entries '.' and '..' 
that are translated to the string \0000 and \0001 in the genisoimage program.
This program was complaining that '.' and '..' both translated to '' and thus cannot generate iso.

This was obviously wrong, so I narrowed the check to this case:
----------
cat  test_strcmp.c<<EOF
#include <stdio.h>
#include <stdlib.h>

void main(void) {
  char stra[2];
  char strb[2];
  int result = 0;

  stra[0] = 0;
  stra[1] = 0;
  strb[0] = 1;
  strb[1] = 0;

  result = strcmp(stra,strb);

  printf("result from strcmp('\\0000','\\0001' is %d)\n",result);
  exit(0);
}
EOF 

Building this program on sparc, 32bit or 64bit, optimized or not, the result is 0.
Patrick Bagget did some more investigation and found that any string composed of a list of 1 followed by a 0 is miscompared.
Here's an example :

{ 0, 0 }  //i.e. original test case. Fails, returns 0
{ 1, 0 } 

{ 1, 0, 0 } //fails, returns 0.
{ 1, 1, 0 }

{ 1, 1, 0, 0 } //fails, returns 0.
{ 1, 1, 1, 0 }

{ 1, 1, 1, 0, 0 } //fails, returns 0.
{ 1, 1, 1, 1, 0 }

{ 1, 1, 1, 1, 0, 0 } //fails, returns 0.
{ 1, 1, 1, 1, 1, 0 }A

There's something wrong in the glibc implementation of strcmp for the case of sparc, and sparc64.

-- System Information:
Debian Release: jessie/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: sparc (sparc64)

Kernel: Linux 3.13-1-sparc64-smp (SMP w/1 CPU core)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages libc6 depends on:
ii  libgcc1  1:4.8.2-16

libc6 recommends no packages.

Versions of packages libc6 suggests:
ii  debconf [debconf-2.0]  1.5.52
pn  glibc-doc              <none>
ii  locales                2.18-4

-- debconf information:
  glibc/restart-failed:
* libraries/restart-without-asking: true
  glibc/disable-screensaver:
  glibc/upgrade: true
  glibc/restart-services:

#include <stdio.h>
#include <stdlib.h>

void main(void) {
  unsigned char stra[2];
  unsigned char strb[2];
  int result = 0;

  stra[0] = 0;
  stra[1] = 0;
  strb[0] = 1;
  strb[1] = 0;

  result = strcmp(stra,strb);
  printf("result from strcmp('\\0000','\\0001' is %d)\n",result);
  result = memcmp(stra,strb,2);
  printf("result from memcmp('\\0000','\\0001' is %d)\n",result);
  exit(0);
}