Bug#810322: libc6:amd64: Unexpected asseration in pthread_mutex_unlock()

[Andreas Messer <andi@bastelmap.de>]

Package: libc6
Version: 2.19-18+deb8u1
Severity: normal

Dear Maintainer,

when starting the iked from ike package the folloowing assertion occurs:

iked: pthread_mutex_unlock.c:87: __pthread_mutex_unlock_usercnt: Assertion `type == PTHREAD_MUTEX_ERRORCHECK_NP' failed.

I have analyzed the problem and figured out that I can reproduce the problem with
the following piece of code:

#include <pthread.h>
#include <time.h>

int main(int argc, char* argv[])
{
  pthread_mutex_t mutex;
  pthread_mutexattr_t attr;
  struct timespec ts;

  pthread_mutexattr_init( &attr );
  pthread_mutexattr_settype( &attr, PTHREAD_MUTEX_ERRORCHECK );
  pthread_mutex_init( &mutex, &attr );

  clock_gettime( CLOCK_REALTIME, &ts );
  ts.tv_sec += 3;
  pthread_mutex_timedlock( &mutex, &ts );
  pthread_mutex_unlock(&mutex);
  return 0;
}

The problem does not occur with any other type of mutex.
Looking into the libc6 source code i figured out that the following happens:

1) pthread_mutex_timedlock() sets the PTHREAD_MUTEX_ELISION_NP flag through 
   FORCE_ELISION macro, which changes mutex->__data.__kind from 0x2 to 0x102
   (pthread_mutex_timedlock.c:93)

2) pthread_mutex_unlock() asserts the mutex->__data.__kind
   to be 0x2 (pthread_mutex_unlock.c:87)

In my opinion the assertation in pthread_mutex_unlock in line 87 is wrong. 
It is:

  assert (type == PTHREAD_MUTEX_ERRORCHECK_NP);

but should be:

  assert (PTHREAD_MUTEX_TYPE(type) == PTHREAD_MUTEX_ERRORCHECK_NP);

as it is in the lines above also.

This problem also causes https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756316
As workaround, the mutex type PTHREAD_MUTEX_ERRORCHECK can be replaced by
PTHREAD_MUTEX_NORMAL in applications at the cost of less application
error checking.

-- System Information:
Debian Release: 8.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.3.3-proton (SMP w/8 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages libc6:amd64 depends on:
ii  libgcc1  1:4.9.2-10

libc6:amd64 recommends no packages.

Versions of packages libc6:amd64 suggests:
ii  debconf [debconf-2.0]  1.5.56
pn  glibc-doc              <none>
ii  locales                2.19-18+deb8u1

-- debconf information excluded