Bug#175907: xlibs: [libX11] Patch clarifies error "socket() failed" for exceeding OPEN_MAX

[Matthew Astley <mca-debbts@grantadesign.com>]

Package: xlibs
Version: 4.2.1-4
Severity: normal
Tags: upstream patch

When using Java's AWT after the JVM has opened many (~250) files,
libX11 XOpenDisplay() fails with the following report:

  _X11TransSocketOpen: socket() failed for tcp
  _X11TransSocketOpenCOTSClient: Unable to open socket for tcp
  _X11TransOpen: transport open failed for tcp/meek:0

I found this rather unhelpful, since strace(1) indicated that socket()
returned a valid fd.  From mailing list queries I've seen, I believe
several other people have had this problem and not figured out why.

Therefore I patched the TRANS(SocketOpen)(int,int) function, as below,
to return a more informative error:

  _X11TransSocketOpen: socket() failed for tcp [fd=258, Exceeded TRANS_OPEN_MAX]
  _X11TransSocketOpenCOTSClient: Unable to open socket for tcp
  _X11TransOpen: transport open failed for tcp/meek:0

I believe this will apply also to older versions of Xlib.

I have not attempted to change the definition of TRANS_OPEN_MAX to
allow more fds because I do not understand the consequences of this,
and I can see that it is a complicated matter.

The Java code in question cheerfully opens over 400 files, and kludges
are required to ensure that this is done _after_ the AWT
initialisation.  I would quite like to avoid this kludge, and would
appreciate suggestions for modifying the TRANS_OPEN_MAX definition in
xc/lib/xtrans/Xtransint.h .

I've included also a small test program.  You will see from this that
my code should be checked for rust... in particular I'm concerned that
strerror isn't thread safe, but there are problems with strerror_r so
I avoided that.  I can't think of an easy way of provoking the
strerror branch in this, so that hasn't been tested.


I hope this is useful,

Matthew  #8-)


*** /tmp/Xtranssock.c.diff
--- xc/lib/xtrans/Xtranssock.c.orig	2001-12-14 19:57:06.000000000 +0000
+++ xc/lib/xtrans/Xtranssock.c	2003-01-08 20:45:38.000000000 +0000
@@ -397,8 +397,11 @@
 #endif
 #endif
       ) {
-	PRMSG (1, "SocketOpen: socket() failed for %s\n",
-	    Sockettrans2devtab[i].transname, 0, 0);
+	/* If socket() succeeded, we've just leaked an fd... */
+	PRMSG (1, "SocketOpen: socket() failed for %s [fd=%d, %s]\n",
+	    Sockettrans2devtab[i].transname,
+	    ciptr->fd,
+	    (ciptr->fd < 0 ? strerror(errno) : "Exceeded TRANS_OPEN_MAX"));
 
 	xfree ((char *) ciptr);
 	return NULL;



*** x11-maxopen.c
// gcc -o x11-maxopen{,.c} -L/usr/X11R6/lib -lX11 && ./x11-maxopen foo:0
#include <stdio.h>
#include <X11/Xlib.h>
#include <string.h>
#include <errno.h>
#include <netinet/in.h>

int main(int argc, char **argv) {
  Display *x11_disp;
  int i, fd;
  if (argc != 2) {
    fprintf(stderr, "Syntax: %s <display_name>\n", argv[0]);
    return 1;
  }

  for (i=0; i < 255; i++) {
    fd = socket(PF_INET, SOCK_STREAM, IPPROTO_IP);
    if (fd > 0) continue;
    fprintf(stderr, "Failed to allocate pointless socket #%d: %s\n", i, strerror(errno));
    return 1;
  }
  printf("Last allocated fd for pointless socket: %d\n", fd);

  x11_disp = XOpenDisplay( argv[1] );
  printf("x11_disp = 0x%X\n", (int)x11_disp);
}



-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux troop.granta.internal 2.4.5 #1 Wed Mar 6 19:11:51 GMT 2002 i686
Locale: LANG=en_GB.ISO-8859-1, LC_CTYPE=en_GB.ISO-8859-1

Versions of packages xlibs depends on:
ii  libc6                         2.3.1-5    GNU C Library: Shared libraries an
ii  libfreetype6                  2.1.3-4    FreeType 2 font engine, shared lib
ii  xfree86-common                4.2.1-1    X Window System (XFree86) infrastr

-- no debconf information