Bug#901883: xwayland: Firefox causes Xwayland to freeze and eat 100% CPU

[tino.mettler+debbugs@tikei.de]

Package: xwayland
Version: 2:1.20.0-2
Severity: normal
Tags: patch

Dear Maintainer,

while trying to watch the German TV show "Die Sendung mit der Maus" from
https://www.wdrmaus.de/aktuelle-sendung/, my Gnome desktop freezes.  I can
SSH into the machine and see that Xwayland eats 100% CPU.  The freeze can be
reproduced, it happened every time I accessed the above URL in Firefox 60.

I found a similar issue in the Red Hat bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1579067

The issue mentions patches in
https://patchwork.freedesktop.org/series/43618/.  I applied them to the
xorg-server source, built and installed the package and the freeze does not
happen anymore.

The patches are also attached to this bugreport.

Regards,
Tino

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.17.2 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xwayland depends on:
ii  libaudit1           1:2.8.3-1
ii  libbsd0             0.9.1-1
ii  libc6               2.27-3
ii  libdrm2             2.4.92-1
ii  libegl1             1.0.0+git20180308-3
ii  libepoxy0           1.4.3-1
ii  libgbm1             18.1.1-1
ii  libgcrypt20         1.8.3-1
ii  libgl1              1.0.0+git20180308-3
ii  libpixman-1-0       0.34.0-2
ii  libselinux1         2.8-1
ii  libsystemd0         238-5
ii  libwayland-client0  1.15.0-2
ii  libxau6             1:1.0.8-1+b2
ii  libxdmcp6           1:1.1.2-3
ii  libxfont2           1:2.0.3-1
ii  libxshmfence1       1.3-1
ii  xserver-common      2:1.20.0-2

xwayland recommends no packages.

xwayland suggests no packages.

-- debconf-show failed

diff --git a/glamor/glamor.c b/glamor/glamor.c
index d984d20f3..e2c74d17a 100644
--- a/glamor/glamor.c
+++ b/glamor/glamor.c
@@ -836,20 +836,20 @@ glamor_fds_from_pixmap(ScreenPtr screen, PixmapPtr pixmap, int *fds,
         glamor_get_screen_private(pixmap->drawable.pScreen);
 
     if (!glamor_priv->dri3_enabled)
-        return -1;
+        return 0;
     switch (pixmap_priv->type) {
     case GLAMOR_TEXTURE_DRM:
     case GLAMOR_TEXTURE_ONLY:
         if (!glamor_pixmap_ensure_fbo(pixmap, pixmap->drawable.depth == 30 ?
                                       GL_RGB10_A2 : GL_RGBA, 0))
-            return -1;
+            return 0;
         return glamor_egl_fds_from_pixmap(screen, pixmap, fds,
                                           strides, offsets,
                                           modifier);
     default:
         break;
     }
-    return -1;
+    return 0;
 }
 
 _X_EXPORT int

diff --git a/glamor/glamor.c b/glamor/glamor.c
index e2c74d17a..63f0947fa 100644
--- a/glamor/glamor.c
+++ b/glamor/glamor.c
@@ -865,17 +865,15 @@ glamor_fd_from_pixmap(ScreenPtr screen,
                                  &modifier);
 
     /* Pixmaps with multi-planes/modifier are not supported in this interface */
-    if (ret > 1) {
-        while (ret > 0)
-            close(fds[--ret]);
-        return -1;
+    if (ret == 1 && offsets[0] == 0) {
+        *stride = strides[0];
+        *size = pixmap->drawable.height * *stride;
+        return fds[0];
     }
 
-    ret = fds[0];
-    *stride = strides[0];
-    *size = pixmap->drawable.height * *stride;
-
-    return ret;
+    while (ret > 0)
+        close(fds[--ret]);
+    return -1;
 }
 
 _X_EXPORT int