gcc-10: regression in 10.2.0-9 causes segmentation fault in vlc
Control: reassign -1 gcc-10 10.2.0-9
Control: retitle -1 gcc-10: regression in 10.2.0-9 causes segmentation fault in vlc
Control: affects -1 vlc
Control: severity -1 serious
Hi,
the vlc crash is caused by a compiler regression introduced in gcc-10 version 10.2.0-9, which was used to compile vlc 3.0.11.1-2.
With the buggy compiler, the function picture_Clone from src/misc/picture.c gets miscompiled, in particular the following loop:
https://salsa.debian.org/multimedia-team/vlc/-/blob/1d01d5e56132e9917c53c5a97f10d42426081512/src/misc/picture.c#L401-405
for (int i = 0; i < picture->i_planes; i++) {
res.p[i].p_pixels = picture->p[i].p_pixels;
res.p[i].i_lines = picture->p[i].i_lines;
res.p[i].i_pitch = picture->p[i].i_pitch;
}
A correctly working gcc simply unrolls the loop:
$ objdump --disassemble=picture_Clone /usr/lib/x86_64-linux-gnu/libvlccore.so.9.0.0 | head -n 62 | tail -n 30
a8c04: 85 c9 test %ecx,%ecx
a8c06: 0f 8e 8c 00 00 00 jle a8c98 <picture_Clone@@Base+0xe8>
a8c0c: 48 8b b7 b0 00 00 00 mov 0xb0(%rdi),%rsi
a8c13: 48 8b bf b8 00 00 00 mov 0xb8(%rdi),%rdi
a8c1a: 48 89 74 24 10 mov %rsi,0x10(%rsp)
a8c1f: 48 89 7c 24 18 mov %rdi,0x18(%rsp)
a8c24: 83 f9 01 cmp $0x1,%ecx
a8c27: 74 6f je a8c98 <picture_Clone@@Base+0xe8>
a8c29: 4c 8b 83 d0 00 00 00 mov 0xd0(%rbx),%r8
a8c30: 4c 8b 8b d8 00 00 00 mov 0xd8(%rbx),%r9
a8c37: 4c 89 44 24 20 mov %r8,0x20(%rsp)
a8c3c: 4c 89 4c 24 28 mov %r9,0x28(%rsp)
a8c41: 83 f9 02 cmp $0x2,%ecx
a8c44: 74 52 je a8c98 <picture_Clone@@Base+0xe8>
a8c46: 4c 8b 93 f0 00 00 00 mov 0xf0(%rbx),%r10
a8c4d: 4c 8b 9b f8 00 00 00 mov 0xf8(%rbx),%r11
a8c54: 4c 89 54 24 30 mov %r10,0x30(%rsp)
a8c59: 4c 89 5c 24 38 mov %r11,0x38(%rsp)
a8c5e: 83 f9 03 cmp $0x3,%ecx
a8c61: 74 35 je a8c98 <picture_Clone@@Base+0xe8>
a8c63: 4c 8b a3 10 01 00 00 mov 0x110(%rbx),%r12
a8c6a: 48 8b 83 18 01 00 00 mov 0x118(%rbx),%rax
a8c71: 4c 89 64 24 40 mov %r12,0x40(%rsp)
a8c76: 48 89 44 24 48 mov %rax,0x48(%rsp)
a8c7b: 83 f9 04 cmp $0x4,%ecx
a8c7e: 74 18 je a8c98 <picture_Clone@@Base+0xe8>
a8c80: 48 8b 93 30 01 00 00 mov 0x130(%rbx),%rdx
a8c87: 48 8b 8b 38 01 00 00 mov 0x138(%rbx),%rcx
a8c8e: 48 89 54 24 50 mov %rdx,0x50(%rsp)
a8c93: 48 89 4c 24 58 mov %rcx,0x58(%rsp)
The buggy gcc does something strange instead:
$ objdump --disassemble=picture_Clone /usr/lib/x86_64-linux-gnu/libvlccore.so.9.0.0 | head -n 80 | tail -n 48
a8c04: 85 f6 test %esi,%esi
a8c06: 0f 8e ca 00 00 00 jle a8cd6 <picture_Clone@@Base+0x126>
a8c0c: 8d 46 ff lea -0x1(%rsi),%eax
a8c0f: 83 f8 01 cmp $0x1,%eax
a8c12: 0f 86 18 01 00 00 jbe a8d30 <picture_Clone@@Base+0x180>
a8c18: 48 8b bf b8 00 00 00 mov 0xb8(%rdi),%rdi
a8c1f: 48 8b 8b b0 00 00 00 mov 0xb0(%rbx),%rcx
a8c26: 41 89 c1 mov %eax,%r9d
a8c29: 4c 8b 83 d8 00 00 00 mov 0xd8(%rbx),%r8
a8c30: 41 d1 e9 shr %r9d
a8c33: 48 89 4c 24 10 mov %rcx,0x10(%rsp)
a8c38: 48 89 7c 24 20 mov %rdi,0x20(%rsp)
a8c3d: 48 89 7c 24 18 mov %rdi,0x18(%rsp)
a8c42: 4c 89 44 24 28 mov %r8,0x28(%rsp)
a8c47: 41 83 f9 01 cmp $0x1,%r9d
a8c4b: 74 30 je a8c7d <picture_Clone@@Base+0xcd>
a8c4d: 4c 8b 93 c8 00 00 00 mov 0xc8(%rbx),%r10
a8c54: 4c 8b 9b c0 00 00 00 mov 0xc0(%rbx),%r11
a8c5b: 4c 8b a3 18 01 00 00 mov 0x118(%rbx),%r12
a8c62: 48 8b 93 f8 00 00 00 mov 0xf8(%rbx),%rdx
a8c69: 4c 89 5c 24 30 mov %r11,0x30(%rsp)
a8c6e: 4c 89 54 24 40 mov %r10,0x40(%rsp)
a8c73: 48 89 54 24 38 mov %rdx,0x38(%rsp)
a8c78: 4c 89 64 24 48 mov %r12,0x48(%rsp)
a8c7d: 83 e0 fe and $0xfffffffe,%eax
a8c80: 4c 63 c0 movslq %eax,%r8
a8c83: 83 c0 01 add $0x1,%eax
a8c86: 49 8d 48 01 lea 0x1(%r8),%rcx
a8c8a: 49 c1 e0 05 shl $0x5,%r8
a8c8e: 4a 8b bc 03 b0 00 00 mov 0xb0(%rbx,%r8,1),%rdi
a8c95: 00
a8c96: 4e 8b 8c 03 b8 00 00 mov 0xb8(%rbx,%r8,1),%r9
a8c9d: 00
a8c9e: 48 c1 e1 04 shl $0x4,%rcx
a8ca2: 48 89 3c 0c mov %rdi,(%rsp,%rcx,1)
a8ca6: 4c 89 4c 0c 08 mov %r9,0x8(%rsp,%rcx,1)
a8cab: 39 c6 cmp %eax,%esi
a8cad: 7e 27 jle a8cd6 <picture_Clone@@Base+0x126>
a8caf: 48 98 cltq
a8cb1: 48 8d 70 01 lea 0x1(%rax),%rsi
a8cb5: 48 c1 e0 05 shl $0x5,%rax
a8cb9: 4c 8b 94 03 b0 00 00 mov 0xb0(%rbx,%rax,1),%r10
a8cc0: 00
a8cc1: 48 8b 84 03 b8 00 00 mov 0xb8(%rbx,%rax,1),%rax
a8cc8: 00
a8cc9: 48 c1 e6 04 shl $0x4,%rsi
a8ccd: 4c 89 14 34 mov %r10,(%rsp,%rsi,1)
a8cd1: 48 89 44 34 08 mov %rax,0x8(%rsp,%rsi,1)
The specific cause of the vlc crash is that the 'mov %rdi,0x20(%rsp)' at a8c38 writes a bogus value to the res.p[1].p_pixels pointer, as rdi contains picture->p[0].i_lines and picture->p[0].i_pitch. The rdi register is correctly used by 'mov %rdi,0x18(%rsp)' at a8c3d to write them to res.p[0].i_lines and res.p[0].i_pitch.
When that p_pixels pointer is later accessed, it results in a segmentation fault.
The problem in gcc is triggered by this commit:https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e93428a8b056aed83a7678d4dc8272131ab671ba
>From e93428a8b056aed83a7678d4dc8272131ab671ba Mon Sep 17 00:00:00 2001
From: Richard Biener <rguenther@suse.de>
Date: Mon, 14 Sep 2020 11:25:04 +0200
Subject: [PATCH] tree-optimization/97043 - fix latent wrong-code with SLP
vectorization
When the unrolling decision comes late and would have prevented
eliding a SLP load permutation we can end up generating aligned
loads when the load is in fact unaligned. Most of the time
alignment analysis figures out the load is in fact unaligned
but that cannot be relied upon.
The following removes the SLP load permutation eliding based on
the still premature vectorization factor.
2020-09-14 Richard Biener <rguenther@suse.de>
PR tree-optimization/97043
* tree-vect-slp.c (vect_analyze_slp_instance): Do not
elide a load permutation if the current vectorization
factor is one.
---
gcc/tree-vect-slp.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/gcc/tree-vect-slp.c b/gcc/tree-vect-slp.c
index f6331eeea86..3fdf56f9335 100644
--- a/gcc/tree-vect-slp.c
+++ b/gcc/tree-vect-slp.c
@@ -2309,9 +2309,8 @@ vect_analyze_slp_instance (vec_info *vinfo,
/* The load requires permutation when unrolling exposes
a gap either because the group is larger than the SLP
group-size or because there is a gap between the groups. */
- && (known_eq (unrolling_factor, 1U)
- || (group_size == DR_GROUP_SIZE (first_stmt_info)
- && DR_GROUP_GAP (first_stmt_info) == 0)))
+ && group_size == DR_GROUP_SIZE (first_stmt_info)
+ && DR_GROUP_GAP (first_stmt_info) == 0)
{
SLP_TREE_LOAD_PERMUTATION (load_node).release ();
continue;
--
2.18.4
After reverting this commit on top of gcc-10 version 10.2.0-9 and recompiling vlc, the crash does not happen.
Given that a rather simple loop gets miscompiled, this might affect many other packages, as well. Thus a higher severity seems warranted.
The problematic gcc commit should be reverted or gcc otherwise be fixed.
Afterwards, vlc simply needs to be recompiled to work again properly.
Regards,
Ahzo
Reply to: