Bug#1093696: rsync: segfault in write_sparse since rsync 3.2.3-4+deb11u2
Package: rsync
Version: 3.2.3-4+deb11u3
Severity: normal
X-Debbugs-Cc: debian-lts@lists.debian.org
Dear Maintainer,
since the recent security updates to rsync, we're seeing segfaults on
several of our customer's backup servers which are mostly running on
Debian bullseye. The affected versions so far seem to be 3.2.3-4+deb11u2
(the first version that caused segfaults on Debian bullseye servers),
3.2.3-4+deb11u3 (still causing segfaults), and 3.2.7-1+deb12u2 (still
causing segfaults after we upgraded one of the affected servers to Debian
bookworm):
5bdb2681 ~ # zgrep segf /var/log/syslog*(.Om)
/var/log/syslog-20250120:Jan 20 21:07:01 backup kernel: [6120345.736990] rsync[1616623]: segfault at 0 ip 000055918f690fb0 sp 00007ffffcfa0950 error 4 in rsync[55918f653000+52000]
5bdb2681 ~ # grep rsync /var/log/dpkg.log(.Om) | grep upgrade
2025-01-15 03:06:35 upgrade rsync:amd64 3.2.3-4+deb11u1 3.2.3-4+deb11u2
2025-01-18 03:06:36 upgrade rsync:amd64 3.2.3-4+deb11u2 3.2.3-4+deb11u3
2025-01-21 12:02:56 upgrade rsync:amd64 3.2.3-4+deb11u3 3.2.7-1+deb12u2
5bdb2681 ~ # gdb x/usr/bin/rsync /var/tmp/core_rsync_1737403621_1616623 -ex bt
[...]
Core was generated by `rsync -e ssh -o LogLevel=ERROR -o BatchMode=yes --delete --stats --no-human-rea'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055918f690fb0 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
79 fileio.c: Datei oder Verzeichnis nicht gefunden.
#0 0x000055918f690fb0 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
#1 write_file (f=f@entry=3, use_seek=use_seek@entry=1, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=4368) at fileio.c:153
#2 0x000055918f6912db in skip_matched (fd=fd@entry=3, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=4368) at fileio.c:193
#3 0x000055918f667bea in receive_data (f_in=f_in@entry=5, fname_r=fname_r@entry=0x55918f6d2400 <partial_fname> "/var/lib/iserv/backup/mnt/sda4/iserv/.rsync-partial/access.log", fd_r=fd_r@entry=-1, size_r=<optimized out>,
fname=fname@entry=0x7ffffcfa0be0 "var/log/nginx/access.log", fd=fd@entry=3, file=0x5591affa15b8, inplace_sizing=1) at receiver.c:361
#4 0x000055918f668e61 in recv_files (f_in=f_in@entry=5, f_out=f_out@entry=6, local_name=local_name@entry=0x0) at receiver.c:875
#5 0x000055918f673f81 in do_recv (f_in=f_in@entry=5, f_out=6, f_out@entry=4, local_name=local_name@entry=0x0) at main.c:1052
#6 0x000055918f674b20 in client_run (f_in=5, f_out=4, pid=pid@entry=1616622, argc=argc@entry=1, argv=argv@entry=0x5591aaa965d8) at main.c:1365
#7 0x000055918f65462a in start_client (argv=<optimized out>, argc=1) at main.c:1583
#8 main (argc=<optimized out>, argv=<optimized out>) at main.c:1815
4fa1ef0d ~ # zgrep segf /var/log/syslog*(.Om)
/var/log/syslog-20250115.gz:Jan 15 21:31:09 backup kernel: [4480322.015591] rsync[2032758]: segfault at 0 ip 0000560bd2860f60 sp 00007ffee3cca280 error 4 in rsync[560bd2823000+52000]
/var/log/syslog-20250116.gz:Jan 16 21:28:45 backup kernel: [4566579.181588] rsync[2066066]: segfault at 0 ip 000055f08ef02f60 sp 00007ffdb3df4550 error 4 in rsync[55f08eec5000+52000]
/var/log/syslog-20250117.gz:Jan 17 21:30:09 backup kernel: [4653065.047654] rsync[2098363]: segfault at 0 ip 000055d9e9f3df60 sp 00007ffe4cbc7be0 error 4 in rsync[55d9e9f00000+52000]
/var/log/syslog-20250118.gz:Jan 18 21:25:53 backup kernel: [4739210.812916] rsync[2132252]: segfault at 0 ip 000055ad89374fb0 sp 00007ffc3c938460 error 4 in rsync[55ad89337000+52000]
/var/log/syslog-20250119.gz:Jan 19 21:27:52 backup kernel: [4825731.020182] rsync[2162792]: segfault at 0 ip 0000562472ccefb0 sp 00007ffdfc153ea0 error 4 in rsync[562472c91000+52000]
/var/log/syslog-20250120:Jan 20 21:28:57 backup kernel: [4912198.286147] rsync[2199490]: segfault at 0 ip 000055cfd1aeefb0 sp 00007ffc819c79a0 error 4 in rsync[55cfd1ab1000+52000]
/var/log/syslog:Jan 21 08:54:39 backup kernel: [4953340.644919] rsync[2226848]: segfault at 0 ip 00005609330c6fb0 sp 00007ffea8927420 error 4 in rsync[560933089000+52000]
/var/log/syslog:Jan 21 09:26:52 backup kernel: [4955273.333927] rsync[2242720]: segfault at 0 ip 000056117dcf7fb0 sp 00007ffde2f5b4c0 error 4 in rsync[56117dcba000+52000]
/var/log/syslog:Jan 21 13:24:07 backup kernel: [ 2881.529493] rsync[3403]: segfault at 0 ip 000055b95f896c38 sp 00007ffd9c34eeb0 error 4 in rsync[55b95f856000+56000] likely on CPU 0 (core 0, socket 0)
/var/log/syslog:Jan 21 14:10:08 backup kernel: [ 5642.034167] rsync[18926]: segfault at 0 ip 00005652ecc17c38 sp 00007fff0f09d150 error 4 in rsync[5652ecbd7000+56000] likely on CPU 0 (core 0, socket 0)
/var/log/syslog:Jan 21 14:38:11 backup kernel: [ 7325.145931] rsync[40067]: segfault at 0 ip 000055975f5e6c38 sp 00007ffc69968160 error 4 in rsync[55975f5a6000+56000] likely on CPU 0 (core 0, socket 0)
4fa1ef0d ~ # grep rsync /var/log/dpkg.log(.Om) | grep upgrade
2025-01-15 03:38:23 upgrade rsync:amd64 3.2.3-4+deb11u1 3.2.3-4+deb11u2
2025-01-18 03:38:21 upgrade rsync:amd64 3.2.3-4+deb11u2 3.2.3-4+deb11u3
2025-01-21 12:16:18 upgrade rsync:amd64 3.2.3-4+deb11u3 3.2.7-1+deb12u2
4fa1ef0d ~ # gdb deb11u2/usr/bin/rsync /var/tmp/core_rsync_1736973069_2032758
[...]
Core was generated by `rsync -e ssh -o LogLevel=ERROR -o BatchMode=yes --delete --stats --no-human-rea'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000560bd2860f60 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
79 fileio.c: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0 0x0000560bd2860f60 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
#1 write_file (f=f@entry=3, use_seek=use_seek@entry=1, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=43768) at fileio.c:153
#2 0x0000560bd286128b in skip_matched (fd=fd@entry=3, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=43768) at fileio.c:193
#3 0x0000560bd2837b9a in receive_data (f_in=f_in@entry=5, fname_r=fname_r@entry=0x560bd28a2400 <partial_fname> "/var/lib/iserv/backup/mnt/md2/iserv/.rsync-partial/pgdump_latest.sql", fd_r=fd_r@entry=-1,
size_r=<optimized out>, fname=fname@entry=0x7ffee3cca510 "var/backups/postgresql/pgdump_latest.sql", fd=fd@entry=3, file=0x560c019f7430, inplace_sizing=1) at receiver.c:361
#4 0x0000560bd2838e11 in recv_files (f_in=f_in@entry=5, f_out=f_out@entry=6, local_name=local_name@entry=0x0) at receiver.c:875
#5 0x0000560bd2843f31 in do_recv (f_in=f_in@entry=5, f_out=6, f_out@entry=4, local_name=local_name@entry=0x0) at main.c:1052
#6 0x0000560bd2844ad0 in client_run (f_in=5, f_out=4, pid=pid@entry=2032741, argc=argc@entry=1, argv=argv@entry=0x560bfa8e45d8) at main.c:1365
#7 0x0000560bd282462a in start_client (argv=<optimized out>, argc=1) at main.c:1583
#8 main (argc=<optimized out>, argv=<optimized out>) at main.c:1815
4fa1ef0d ~ # gdb =rsync /var/tmp/core_rsync_1737466691_40067
[...]
Core was generated by `rsync -e ssh -o LogLevel=ERROR -o BatchMode=yes --delete --stats --no-human-rea'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055975f5e6c38 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at ./fileio.c:79
79 ./fileio.c: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0 0x000055975f5e6c38 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at ./fileio.c:79
#1 write_file (f=f@entry=3, use_seek=use_seek@entry=1, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=43768) at ./fileio.c:153
#2 0x000055975f5e6f4b in skip_matched (fd=fd@entry=3, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=43768) at ./fileio.c:193
#3 0x000055975f5baeb3 in receive_data (f_in=f_in@entry=5, fname_r=fname_r@entry=0x55975f6297c0 <partial_fname> "/var/lib/iserv/backup/mnt/md2/iserv/.rsync-partial/pgdump_latest.sql", fd_r=fd_r@entry=-1, size_r=<optimized out>,
fname=fname@entry=0x7ffc699683f0 "var/backups/postgresql/pgdump_latest.sql", fd=fd@entry=3, file=0x55977ca61650, inplace_sizing=1) at ./receiver.c:363
#4 0x000055975f5bbfe7 in recv_files (f_in=f_in@entry=5, f_out=f_out@entry=6, local_name=local_name@entry=0x0) at ./receiver.c:892
#5 0x000055975f5c8255 in do_recv (f_in=f_in@entry=5, f_out=6, f_out@entry=4, local_name=local_name@entry=0x0) at ./main.c:1056
#6 0x000055975f5c8de7 in client_run (f_in=5, f_out=4, pid=pid@entry=40060, argc=argc@entry=1, argv=argv@entry=0x55976eed95d8) at ./main.c:1370
#7 0x000055975f5a764c in start_client (argv=<optimized out>, argc=1) at ./main.c:1601
#8 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:1861
7aca0bb0 ~ # zgrep segf /var/log/syslog*(.Om)
/var/log/syslog-20250120:Jan 20 21:38:43 backup kernel: [7594798.844664] rsync[3425240]: segfault at 0 ip 000056459d52bfb0 sp 00007fff3477f630 error 4 in rsync[56459d4ee000+52000]
7aca0bb0 ~ # grep rsync /var/log/dpkg.log(.Om) | grep upgrade
2025-01-15 02:30:52 upgrade rsync:amd64 3.2.3-4+deb11u1 3.2.3-4+deb11u2
2025-01-18 02:30:51 upgrade rsync:amd64 3.2.3-4+deb11u2 3.2.3-4+deb11u3
7aca0bb0 ~ # gdb =rsync /var/tmp/core_rsync_1737405523_3425240 -ex bt -ex quit
[...]
Core was generated by `rsync -e ssh -o LogLevel=ERROR -o BatchMode=yes --delete --stats --no-human-rea'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000056459d52bfb0 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
79 fileio.c: Datei oder Verzeichnis nicht gefunden.
#0 0x000056459d52bfb0 in write_sparse (len=1024, buf=0x0, offset=0, use_seek=1, f=3) at fileio.c:79
#1 write_file (f=f@entry=3, use_seek=use_seek@entry=1, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=12368) at fileio.c:153
#2 0x000056459d52c2db in skip_matched (fd=fd@entry=3, offset=offset@entry=0, buf=buf@entry=0x0, len=len@entry=12368) at fileio.c:193
#3 0x000056459d502bea in receive_data (f_in=f_in@entry=5, fname_r=fname_r@entry=0x56459d56d400 <partial_fname> "/var/lib/iserv/backup/mnt/sda3/iserv/.rsync-partial/access.log", fd_r=fd_r@entry=-1, size_r=<optimized out>,
fname=fname@entry=0x7fff3477f8c0 "var/log/squid/access.log", fd=fd@entry=3, file=0x5645c85b6268, inplace_sizing=1) at receiver.c:361
#4 0x000056459d503e61 in recv_files (f_in=f_in@entry=5, f_out=f_out@entry=6, local_name=local_name@entry=0x0) at receiver.c:875
#5 0x000056459d50ef81 in do_recv (f_in=f_in@entry=5, f_out=6, f_out@entry=4, local_name=local_name@entry=0x0) at main.c:1052
#6 0x000056459d50fb20 in client_run (f_in=5, f_out=4, pid=pid@entry=3425239, argc=argc@entry=1, argv=argv@entry=0x5645c679a5d8) at main.c:1365
#7 0x000056459d4ef62a in start_client (argv=<optimized out>, argc=1) at main.c:1583
#8 main (argc=<optimized out>, argv=<optimized out>) at main.c:1815
I unfortunately haven't been able to manually reproduce the issue so far by
directly invoking rsync; the crashes have only occurred when rsync was called
from our backup script. Sometimes the backup script causes rsync to crash after
it has been running for a while (~30 minutes), sometimes it runs to completion.
rsync is called from the backup script with the following arguments:
rsync -e 'ssh -o LogLevel=ERROR -o BatchMode=yes'
--delete --stats --no-human-readable
--numeric-ids -aH -A -f 'P /rsync.out' -f 'P /rsync.err' --sparse
--partial-dir=/var/lib/iserv/backup/mnt/md2/iserv/.rsync-partial
--exclude-from=/usr/share/iserv/backup/exclude
--link-dest=/var/lib/iserv/backup/mnt/md2/iserv/2025-01-14T20:00:05+00:00
172.27.16.2:/ /var/lib/iserv/backup/mnt/md2/iserv/partial/
The filenames in receive_data seem to indicate that it might be mostly log
files that cause rsync to crash; maybe it's caused by concurrent write access
to these files?
-- System Information:
Debian Release: 11.11
APT prefers oldstable-updates
APT policy: (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-33-amd64 (SMP w/4 CPU threads)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages rsync depends on:
ii init-system-helpers 1.60
ii libacl1 2.2.53-10
ii libc6 2.31-13+deb11u11
ii liblz4-1 1.9.3-2
ii libpopt0 1.18-2
ii libssl1.1 1.1.1w-0+deb11u2
ii libxxhash0 0.8.0-2
ii libzstd1 1.4.8+dfsg-2.1
ii lsb-base 11.1.0
ii zlib1g 1:1.2.11.dfsg-2+deb11u2
rsync recommends no packages.
Versions of packages rsync suggests:
ii openssh-client 1:8.4p1-5+deb11u3
ii openssh-server 1:8.4p1-5+deb11u3
ii python3 3.9.2-3
-- no debconf information
Reply to: