Re: Speed Problem Copying Files
Am 10.05.2019 um 07:48 schrieb David Christensen:
> On 5/9/19 1:49 AM, Lothar Schilling wrote:
>> Hi everybody,
>>
>> for years I have used CentOS for our server landscape. Now I decided to
>> give Debian a try. I just set up a Stretch 9.8 system supposed to become
>> our main backup server. So I set up a backup job wih rsync. But the
>> going is really very very slooooow. Trying to figure out what's
>> happening:
>>
>> * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>> * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>> that's fine as well
>>
>> But whenever I try to rsync, cp or scp - whether copy files on the local
>> hard disk only or over the network - speed goes down to 500 kB/s.
>>
>> Filesystem is ext4.
>>
>> I don't have any clue about what's going on. Any kind of help would be
>> appreciated, thank you!
>>
>> Lothar
>
>
> On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14
> schrieb Jonas Smedegaard:
> >> Is _only_ transfer speed affected?
> >>
> >> I am no expert in this, but imagine that if you rsync massive amounts
> >> involving hardlinks then memory becomes a problem too.
> >>
> >> Perhaps run atop to monitor bottlenecks live
> >>
> >>
> >> - Jonas
> >>
> > It is most definitely not a memory or cpu problem. It's a HP Proliant
> > with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
> > problem stays the same if I just copy one large file.
>
>
> On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51
> schrieb Kevin DAGNEAUX:
> >> Check if it's related to the disk speed (hdparm and/or iotop).
> >>
> >> Kevin
> >>
> >
> > hdparm -tT /dev/sda
> > /dev/sda:
> > Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
> > Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72 MB/sec
> >
> > iotop -o (for rsync and cp)
> > Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
> > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
> > TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> > 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
> > --info=progress2 /daten/testfile /daten/testfile2
> >
> > iotop -o (for dd)
> > Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
> > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
> > TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> > 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
> > if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>
>
> On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27
> schrieb Martin:
> >> [..]
> >>> hdparm -tT /dev/sda
> >>> /dev/sda:
> >>> Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
> >>> Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72
> MB/sec
> >>>
> >>> iotop -o (for rsync and cp)
> >>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
> >>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
> >>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO>
> COMMAND
> >>> 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
> >>> --info=progress2 /daten/testfile /daten/testfile2
> >>>
> >>> iotop -o (for dd)
> >>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
> >>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
> >>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO>
> COMMAND
> >>> 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
> >>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
> >> Show us the 'dd if=/daten/testfile bs=1G oflag=direct
> of=/dev/null', please.
> >> If this is as slow as this ~480k/s above, check your disk's health
> status. Like with smartmontools or some disk-utility software.
> >>
> >> Martin
> >>
> > Fast enough...
> >
> > dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> > 10+0 Datensätze ein
> > 10+0 Datensätze aus
> > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
> >
> > dd if=/daten/testfile of=/dev/null
> > 20971520+0 Datensätze ein
> > 20971520+0 Datensätze aus
> > 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
>
>
> rsync(1) can be fickle, especially when host operating systems and/or
> rsync versions mismatch. But, you also seem to have file system issues.
>
>
> The above information could be more useful if it were presented in
> context -- e.g. complete console sessions -- meaningful prompts (date,
> time, username, hostname, working directory), exact commands input,
> and exact output printed.
>
>
> Please use the Bash shell for the root account and the following line
> to /root/.bashrc:
>
> export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
>
>
> Then start a new shell, su(1) to root, change to a directory within
> the file system that is having issues, and run the following commands.
> Adjust the commands for your hosts and network, as required ('po' is
> my workstation, 'dpchrist' is my user account, 'samba' is my file
> server, and '/var/local/samba/dpchrist' is my storage directory on my
> file server):
>
> 2019-05-09 21:59:55 root@po /mnt/scratch
> # cat /etc/debian_version
> 9.9
>
> 2019-05-09 22:00:00 root@po /mnt/scratch
> # uname -a
> Linux po 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64
> GNU/Linux
>
> 2019-05-09 22:00:05 root@po /mnt/scratch
> # lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 55.9G 0 disk
> |-sda1 8:1 0 953M 0 part /boot
> |-sda2 8:2 0 1.9G 0 part
> | `-sda2_crypt 253:1 0 1.9G 0 crypt [SWAP]
> |-sda3 8:3 0 9.3G 0 part
> | `-sda3_crypt 253:0 0 9.3G 0 crypt /
> `-sda4 8:4 0 38.2G 0 part
> `-sda4_crypt 253:2 0 38.2G 0 crypt /mnt/scratch
> sr0 11:0 1 1024M 0 rom
>
> 2019-05-09 22:00:10 root@po /mnt/scratch
> # mount | egrep 'sd[a-z]'
> /dev/mapper/sda3_crypt on / type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/sda1 on /boot type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/mapper/sda4_crypt on /mnt/scratch type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
>
> 2019-05-09 22:00:19 root@po /mnt/scratch
> # df | egrep 'sd[a-z]'
> /dev/mapper/sda3_crypt 9763840 5540800 4160128 58% /
> /dev/sda1 975872 80892 800244 10% /boot
> /dev/mapper/sda4_crypt 40056832 1066752 38732224 3% /mnt/scratch
>
> 2019-05-09 22:00:27 root@po /mnt/scratch
> # time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.987 s, 67.2 MB/s
>
> real 0m15.988s
> user 0m0.000s
> sys 0m3.976s
>
> 2019-05-09 22:02:13 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && time dd if=foo of=/dev/null
> bs=1M count=1K
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49821 s, 430 MB/s
>
> real 0m2.518s
> user 0m0.000s
> sys 0m0.252s
>
> 2019-05-09 22:27:32 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
> uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && ssh $uh rm -f
> $p/$g && rsync -a --stats $f $uh:$p/$g
>
> Number of files: 1 (reg: 1)
> Number of created files: 1 (reg: 1)
> Number of deleted files: 0
> Number of regular files transferred: 1
> Total file size: 1,073,741,824 bytes
> Total transferred file size: 1,073,741,824 bytes
> Literal data: 1,073,741,824 bytes
> Matched data: 0 bytes
> File list size: 0
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 1,074,004,051
> Total bytes received: 35
>
> sent 1,074,004,051 bytes received 35 bytes 113,053,061.68 bytes/sec
> total size is 1,073,741,824 speedup is 1.00
>
> 2019-05-09 22:27:52 root@po /mnt/scratch
> # echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar &&
> uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && rm -f $g &&
> rsync -a --stats $uh:$p/$g $g
>
> Number of files: 1 (reg: 1)
> Number of created files: 1 (reg: 1)
> Number of deleted files: 0
> Number of regular files transferred: 1
> Total file size: 1,073,741,824 bytes
> Total transferred file size: 1,073,741,824 bytes
> Literal data: 1,073,741,824 bytes
> Matched data: 0 bytes
> File list size: 43
> File list generation time: 0.022 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 43
> Total bytes received: 1,074,004,061
>
> sent 43 bytes received 1,074,004,061 bytes 102,286,105.14 bytes/sec
> total size is 1,073,741,824 speedup is 1.00
>
>
> Add in other commands as you see fit, such as whatever generated the
> output in your previous posts.
>
>
> Please reply with your *complete* console session (you can remove all
> of the above in your reply).
>
>
> David
>
Wow, thank you! Getting into that on Monday hoping I can deliver...
Reply to: