Re: Speed Problem Copying Files
Am 10.05.2019 um 10:51 schrieb Lothar Schilling:
> Am 10.05.2019 um 07:48 schrieb David Christensen:
>> On 5/9/19 1:49 AM, Lothar Schilling wrote:
>>> Hi everybody,
>>>
>>> for years I have used CentOS for our server landscape. Now I decided to
>>> give Debian a try. I just set up a Stretch 9.8 system supposed to become
>>> our main backup server. So I set up a backup job wih rsync. But the
>>> going is really very very slooooow. Trying to figure out what's
>>> happening:
>>>
>>> * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
>>> * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
>>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
>>> that's fine as well
>>>
>>> But whenever I try to rsync, cp or scp - whether copy files on the local
>>> hard disk only or over the network - speed goes down to 500 kB/s.
>>>
>>> Filesystem is ext4.
>>>
>>> I don't have any clue about what's going on. Any kind of help would be
>>> appreciated, thank you!
>>>
>>> Lothar
>>
>> On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14
>> schrieb Jonas Smedegaard:
>>>> Is _only_ transfer speed affected?
>>>>
>>>> I am no expert in this, but imagine that if you rsync massive amounts
>>>> involving hardlinks then memory becomes a problem too.
>>>>
>>>> Perhaps run atop to monitor bottlenecks live
>>>>
>>>>
>>>> - Jonas
>>>>
>>> It is most definitely not a memory or cpu problem. It's a HP Proliant
>>> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
>>> problem stays the same if I just copy one large file.
>>
>> On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51
>> schrieb Kevin DAGNEAUX:
>>>> Check if it's related to the disk speed (hdparm and/or iotop).
>>>>
>>>> Kevin
>>>>
>>> hdparm -tT /dev/sda
>>> /dev/sda:
>>> Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
>>> Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72 MB/sec
>>>
>>> iotop -o (for rsync and cp)
>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>>> 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
>>> --info=progress2 /daten/testfile /daten/testfile2
>>>
>>> iotop -o (for dd)
>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>>> 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>>
>> On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27
>> schrieb Martin:
>>>> [..]
>>>>> hdparm -tT /dev/sda
>>>>> /dev/sda:
>>>>> Timing cached reads: 13348 MB in 2.00 seconds = 6683.42 MB/sec
>>>>> Timing buffered disk reads: 1014 MB in 3.00 seconds = 337.72
>> MB/sec
>>>>> iotop -o (for rsync and cp)
>>>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 476.15 K/s
>>>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 487.86 K/s
>>>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO>
>> COMMAND
>>>>> 19531 be/4 root 0.00 B/s 476.15 K/s 0.00 % 99.24 % rsync
>>>>> --info=progress2 /daten/testfile /daten/testfile2
>>>>>
>>>>> iotop -o (for dd)
>>>>> Total DISK READ : 0.00 B/s | Total DISK WRITE : 297.68 M/s
>>>>> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 297.68 M/s
>>>>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO>
>> COMMAND
>>>>> 19557 be/4 root 0.00 B/s 297.68 M/s 0.00 % 99.99 % dd
>>>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>>>> Show us the 'dd if=/daten/testfile bs=1G oflag=direct
>> of=/dev/null', please.
>>>> If this is as slow as this ~480k/s above, check your disk's health
>> status. Like with smartmontools or some disk-utility software.
>>>> Martin
>>>>
>>> Fast enough...
>>>
>>> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
>>> 10+0 Datensätze ein
>>> 10+0 Datensätze aus
>>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
>>>
>>> dd if=/daten/testfile of=/dev/null
>>> 20971520+0 Datensätze ein
>>> 20971520+0 Datensätze aus
>>> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s
>> Please reply with your *complete* console session (you can remove all
>> of the above in your reply).
>>
>> David
>>
# cat /etc/debian_version
9.9
# uname -a
Linux [my.server.com] 4.9.0-9-686-pae #1 SMP Debian 4.9.168-1
(2019-04-12) i686 GNU/Linux
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 3,7T 0 disk
├─sda1 8:1 0 476M 0 part /boot
├─sda2 8:2 0 93,1G 0 part /
├─sda3 8:3 0 18,6G 0 part [SWAP]
└─sda4 8:4 0 931,3G 0 part /daten
# mount | egrep 'sd[a-z]'
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda4 on /daten type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
# df | egrep 'sd[a-z]'
/dev/sda2 95596964 2241952 88455840 3% /
/dev/sda4 960185376 41249236 870091648 5% /daten
/dev/sda1 463826 57063 378296 14% /boot
# time dd if=/dev/urandom of=test bs=1M count=100 conv=fsync
100+0 Datensätze ein
100+0 Datensätze aus
104857600 Bytes (105 MB, 100 MiB) kopiert, 192,778 s, 544 kB/s
real 3m12,781s
user 0m0,000s
sys 0m1,480s
# echo 3 > /proc/sys/vm/drop_caches && time dd if=test of=/dev/null
204800+0 Datensätze ein
204800+0 Datensätze aus
104857600 Bytes (105 MB, 100 MiB) kopiert, 0,600354 s, 175 MB/s
real 0m0,675s
user 0m0,048s
sys 0m0,340s
# echo 3 > /proc/sys/vm/drop_caches && f=test &&
uh=root@[my.other.server.com] && p=/tmp/test && ssh $uh rm -f $p/$g &&
rsync -a --stats $f $uh:$p
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.021 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 104,883,277
Total bytes received: 34
sent 104,883,277 bytes received 34 bytes 13,984,441.47 bytes/sec
total size is 104,857,600 speedup is 1.00
# echo 3 > /proc/sys/vm/drop_caches && f=test && p=/tmp && rsync -a
--stats $f $p
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 104,857,600 bytes
Total transferred file size: 104,857,600 bytes
Literal data: 104,857,600 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.021 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 104,883,283
Total bytes received: 35
sent 104,883,283 bytes received 35 bytes 629,929.84 bytes/sec
total size is 104,857,600 speedup is 1.00
Reply to: