Slow network performance with KVM
Hi everybody
I've setup a Debian Squeeze box as a KVM host. The box is nothing fancy: i5-2400 CPU @ 3.10GHz, 16GB DDR3, a consumer grade SATA HDD. I have guests also running Debian Squeeze. I noticed a rather poor network guest-guest performance. All guests are using virtio-net devices. I noticed that during the iperf test I get a high CPU usage on the the Host. The highest usage (up to 200% on 4CPU box) is by the kvm process of the iperf server. The results of a few tests show:
root@icinga:~# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 172.29.5.10 port 43975 connected with 172.29.5.14 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-120.0 sec 2.60 GBytes 186 Mbits/sec
root@icinga:~# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[ 3] local 172.29.5.10 port 44387 connected with 172.29.5.14 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-120.0 sec 2.55 GBytes 183 Mbits/sec
When running the test I observe a large CPU usage increase on the Host system (first 4 lines - before starting iperf):
root@vmhost:/var/log/libvirt/qemu# vmstat 5 10000
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 506548 1602376 85604 0 0 59 93 11 15 6 5 86 2
1 0 0 505360 1602732 85628 0 0 0 284 23089 42288 9 5 84 1
1 0 0 505560 1603068 85628 0 0 0 403 23388 43214 7 5 88 0
0 0 0 505072 1603656 85616 0 0 33 190 22469 41945 6 5 89 0
5 0 0 504516 1603780 85628 0 0 0 546 26539 47286 39 6 45 10
3 0 0 504324 1604064 85624 0 0 0 288 28181 48367 48 6 46 0
4 0 0 503336 1604900 85612 0 0 37 884 29423 50074 48 7 41 3
4 0 0 503328 1605212 85628 0 0 0 131 27129 47701 46 6 48 0
9 0 0 503264 1605552 85620 0 0 0 426 28119 48165 50 6 44 0
4 0 0 501928 1606424 85512 0 0 48 512 27317 48005 47 6 46 2
4 0 0 500704 1607264 85628 0 0 0 533 27629 48272 48 5 47 0
5 0 0 500084 1607512 85628 0 0 0 168 27463 48135 45 6 50 0
7 0 0 499568 1607948 85624 0 0 1 373 27446 48136 49 6 44 1
4 0 0 499816 1608172 85628 0 0 0 210 27192 46950 50 6 44 0
3 0 0 528436 1579280 85600 0 0 0 333 28237 48356 49 6 44 1
6 0 0 528256 1580048 85564 0 0 43 506 27055 46855 46 6 44 4
3 0 0 527944 1580280 85616 0 0 0 330 27503 47664 44 6 46 4
3 0 0 526720 1580836 85592 0 0 0 349 27827 47899 48 6 46 0
4 0 0 525444 1581392 85596 0 0 10 746 28851 49113 49 7 42 2
3 0 0 524996 1581812 85616 0 0 0 298 27598 48460 45 6 49 0
17 0 0 525236 1582284 85612 0 0 0 526 29020 49975 48 7 45 0
2 0 0 523832 1583332 85608 0 0 85 574 27679 48535 46 6 45 3
4 0 0 523368 1583676 85616 0 0 0 316 27642 48247 47 6 47 0
9 0 0 534340 1572292 85532 0 0 0 94 27368 47914 45 6 49 0
8 0 0 534200 1572484 85532 0 0 7 201 27742 48087 48 6 46 0
5 0 0 533912 1572792 85472 0 0 1 369 27154 46905 47 6 43 3
4 0 0 533732 1573096 85528 0 0 0 442 28313 48727 49 6 44 0
9 0 0 533840 1573632 85532 0 0 60 219 27323 47705 46 5 47 1
2 0 0 531752 1574048 85532 0 0 0 1098 24148 44185 15 6 55 24
7 0 0 531864 1574384 85516 0 0 0 326 23241 42814 7 5 87 0
0 5 0 526492 1575004 85528 0 0 23 775 23897 43717 7 6 78 9
1 0 0 530552 1575516 85552 0 0 0 395 22800 42004 5 6 87 2
Surprisingly I get a slightly better (still far from perfect) results running a test from a CentOS 6 guest as iperf client (server still on the same Debain guest):
[root@lovely sysconfig]# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[ 3] local 172.29.5.8 port 58957 connected with 172.29.5.14 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-120.0 sec 3.34 GBytes 239 Mbits/sec
[root@lovely sysconfig]# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[ 3] local 172.29.5.8 port 58958 connected with 172.29.5.14 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-120.1 sec 3.34 GBytes 239 Mbits/sec
The libvirt XML of one of the VMs:
root@vmhost:/var/log/libvirt/qemu# virsh dumpxml icinga
<domain type='kvm' id='9'>
<name>icinga</name>
<uuid>bcf8f7d5-6842-76d9-1fcb-c665298c0852</uuid>
<memory>1048576</memory>
<currentMemory>1048576</currentMemory>
<vcpu>4</vcpu>
<os>
<type arch='x86_64' machine='pc-0.12'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/rootvg/virt-icinga'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</disk>
<interface type='bridge'>
<mac address='52:54:00:61:b3:2d'/>
<source bridge='br0'/>
<target dev='vnet9'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/8'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/8'>
<source path='/dev/pts/8'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</memballoon>
</devices>
</domain>
Is there any way to get a better network performance? I've read somewhere on the internet somebody complaining about 22Gbit/s and here I cannot even reach half of a Gbit. Some kernel / virtio driver settings I should look at? I know that this is a hard to answer question but maybe I'm missing something obvious here?
Pozdrawiam / Kind Regards
Bogdan Katynski
Reply to: