[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Slow network performance with KVM



Hi everybody

I've setup a Debian Squeeze box as a KVM host. The box is nothing fancy: i5-2400 CPU @ 3.10GHz, 16GB DDR3, a consumer grade SATA HDD. I have guests also running Debian Squeeze. I noticed a rather poor network guest-guest performance. All guests are using virtio-net devices. I noticed that during the iperf test I get a high CPU usage on the the Host. The highest usage (up to 200% on 4CPU box) is by the kvm process of the iperf server. The results of a few tests show:

root@icinga:~# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 172.29.5.10 port 43975 connected with 172.29.5.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-120.0 sec  2.60 GBytes    186 Mbits/sec
root@icinga:~# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 172.29.5.10 port 44387 connected with 172.29.5.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-120.0 sec  2.55 GBytes    183 Mbits/sec

When running the test I observe a large CPU usage increase on the Host system (first 4 lines - before starting iperf):

root@vmhost:/var/log/libvirt/qemu# vmstat 5 10000
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 506548 1602376  85604    0    0    59    93   11   15  6  5 86  2
 1  0      0 505360 1602732  85628    0    0     0   284 23089 42288  9  5 84  1
 1  0      0 505560 1603068  85628    0    0     0   403 23388 43214  7  5 88  0
 0  0      0 505072 1603656  85616    0    0    33   190 22469 41945  6  5 89  0
 5  0      0 504516 1603780  85628    0    0     0   546 26539 47286 39  6 45 10
 3  0      0 504324 1604064  85624    0    0     0   288 28181 48367 48  6 46  0
 4  0      0 503336 1604900  85612    0    0    37   884 29423 50074 48  7 41  3
 4  0      0 503328 1605212  85628    0    0     0   131 27129 47701 46  6 48  0
 9  0      0 503264 1605552  85620    0    0     0   426 28119 48165 50  6 44  0
 4  0      0 501928 1606424  85512    0    0    48   512 27317 48005 47  6 46  2
 4  0      0 500704 1607264  85628    0    0     0   533 27629 48272 48  5 47  0
 5  0      0 500084 1607512  85628    0    0     0   168 27463 48135 45  6 50  0
 7  0      0 499568 1607948  85624    0    0     1   373 27446 48136 49  6 44  1
 4  0      0 499816 1608172  85628    0    0     0   210 27192 46950 50  6 44  0
 3  0      0 528436 1579280  85600    0    0     0   333 28237 48356 49  6 44  1
 6  0      0 528256 1580048  85564    0    0    43   506 27055 46855 46  6 44  4
 3  0      0 527944 1580280  85616    0    0     0   330 27503 47664 44  6 46  4
 3  0      0 526720 1580836  85592    0    0     0   349 27827 47899 48  6 46  0
 4  0      0 525444 1581392  85596    0    0    10   746 28851 49113 49  7 42  2
 3  0      0 524996 1581812  85616    0    0     0   298 27598 48460 45  6 49  0
17  0      0 525236 1582284  85612    0    0     0   526 29020 49975 48  7 45  0
 2  0      0 523832 1583332  85608    0    0    85   574 27679 48535 46  6 45  3
 4  0      0 523368 1583676  85616    0    0     0   316 27642 48247 47  6 47  0
 9  0      0 534340 1572292  85532    0    0     0    94 27368 47914 45  6 49  0
 8  0      0 534200 1572484  85532    0    0     7   201 27742 48087 48  6 46  0
 5  0      0 533912 1572792  85472    0    0     1   369 27154 46905 47  6 43  3
 4  0      0 533732 1573096  85528    0    0     0   442 28313 48727 49  6 44  0
 9  0      0 533840 1573632  85532    0    0    60   219 27323 47705 46  5 47  1
 2  0      0 531752 1574048  85532    0    0     0  1098 24148 44185 15  6 55 24
 7  0      0 531864 1574384  85516    0    0     0   326 23241 42814  7  5 87  0
 0  5      0 526492 1575004  85528    0    0    23   775 23897 43717  7  6 78  9
 1  0      0 530552 1575516  85552    0    0     0   395 22800 42004  5  6 87  2

Surprisingly I get a slightly better (still far from perfect) results running a test from a CentOS 6 guest as iperf client (server still on the same Debain guest):

[root@lovely sysconfig]# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 172.29.5.8 port 58957 connected with 172.29.5.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-120.0 sec  3.34 GBytes   239 Mbits/sec
[root@lovely sysconfig]# iperf -c otrs -t 120
------------------------------------------------------------
Client connecting to otrs, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 172.29.5.8 port 58958 connected with 172.29.5.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-120.1 sec  3.34 GBytes   239 Mbits/sec

The libvirt XML of one of the VMs:

root@vmhost:/var/log/libvirt/qemu# virsh dumpxml icinga
<domain type='kvm' id='9'>
  <name>icinga</name>
  <uuid>bcf8f7d5-6842-76d9-1fcb-c665298c0852</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/rootvg/virt-icinga'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:61:b3:2d'/>
      <source bridge='br0'/>
      <target dev='vnet9'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/8'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/8'>
      <source path='/dev/pts/8'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Is there any way to get a better network performance? I've read somewhere on the internet somebody complaining about 22Gbit/s and here I cannot even reach half of a Gbit. Some kernel / virtio driver settings I should look at? I know that this is a hard to answer question but maybe I'm missing something obvious here?

Pozdrawiam / Kind Regards
Bogdan Katynski


Reply to: