I am experiencing some issues with load after upgrading some of my Squeeze boxes to Wheezy. I have 7 app servers, all with identical hardware with identical packages and code. I upgraded one of my boxes to wheezy, along with the custom packages we use for Python, PHP, etc… Same versions of the software, just built on Wheezy instead of Squeeze. My problem is that my Wheezy boxes have a load of over 3 and are not staying up during our peak time, whereas our squeeze boxes have a load of less than 1. The interesting part, is that despite the high load, my wheezy boxes are actually performing quite well, and are outperforming my squeeze boxes by 2-3 ms. Never the less, the high load is giving us cause for concern and is stopping us from migrating completely, and we're wondering if anybody else is seeing the same thing or can give us some assistance on where to go from here. I believe I have tracked down the issue with our load to be an interrupt issue. My interrupts on wheezy are way higher. CPU, I/O, Memory and Context Switches are all the same (measured with top, atop, iotop, vmstat). It doesn't appear to be a hardware issue, as I deployed wheezy and our code base to a different and faster motherboard/cpu combo, and the issue remained. The items that stands out is that my "Rescheduling Interrupts" and "timer" are interrupting like crazy on wheezy compared to squeeze. Here is my output of total interrupts on Squeeze vs Wheezy for two different machines, rebooted and placed into service at the exact same time, with traffic distributed to them via round robin, so it should be fairly equal.
Rescheduling Interrupts: 4109580 on Wheezy vs 67418 on Squeeze. Timer: 504238 on Wheezy vs 50 on Squeeze.
Thoughts? Suggestions?
Here is my squeeze box interrupts:
# sudo cat /proc/interrupts | awk '{ print $18, $19, $2+$3+$4+$5+$6+$7+$8+$9+$10+$11+$12+$13+$14+$15+$16+$17 }'
0
IO-APIC-edge timer 50
IO-APIC-edge i8042 2
IO-APIC-edge serial 8
IO-APIC-edge rtc0 1
IO-APIC-fasteoi acpi 0
IO-APIC-edge i8042 4
IO-APIC-fasteoi uhci_hcd:usb2 0
IO-APIC-fasteoi ehci_hcd:usb1, 2
IO-APIC-fasteoi ata_piix, 24014
IO-APIC-fasteoi uhci_hcd:usb4 48
IO-APIC-fasteoi ehci_hcd:usb3, 0
PCI-MSI-edge eth0 1
PCI-MSI-edge eth0-TxRx-0 919924
PCI-MSI-edge eth0-TxRx-1 1206377
PCI-MSI-edge eth0-TxRx-2 1208344
PCI-MSI-edge eth0-TxRx-3 817225
PCI-MSI-edge eth0-TxRx-4 1165734
PCI-MSI-edge eth0-TxRx-5 1314252
PCI-MSI-edge eth0-TxRx-6 998115
PCI-MSI-edge eth0-TxRx-7 1229384
PCI-MSI-edge eth1 1
PCI-MSI-edge eth1-TxRx-0 4776
PCI-MSI-edge eth1-TxRx-1 4444
PCI-MSI-edge eth1-TxRx-2 5557
PCI-MSI-edge eth1-TxRx-3 5344
PCI-MSI-edge eth1-TxRx-4 5827
PCI-MSI-edge eth1-TxRx-5 5060
PCI-MSI-edge eth1-TxRx-6 4078
PCI-MSI-edge eth1-TxRx-7 4317
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
Non-maskable interrupts 0
Local timer 3968846
Spurious interrupts 0
Performance monitoring 0
Performance pending 0
Rescheduling interrupts 67418
Function call 16404
TLB shootdowns 33915
Thermal event 0
Threshold APIC 0
Machine check 0
Machine check 128
Here is my wheezy interrupts:
# sudo cat /proc/interrupts | awk '{ print $18, $19, $2+$3+$4+$5+$6+$7+$8+$9+$10+$11+$12+$13+$14+$15+$16+$17 }'
IO-APIC-edge timer 504238
IO-APIC-edge i8042 3
IO-APIC-edge serial 12
IO-APIC-edge rtc0 1
IO-APIC-fasteoi acpi 0
IO-APIC-edge i8042 4
IO-APIC-fasteoi uhci_hcd:usb3 0
IO-APIC-fasteoi ehci_hcd:usb1, 2
IO-APIC-fasteoi ata_piix, 21189
IO-APIC-fasteoi uhci_hcd:usb4 47
IO-APIC-fasteoi ehci_hcd:usb2, 0
PCI-MSI-edge eth0 1
PCI-MSI-edge eth0-TxRx-0 1506134
PCI-MSI-edge eth0-TxRx-1 1102085
PCI-MSI-edge eth0-TxRx-2 1399087
PCI-MSI-edge eth0-TxRx-3 1123149
PCI-MSI-edge eth0-TxRx-4 849678
PCI-MSI-edge eth0-TxRx-5 1428705
PCI-MSI-edge eth0-TxRx-6 897420
PCI-MSI-edge eth0-TxRx-7 1321820
PCI-MSI-edge eth1 1
PCI-MSI-edge eth1-TxRx-0 4290
PCI-MSI-edge eth1-TxRx-1 4217
PCI-MSI-edge eth1-TxRx-2 3685
PCI-MSI-edge eth1-TxRx-3 4081
PCI-MSI-edge eth1-TxRx-4 5532
PCI-MSI-edge eth1-TxRx-5 6604
PCI-MSI-edge eth1-TxRx-6 3996
PCI-MSI-edge eth1-TxRx-7 4560
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
PCI-MSI-edge ioat-msix 3
Non-maskable interrupts 3847
Local timer 3846061
Spurious interrupts 0
Performance monitoring 3847
IRQ work 0
Rescheduling interrupts 4109580
Function call 13442
TLB shootdowns 1745
Thermal event 0
Threshold APIC 0
Machine check 0
Machine check 128 |