Bug#600031: updating kernel causes slow write speed to raid
Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4.
using raid5. Also to note that when booted with windows server 2003
using Everest disk benchmark it does an average of 34MB/s (peaking to
50MB/s).
Attached are bonnie++ output for the two kernel versions and ext3 and
ext4 (ext4 won't mount with 2.6.26 kernel).
And meminfo for both kernels taken when the system is loaded and unloaded.
Dave
On 13 October 2010 00:15, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Tue, 2010-10-12 at 23:07 +0100, Dave Clarke wrote:
>> Package: linux-image
>> Version: 2.6.32-5-686
>>
>> After dist-upgrade from Lenny to Squeeze write speed is halved. Shown
>> with bonnie++. Before upgrade write speed was 21MB/s, after upgrade
>> 10MB/s.
>
> Let's see the output from bonnie++. Also are you running this locally
> or over NFS or other network filesystem?
>
>> Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA.
>> Celeron(R) CPU 2.40GHz, 512MiB RAM.
>>
>> Confirmed link to kernel with:
>>
>> otherserver$ scp 4gbfile raid_server:
>> 100% 4000MB 6.6MB/s 10:06
>>
>> reboot server with kernel 2.6.26-2-686 leftover from Lenny.
>> otherserver$ scp 4gbfile raid_server:
>> 100% 4000MB 11.3MB/s 05:55
>>
>> Also of note that when copying a large file the system becomes heavily loaded.
>> iowait: 80.1%wa
>> load average: 3.58, 1.48, 0.66
>> Mem: 514428k total, 508276k used, 6152k free, 10084k buffers
>
> That seems like too little memory for a file server. Can you send the
> contents of /proc/meminfo?
>
> Also what filesystem is used on the RAID?
>
> Ben.
>
> --
> Ben Hutchings
> Once a job is fouled up, anything done to improve it makes it worse.
>
# Linux heterogeneous 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686 GNU/Linux
# 10 reps, for consistancy.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 421 96 17752 10 7734 4 1515 95 61396 17 516.5 29
Latency 53894us 1387ms 4065ms 58119us 28090us 1526ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 7452 27 +++++ +++ 11835 24 9917 34 +++++ +++ 12541 27
Latency 29878us 1355us 1420us 1142us 25us 107us
1.96,1.96,heterogeneous,1,1286995872,1G,,421,96,17752,10,7734,4,1515,95,61396,17,516.5,29,16,,,,,7452,27,+++++,+++,11835,24,9917,34,+++++,+++,12541,27,53894us,1387ms,4065ms,58119us,28090us,1526ms,29878us,1355us,1420us,1142us,25us,107us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 340 95 19407 12 7697 4 989 98 60791 17 512.0 36
Latency 56156us 1279ms 5718ms 48099us 59930us 99728us
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8668 30 +++++ +++ 6363 12 9615 31 +++++ +++ 12282 25
Latency 62301us 1336us 1362us 1096us 154us 104us
1.96,1.96,heterogeneous,1,1286993983,1G,,340,95,19407,12,7697,4,989,98,60791,17,512.0,36,16,,,,,8668,30,+++++,+++,6363,12,9615,31,+++++,+++,12282,25,56156us,1279ms,5718ms,48099us,59930us,99728us,62301us,1336us,1362us,1096us,154us,104us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 317 97 17955 11 7684 4 1424 97 58693 18 523.3 35
Latency 29157us 2438ms 4469ms 34983us 110ms 46833us
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 7668 28 +++++ +++ 12340 25 10329 36 +++++ +++ 10508 24
Latency 13663us 1364us 1382us 1110us 68us 82us
1.96,1.96,heterogeneous,1,1286994431,1G,,317,97,17955,11,7684,4,1424,97,58693,18,523.3,35,16,,,,,7668,28,+++++,+++,12340,25,10329,36,+++++,+++,10508,24,29157us,2438ms,4469ms,34983us,110ms,46833us,13663us,1364us,1382us,1110us,68us,82us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 351 97 19781 12 7751 4 1525 97 61980 18 514.0 34
Latency 26969us 1422ms 5937ms 51548us 42096us 757ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8408 28 +++++ +++ 12187 24 10183 33 +++++ +++ 12706 26
Latency 74856us 1332us 1388us 1668us 15us 173us
1.96,1.96,heterogeneous,1,1286994675,1G,,351,97,19781,12,7751,4,1525,97,61980,18,514.0,34,16,,,,,8408,28,+++++,+++,12187,24,10183,33,+++++,+++,12706,26,26969us,1422ms,5937ms,51548us,42096us,757ms,74856us,1332us,1388us,1668us,15us,173us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 345 95 20209 12 7681 4 1553 98 61279 17 520.3 31
Latency 59847us 920ms 5968ms 25166us 44631us 81126us
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8057 28 +++++ +++ 11396 23 9803 31 +++++ +++ 10796 22
Latency 12258us 1291us 1377us 1059us 142us 103us
1.96,1.96,heterogeneous,1,1286994929,1G,,345,95,20209,12,7681,4,1553,98,61279,17,520.3,31,16,,,,,8057,28,+++++,+++,11396,23,9803,31,+++++,+++,10796,22,59847us,920ms,5968ms,25166us,44631us,81126us,12258us,1291us,1377us,1059us,142us,103us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 411 95 9471 5 5843 3 1268 96 83927 18 232.4 10
Latency 82700us 3337ms 3593ms 59243us 29175us 764ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2646 11 +++++ +++ 9747 25 8050 34 +++++ +++ 9932 27
Latency 57065us 1214us 1344us 4109us 225us 1457us
1.96,1.96,heterogeneous,1,1286990873,1G,,411,95,9471,5,5843,3,1268,96,83927,18,232.4,10,16,,,,,2646,11,+++++,+++,9747,25,8050,34,+++++,+++,9932,27,82700us,3337ms,3593ms,59243us,29175us,764ms,57065us,1214us,1344us,4109us,225us,1457us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 324 93 13464 7 6883 3 1841 97 95552 19 232.1 9
Latency 202ms 2870ms 4815ms 39835us 15851us 781ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3795 16 +++++ +++ 9754 24 8072 34 +++++ +++ 9721 26
Latency 38191us 1102us 1170us 490us 72us 4077us
1.96,1.96,heterogeneous,1,1286988623,1G,,324,93,13464,7,6883,3,1841,97,95552,19,232.1,9,16,,,,,3795,16,+++++,+++,9754,24,8072,34,+++++,+++,9721,26,202ms,2870ms,4815ms,39835us,15851us,781ms,38191us,1102us,1170us,490us,72us,4077us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 394 91 13580 7 7217 3 1820 95 93597 18 212.2 9
Latency 284ms 3684ms 2505ms 46133us 42601us 767ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 5198 22 +++++ +++ 9551 24 7880 33 +++++ +++ 9563 27
Latency 48704us 1102us 1191us 497us 222us 113us
1.96,1.96,heterogeneous,1,1286995401,1G,,394,91,13580,7,7217,3,1820,95,93597,18,212.2,9,16,,,,,5198,22,+++++,+++,9551,24,7880,33,+++++,+++,9563,27,284ms,3684ms,2505ms,46133us,42601us,767ms,48704us,1102us,1191us,497us,222us,113us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 349 95 13128 4 6942 3 1849 97 101103 19 163.7 7
Latency 43757us 1339ms 1238ms 19461us 31574us 1201ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6179 32 +++++ +++ 10766 30 8030 39 +++++ +++ 10521 30
Latency 4120us 1153us 1227us 2386us 1480us 187us
1.96,1.96,heterogeneous,1,1286988387,1G,,349,95,13128,4,6942,3,1849,97,101103,19,163.7,7,16,,,,,6179,32,+++++,+++,10766,30,8030,39,+++++,+++,10521,30,43757us,1339ms,1238ms,19461us,31574us,1201ms,4120us,1153us,1227us,2386us,1480us,187us
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
heterogeneous 1G 398 96 13922 5 7655 3 1782 93 100252 18 226.6 9
Latency 50148us 923ms 1733ms 88175us 27818us 798ms
Version 1.96 ------Sequential Create------ --------Random Create--------
heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3173 16 +++++ +++ 10747 31 8102 40 +++++ +++ 11008 32
Latency 35254us 1147us 1381us 426us 32us 265us
1.96,1.96,heterogeneous,1,1286987900,1G,,398,96,13922,5,7655,3,1782,93,100252,18,226.6,9,16,,,,,3173,16,+++++,+++,10747,31,8102,40,+++++,+++,11008,32,50148us,923ms,1733ms,88175us,27818us,798ms,35254us,1147us,1381us,426us,32us,265us
== Loaded (bonnie++ benchmark) ==
MemTotal: 516044 kB
MemFree: 5664 kB
Buffers: 7856 kB
Cached: 461372 kB
SwapCached: 0 kB
Active: 25800 kB
Inactive: 458072 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 516044 kB
LowFree: 5664 kB
SwapTotal: 1508344 kB
SwapFree: 1507692 kB
Dirty: 192900 kB
Writeback: 132 kB
AnonPages: 14624 kB
Mapped: 8264 kB
Slab: 19944 kB
SReclaimable: 14356 kB
SUnreclaim: 5588 kB
PageTables: 636 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1766364 kB
Committed_AS: 179080 kB
VmallocTotal: 507896 kB
VmallocUsed: 3612 kB
VmallocChunk: 503784 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
== Unloaded ==
MemTotal: 516044 kB
MemFree: 475488 kB
Buffers: 316 kB
Cached: 8716 kB
SwapCached: 6940 kB
Active: 5644 kB
Inactive: 20508 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 516044 kB
LowFree: 475488 kB
SwapTotal: 1508344 kB
SwapFree: 1497064 kB
Dirty: 344 kB
Writeback: 0 kB
AnonPages: 10400 kB
Mapped: 6212 kB
Slab: 7840 kB
SReclaimable: 2416 kB
SUnreclaim: 5424 kB
PageTables: 620 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1766364 kB
Committed_AS: 178676 kB
VmallocTotal: 507896 kB
VmallocUsed: 3612 kB
VmallocChunk: 503784 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
== Loaded (bonnie++ benchmark) ==
MemTotal: 514428 kB
MemFree: 6488 kB
Buffers: 1108 kB
Cached: 466656 kB
SwapCached: 4672 kB
Active: 255448 kB
Inactive: 224660 kB
Active(anon): 1568 kB
Inactive(anon): 10852 kB
Active(file): 253880 kB
Inactive(file): 213808 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 514428 kB
LowFree: 6488 kB
SwapTotal: 1508344 kB
SwapFree: 1496036 kB
Dirty: 169888 kB
Writeback: 0 kB
AnonPages: 8576 kB
Mapped: 2168 kB
Shmem: 32 kB
Slab: 21236 kB
SReclaimable: 14844 kB
SUnreclaim: 6392 kB
KernelStack: 832 kB
PageTables: 784 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1765556 kB
Committed_AS: 190084 kB
VmallocTotal: 504056 kB
VmallocUsed: 7504 kB
VmallocChunk: 484316 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 16128 kB
DirectMap4M: 507904 kB
== Unloaded ==
MemTotal: 514428 kB
MemFree: 452716 kB
Buffers: 9964 kB
Cached: 22224 kB
SwapCached: 4640 kB
Active: 16248 kB
Inactive: 29004 kB
Active(anon): 2340 kB
Inactive(anon): 10756 kB
Active(file): 13908 kB
Inactive(file): 18248 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 514428 kB
LowFree: 452716 kB
SwapTotal: 1508344 kB
SwapFree: 1496068 kB
Dirty: 1560 kB
Writeback: 0 kB
AnonPages: 9296 kB
Mapped: 3192 kB
Shmem: 32 kB
Slab: 9472 kB
SReclaimable: 3376 kB
SUnreclaim: 6096 kB
KernelStack: 824 kB
PageTables: 768 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1765556 kB
Committed_AS: 190512 kB
VmallocTotal: 504056 kB
VmallocUsed: 7504 kB
VmallocChunk: 484316 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 16128 kB
DirectMap4M: 507904 kB
Reply to: