Re: IOWait Problem
Nabend,
Am Mittwoch, 17. Oktober 2012, 19:56:41 schrieb Martin Steigerwald:
> Das ist ziemlich schlecht zu lesen. Ja, KMailhats jetzt nochmal
> umgebrochen, ist aber für meine Antwort egal.
>
> Ich bitte Dich, diese Angaben ohne Zeilenumbruch zu senden.
hab sie einfach mal als text-attachment angehangen.
Roundcube konnte die format=flowed Mail problemlos darstellen.
KMail nutze ich seit dem akonadi Wahnsinn kaum noch ;)
> Zusätzlich fehlen eine ganze Reihe Angaben zum System selbst. Wie z.B. wo
> es die Daten speichert, Hauptspeichermenge, CPU usw. Ja teilweise läßt
> sich das aus den Sysstat-Daten rauslesen. Ich bitte dich aber, das ganze
> etwas netter aufzubereiten, wenn du hier schon um kostenlosen Support für
> einen Unternehmens-Server bittest.
- 8x Intel(R) Xeon(R) CPU X5365 @ 3.00GHz"
- 16GB Ram
- 10k 72GB Disks im HW-Raid1 (hier müsste ich den Admin fragen, bin eigentlich
nur für die Software im JBoss zuständig)
> Auch fehlt ein Blick auf die Prozesse.
auf der Kiste läuft sonst nix relevantes: ssh/ntpd/nagios-statd/nfs-
client/bacula-fd. Alles keine Speicherfresser....
mal aktuelle Daten um eine Vorstellung zu bekommen:
# ps -eo cputime,etime,pmem,rss,sz,vsz,comm | grep -v "^00:00"
TIME ELAPSED %MEM RSS SZ VSZ COMMAND
00:01:47 108-11:24:18 0.0 0 0 0 events/7
00:09:27 108-11:24:18 0.0 0 0 0 kswapd0
01:52:20 108-11:24:06 0.0 0 0 0 kjournald
00:01:07 108-11:24:05 0.0 0 0 0 edac-poller
00:06:01 108-11:24:03 0.0 0 0 0 flush-8:0
00:03:19 108-11:22:39 0.0 0 0 0 bond0
00:01:00 108-11:22:37 0.0 6092 8391 33564 nagios-statd
00:03:08 108-11:22:37 0.0 1024 8533 34132 ntpd
00:49:26 108-11:22:36 0.0 3384 38131 152524 bacula-fd
00:01:17 36-04:50:10 0.0 1116 29929 119716 rsyslogd
00:01:17 23-04:35:53 0.0 7236 11091 44364 munin-node
01:29:56 07:08:49 65.2 10747944 4430036 17720144 java
Der gesamte Speicher ist für den JBoss, der deshalb auch mit reservierten
12,5GB Heap und 512MB PermGen gestartet wird.
iotop und co. wurde bereits herangezogen. der jboss erzeugt keine große io-
Last (die lokale Platte enthält quasi keine Laufzeitdaten die cache-relevant
sind).
Man sieht nur kernel-threads mit hoher io-Last. (falls es ohne Swap nochmal
auftritt protokolliere ich das)
>
> Für einen recht netten Überblick empfehle ich atop zu verwenden. Das kann
> I/O nach Prozess anzeigen.
>
> Interessanterweise gleich dann, wenn das Problem auftritt.
> Laut
>
> > 14:05:01 kbmemfree kbmemused %memused
> > kbbuffers kbcached kbcommit %commit
> > 14:15:01 107832 16363460 99,35 162796 809300 14976404
> >
> > 77,12
> ist die Speicherauslastung ziemlich hoch, wenn ich mich da nicht in den
> Spalten vertan hab. Und das, was für Caches bereit steht, ziemlich
> niedrig.
>
> Mich wundert das nicht, dass die Kiste lahm ist. Und das Swap würde ich
> auch nicht ausschalten.
800MB cache und 160MB buffers-cache sind in meinen Augen für die Kiste
eigentlich ausreichend. Wie gesagt da liegen keine relevanten Daten auf der
Platte die der JBoss braucht, die holt er aus dem Alfresco bzw. Postgresql.
Der Rechner hat auch in der restlichen Zeit keine nennenswerte IO-Last.
MfG
msc
Linux 2.6.32-5-amd64 (theodosius-ii) 10/17/2012 _x86_64_ (8 CPU)
02:05:01 PM CPU %user %nice %system %iowait %steal %idle
02:15:01 PM all 6.40 0.00 0.54 2.04 0.00 91.02
02:25:01 PM all 4.91 0.00 0.43 1.61 0.00 93.05
02:35:01 PM all 5.21 0.00 0.48 1.80 0.00 92.51
02:45:01 PM all 4.45 0.00 0.41 1.29 0.00 93.85
02:55:01 PM all 4.31 0.00 0.44 1.52 0.00 93.74
03:05:01 PM all 4.88 0.00 0.43 1.70 0.00 92.99
03:15:01 PM all 4.57 0.00 0.44 1.60 0.00 93.39
03:25:01 PM all 4.44 0.00 0.40 1.37 0.00 93.79
03:35:02 PM all 5.50 0.00 1.17 68.50 0.00 24.83
03:45:01 PM all 14.77 0.00 2.10 31.77 0.00 51.37
03:55:02 PM all 6.07 0.00 0.45 2.12 0.00 91.36
04:05:01 PM all 5.20 0.00 0.61 2.01 0.00 92.18
04:15:01 PM all 5.10 0.00 0.33 1.10 0.00 93.47
04:25:01 PM all 5.42 0.00 0.40 1.50 0.00 92.69
04:35:01 PM all 4.67 0.00 0.36 1.39 0.00 93.58
04:45:01 PM all 3.84 0.00 0.28 0.99 0.00 94.88
04:55:01 PM all 4.93 0.00 0.36 1.61 0.00 93.10
05:05:01 PM all 4.77 0.00 0.40 1.80 0.00 93.03
05:15:01 PM all 4.63 0.00 0.33 1.32 0.00 93.73
05:25:01 PM all 3.26 0.00 0.28 1.33 0.00 95.14
05:35:01 PM all 2.69 0.00 0.33 2.18 0.00 94.80
05:45:01 PM all 2.87 0.00 0.20 0.68 0.00 96.24
05:55:01 PM all 3.09 0.00 0.28 1.44 0.00 95.19
Average: all 5.04 0.00 0.50 5.77 0.00 88.69
02:05:01 PM proc/s cswch/s
02:15:01 PM 46.38 1651.56
02:25:01 PM 37.51 1425.94
02:35:01 PM 37.41 1453.27
02:45:01 PM 32.90 1288.79
02:55:01 PM 34.60 1364.16
03:05:01 PM 35.51 1408.11
03:15:01 PM 33.93 1314.47
03:25:01 PM 30.69 1258.77
03:35:02 PM 113.05 4081.97
03:45:01 PM 303.83 9054.88
03:55:02 PM 37.85 1534.63
04:05:01 PM 55.41 1729.72
04:15:01 PM 32.62 1333.07
04:25:01 PM 42.26 1609.79
04:35:01 PM 39.21 1462.45
04:45:01 PM 30.25 1173.36
04:55:01 PM 40.18 1510.24
05:05:01 PM 44.07 1658.46
05:15:01 PM 36.01 1345.88
05:25:01 PM 30.44 1273.61
05:35:01 PM 36.94 1670.64
05:45:01 PM 19.90 904.17
05:55:01 PM 29.40 1332.94
Average: 51.31 1862.52
02:05:01 PM pswpin/s pswpout/s
02:15:01 PM 0.00 0.00
02:25:01 PM 0.00 0.00
02:35:01 PM 0.00 0.00
02:45:01 PM 0.00 0.00
02:55:01 PM 0.00 0.00
03:05:01 PM 0.00 0.00
03:15:01 PM 0.01 0.00
03:25:01 PM 0.00 0.00
03:35:02 PM 0.01 0.00
03:45:01 PM 0.38 0.03
03:55:02 PM 0.01 0.00
04:05:01 PM 0.17 0.00
04:15:01 PM 0.00 0.00
04:25:01 PM 0.00 0.00
04:35:01 PM 0.00 0.00
04:45:01 PM 0.00 0.00
04:55:01 PM 0.00 0.00
05:05:01 PM 0.00 0.00
05:15:01 PM 0.00 0.00
05:25:01 PM 0.00 0.00
05:35:01 PM 0.00 0.00
05:45:01 PM 0.08 0.00
05:55:01 PM 0.00 0.00
Average: 0.03 0.00
02:05:01 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
02:15:01 PM 3.97 499.94 1343.18 0.01 1050.67 22.93 0.00 21.76 94.88
02:25:01 PM 3.28 413.38 1267.09 0.00 939.84 28.80 0.00 27.52 95.56
02:35:01 PM 3.25 408.05 1244.56 0.01 943.34 32.14 0.00 31.38 97.66
02:45:01 PM 4.47 357.73 1207.43 0.00 849.99 0.00 0.00 0.00 0.00
02:55:01 PM 2.87 380.97 1238.77 0.00 896.71 14.40 0.00 13.81 95.93
03:05:01 PM 7.51 398.05 1243.23 0.01 931.45 22.03 0.00 20.91 94.92
03:15:01 PM 4.94 397.03 1231.90 0.05 894.47 20.53 0.00 19.53 95.11
03:25:01 PM 9.25 349.27 1256.68 0.10 925.90 47.26 0.00 46.50 98.40
03:35:02 PM 2674.15 591.19 1765.48 0.27 2204.80 604.31 25.38 611.91 97.18
03:45:01 PM 1357.10 1297.74 6129.43 0.84 10295.62 136.42 131.22 245.03 91.55
03:55:02 PM 154.44 769.30 3598.76 0.76 1724.55 0.00 0.00 0.00 0.00
04:05:01 PM 82.69 774.84 7421.37 0.37 2894.61 0.00 0.00 0.00 0.00
04:15:01 PM 22.62 301.03 1439.21 0.00 882.90 0.00 0.00 0.00 0.00
04:25:01 PM 6.10 376.54 1556.68 0.00 1032.10 0.00 0.00 0.00 0.00
04:35:01 PM 6.00 347.75 1433.84 0.00 933.25 0.00 0.00 0.00 0.00
04:45:01 PM 5.57 274.74 1303.60 0.00 849.14 0.00 0.00 0.00 0.00
04:55:01 PM 5.73 390.06 1339.76 0.00 1000.01 0.00 0.00 0.00 0.00
05:05:01 PM 6.23 458.28 1442.22 0.00 995.62 0.00 0.00 0.00 0.00
05:15:01 PM 3.27 387.65 1427.55 0.00 932.74 0.00 0.00 0.00 0.00
05:25:01 PM 1.25 343.51 1371.31 0.00 852.13 0.00 0.00 0.00 0.00
05:35:01 PM 0.62 427.99 1332.02 0.00 908.71 0.00 0.00 0.00 0.00
05:45:01 PM 50.62 228.54 1154.15 0.05 696.13 0.00 0.00 0.00 0.00
05:55:01 PM 4.81 348.66 1256.96 0.03 836.50 0.00 0.00 0.00 0.00
Average: 192.41 457.44 1912.65 0.11 1498.28 40.44 6.80 45.20 95.67
02:05:01 PM tps rtps wtps bread/s bwrtn/s
02:15:01 PM 27.17 0.42 26.74 7.93 999.88
02:25:01 PM 24.17 0.31 23.86 6.56 826.76
02:35:01 PM 24.26 0.32 23.94 6.49 816.11
02:45:01 PM 20.17 0.39 19.77 8.95 715.44
02:55:01 PM 21.84 0.25 21.59 5.73 761.96
03:05:01 PM 23.09 0.55 22.54 15.01 796.09
03:15:01 PM 22.41 0.30 22.11 9.88 794.07
03:25:01 PM 21.31 1.11 20.20 18.51 698.54
03:35:02 PM 246.21 221.52 24.69 5346.79 1158.73
03:45:01 PM 157.22 126.51 30.71 2715.72 2619.21
03:55:02 PM 34.82 11.45 23.37 308.88 1538.62
04:05:01 PM 32.02 7.49 24.53 165.39 1549.68
04:15:01 PM 15.85 1.21 14.64 45.24 602.06
04:25:01 PM 19.67 0.62 19.05 12.20 753.08
04:35:01 PM 17.98 0.46 17.52 12.00 695.50
04:45:01 PM 14.96 1.11 13.86 11.13 549.46
04:55:01 PM 20.74 0.69 20.04 11.48 780.13
05:05:01 PM 23.71 0.53 23.17 12.45 916.57
05:15:01 PM 19.31 0.27 19.04 6.55 775.29
05:25:01 PM 18.56 0.13 18.43 2.51 687.02
05:35:01 PM 27.28 0.08 27.20 1.24 855.99
05:45:01 PM 13.51 1.71 11.79 101.24 457.06
05:55:01 PM 19.96 0.22 19.74 9.61 697.32
Average: 37.68 16.44 21.24 384.82 914.88
02:05:01 PM frmpg/s bufpg/s campg/s
02:15:01 PM -5.66 0.79 4.56
02:25:01 PM 8.23 -0.08 -8.98
02:35:01 PM 16.31 -1.15 -12.39
02:45:01 PM -20.63 0.82 18.01
02:55:01 PM -4.17 0.44 3.88
03:05:01 PM 3.62 0.20 -3.25
03:15:01 PM -1.13 0.24 0.44
03:25:01 PM 27.87 -0.74 -26.36
03:35:02 PM -39.96 -9.14 67.36
03:45:01 PM 3499.04 14.01 -92.16
03:55:02 PM -349.87 -70.41 136.57
04:05:01 PM -274.57 7.25 88.38
04:15:01 PM -236.99 6.72 21.92
04:25:01 PM -212.99 7.59 21.71
04:35:01 PM -172.50 5.56 17.90
04:45:01 PM -103.47 4.66 15.71
04:55:01 PM -25.85 4.97 18.53
05:05:01 PM -143.29 4.49 24.05
05:15:01 PM -171.62 2.92 18.15
05:25:01 PM -144.62 2.51 14.38
05:35:01 PM -78.76 3.37 10.56
05:45:01 PM -71.48 1.10 22.96
05:55:01 PM -44.08 1.75 18.54
Average: 63.03 -0.53 16.55
02:05:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
02:15:01 PM 107832 16363460 99.35 162796 809300 14976404 77.12
02:25:01 PM 127584 16343708 99.23 162600 787740 14967640 77.08
02:35:01 PM 166724 16304568 98.99 159832 758008 14942380 76.95
02:45:01 PM 117208 16354084 99.29 161812 801228 14996204 77.22
02:55:01 PM 107192 16364100 99.35 162880 810552 14956200 77.02
03:05:01 PM 115892 16355400 99.30 163356 802748 14956892 77.02
03:15:01 PM 113180 16358112 99.31 163928 803812 14981480 77.15
03:25:01 PM 180080 16291212 98.91 162152 740552 14962260 77.05
03:35:02 PM 83948 16387344 99.49 140164 902612 15022744 77.36
03:45:01 PM 8472136 7999156 48.56 173760 681680 13902536 71.59
03:55:02 PM 7632376 8838916 53.66 4756 1009464 14173556 72.99
04:05:01 PM 6974480 9496812 57.66 22136 1221228 14177900 73.01
04:15:01 PM 6405676 10065616 61.11 38268 1273848 14174000 72.99
04:25:01 PM 5894472 10576820 64.21 56492 1325956 14195320 73.10
04:35:01 PM 5480476 10990816 66.73 69844 1368920 14227872 73.27
04:45:01 PM 5232140 11239152 68.23 81020 1406624 14215960 73.21
04:55:01 PM 5170088 11301204 68.61 92940 1451092 14292916 73.60
05:05:01 PM 4826172 11645120 70.70 103712 1508816 14281828 73.54
05:15:01 PM 4414272 12057020 73.20 110728 1552372 14255244 73.41
05:25:01 PM 4067184 12404108 75.31 116752 1586884 14242796 73.34
05:35:01 PM 3878148 12593144 76.46 124836 1612240 14255244 73.41
05:45:01 PM 3706588 12764704 77.50 127488 1667336 14268256 73.48
05:55:01 PM 3600796 12870496 78.14 131684 1711840 14324320 73.76
Average: 3342376 13128916 79.71 117128 1156298 14510867 74.72
02:05:01 PM kbswpfree kbswpused %swpused kbswpcad %swpcad
02:15:01 PM 2928824 19064 0.65 7196 37.75
02:25:01 PM 2928824 19064 0.65 7196 37.75
02:35:01 PM 2928824 19064 0.65 7196 37.75
02:45:01 PM 2928824 19064 0.65 7196 37.75
02:55:01 PM 2928824 19064 0.65 7196 37.75
03:05:01 PM 2928824 19064 0.65 7188 37.70
03:15:01 PM 2928824 19064 0.65 7068 37.08
03:25:01 PM 2928824 19064 0.65 7068 37.08
03:35:02 PM 2928824 19064 0.65 7104 37.26
03:45:01 PM 2931284 16604 0.56 7036 42.38
03:55:02 PM 2931296 16592 0.56 7040 42.43
04:05:01 PM 2931460 16428 0.56 7292 44.39
04:15:01 PM 2931460 16428 0.56 7292 44.39
04:25:01 PM 2931460 16428 0.56 7292 44.39
04:35:01 PM 2931460 16428 0.56 7292 44.39
04:45:01 PM 2931460 16428 0.56 7292 44.39
04:55:01 PM 2931460 16428 0.56 7292 44.39
05:05:01 PM 2931460 16428 0.56 7292 44.39
05:15:01 PM 2931460 16428 0.56 7292 44.39
05:25:01 PM 2931460 16428 0.56 7292 44.39
05:35:01 PM 2931460 16428 0.56 7292 44.39
05:45:01 PM 2931540 16348 0.55 7404 45.29
05:55:01 PM 2931540 16348 0.55 7404 45.29
Average: 2930421 17467 0.59 7226 41.37
02:05:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
02:15:01 PM 0 924 0.78 0.71 0.61
02:25:01 PM 1 935 0.36 0.46 0.54
02:35:01 PM 1 894 0.24 0.46 0.51
02:45:01 PM 0 977 0.64 0.48 0.49
02:55:01 PM 0 939 0.55 0.42 0.43
03:05:01 PM 0 907 0.49 0.68 0.55
03:15:01 PM 3 933 0.75 0.56 0.52
03:25:01 PM 0 943 0.21 0.33 0.43
03:35:02 PM 1 992 119.50 99.27 51.16
03:45:01 PM 2 673 1.60 35.23 46.08
03:55:02 PM 1 840 0.97 5.45 24.48
04:05:01 PM 1 843 0.39 1.12 13.11
04:15:01 PM 2 814 0.48 0.58 7.08
04:25:01 PM 0 827 0.25 0.46 3.95
04:35:01 PM 0 845 0.70 0.71 2.38
04:45:01 PM 0 847 0.28 0.31 1.36
04:55:01 PM 1 903 0.49 0.44 0.90
05:05:01 PM 1 902 0.49 0.58 0.75
05:15:01 PM 2 864 0.37 0.46 0.60
05:25:01 PM 0 864 0.25 0.31 0.46
05:35:01 PM 0 877 0.61 0.64 0.52
05:45:01 PM 1 896 0.10 0.28 0.38
05:55:01 PM 0 895 0.40 0.31 0.34
Average: 1 884 5.69 6.53 6.85
02:05:01 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
02:15:01 PM sda 27.17 7.93 999.88 37.10 5.26 193.43 6.16 16.72
02:25:01 PM sda 24.17 6.56 826.76 34.47 4.19 173.33 6.21 15.01
02:35:01 PM sda 24.26 6.49 816.11 33.91 4.71 194.30 6.26 15.18
02:45:01 PM sda 20.17 8.95 715.44 35.92 3.48 172.44 6.13 12.36
02:55:01 PM sda 21.84 5.73 761.96 35.15 4.01 183.44 6.27 13.69
03:05:01 PM sda 23.09 15.01 796.09 35.12 4.25 184.25 6.27 14.48
03:15:01 PM sda 22.41 9.88 794.07 35.88 4.43 197.53 6.25 14.01
03:25:01 PM sda 21.31 18.51 698.54 33.65 3.71 174.15 6.20 13.22
03:35:02 PM sda 246.21 5346.79 1158.73 26.42 140.06 561.76 3.95 97.26
03:45:01 PM sda 157.22 2715.72 2619.21 33.93 92.53 599.35 4.02 63.17
03:55:02 PM sda 34.82 308.88 1538.62 53.06 3.54 101.56 5.01 17.45
04:05:01 PM sda 32.02 165.39 1549.68 53.56 2.61 81.54 5.50 17.61
04:15:01 PM sda 15.85 45.24 602.06 40.84 0.79 49.72 6.88 10.91
04:25:01 PM sda 19.67 12.20 753.08 38.91 1.04 53.09 7.13 14.02
04:35:01 PM sda 17.98 12.00 695.50 39.35 1.03 57.49 7.15 12.86
04:45:01 PM sda 14.96 11.13 549.46 37.46 1.05 70.07 6.71 10.05
04:55:01 PM sda 20.74 11.48 780.13 38.18 2.19 105.62 7.00 14.51
05:05:01 PM sda 23.71 12.45 916.57 39.19 2.95 124.62 6.90 16.36
05:15:01 PM sda 19.31 6.55 775.29 40.48 2.60 134.66 6.68 12.91
05:25:01 PM sda 18.56 2.51 687.02 37.15 2.13 114.93 6.88 12.77
05:35:01 PM sda 27.28 1.24 855.99 31.42 2.92 106.84 7.16 19.53
05:45:01 PM sda 13.51 101.24 457.06 41.34 1.50 111.07 5.76 7.78
05:55:01 PM sda 19.96 9.61 697.32 35.42 2.35 117.60 6.82 13.62
Average: sda 37.68 384.82 914.88 34.50 12.76 338.68 5.26 19.81
Reply to: