Bug#485413: RAM full, swap full: problem still exists
I am just encountering the same problem on my current server.
About the machine:
model name : AMD Athlon(tm) 64 Processor 3700+
cpu MHz : 2199.541
cache size : 1024 KB
MemTotal: 1025708 kB
cat /etc/debian_version: 6.0.7
Apache packages:
ii apache2 2.2.16-6+squeeze11
ii apache2-mpm-prefork 2.2.16-6+squeeze11
ii apache2-utils 2.2.16-6+squeeze11
ii apache2.2-bin 2.2.16-6+squeeze11
ii apache2.2-common 2.2.16-6+squeeze11
PHP5 packages:
rc libapache2-mod-php5 5.3.3-7+squeeze15
ii libapache2-mod-php5filter 5.3.3-7+squeeze15
ii php5-cli 5.3.3-7+squeeze15
ii php5-common 5.3.3-7+squeeze15
ii php5-curl 5.3.3-7+squeeze15
ii php5-dev 5.3.3-7+squeeze15
ii php5-gd 5.3.3-7+squeeze15
ii php5-mcrypt 5.3.3-7+squeeze15
ii php5-mysql 5.3.3-7+squeeze15
PHP caching: eaccelerator-42067ac
Web and PHP applications:
3 x WordPress
1 x MediaWiki
1 x Etherpad lite
1 x status.net
I let that system send me a status mail with some outputs every 15
minutes. The last one before the last crash said:
cat /proc/meminfo:
MemTotal: 1025708 kB
MemFree: 8184 kB
Buffers: 1272 kB
Cached: 31704 kB
SwapCached: 278060 kB
Active: 473396 kB
Inactive: 471092 kB
Active(anon): 455708 kB
Inactive(anon): 455796 kB
Active(file): 17688 kB
Inactive(file): 15296 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2000052 kB
SwapFree: 854820 kB
Dirty: 0 kB
Writeback: 1040 kB
AnonPages: 639924 kB
Mapped: 12632 kB
Shmem: 20 kB
Slab: 19988 kB
SReclaimable: 8800 kB
SUnreclaim: 11188 kB
KernelStack: 1944 kB
PageTables: 28516 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2512904 kB
Committed_AS: 2688324 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 8124 kB
VmallocChunk: 34359723104 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 9152 kB
DirectMap2M: 1038336 kB
Uptime:
15:31:36 up 3 days, 23:08, 0 users, load average: 58.61, 58.42, 57.90
Parts from ps fax:
PID TTY STAT TIME COMMAND
354 ? S<s 0:00 udevd --daemon
1559 ? S< 0:00 \_ udevd --daemon
1560 ? S< 0:00 \_ udevd --daemon
938 ? Sl 0:04 /usr/sbin/rsyslogd -c4
981 ? Ss 0:11 /usr/sbin/apache2 -k start
13140 ? D 0:08 \_ /usr/sbin/apache2 -k start
13148 ? D 0:04 \_ /usr/sbin/apache2 -k start
13160 ? D 0:03 \_ /usr/sbin/apache2 -k start
13172 ? D 0:03 \_ /usr/sbin/apache2 -k start
13179 ? D 0:02 \_ /usr/sbin/apache2 -k start
13180 ? D 0:02 \_ /usr/sbin/apache2 -k start
13181 ? D 0:02 \_ /usr/sbin/apache2 -k start
13182 ? D 0:02 \_ /usr/sbin/apache2 -k start
13183 ? D 0:02 \_ /usr/sbin/apache2 -k start
13184 ? D 0:02 \_ /usr/sbin/apache2 -k start
13185 ? D 0:02 \_ /usr/sbin/apache2 -k start
13186 ? D 0:02 \_ /usr/sbin/apache2 -k start
13187 ? D 0:02 \_ /usr/sbin/apache2 -k start
13188 ? S 0:02 \_ /usr/sbin/apache2 -k start
13189 ? D 0:02 \_ /usr/sbin/apache2 -k start
13190 ? D 0:02 \_ /usr/sbin/apache2 -k start
13191 ? S 0:02 \_ /usr/sbin/apache2 -k start
13192 ? D 0:02 \_ /usr/sbin/apache2 -k start
13193 ? D 0:02 \_ /usr/sbin/apache2 -k start
13200 ? D 0:02 \_ /usr/sbin/apache2 -k start
13201 ? D 0:02 \_ /usr/sbin/apache2 -k start
13202 ? D 0:02 \_ /usr/sbin/apache2 -k start
13203 ? D 0:02 \_ /usr/sbin/apache2 -k start
13204 ? D 0:02 \_ /usr/sbin/apache2 -k start
13205 ? S 0:02 \_ /usr/sbin/apache2 -k start
13206 ? D 0:02 \_ /usr/sbin/apache2 -k start
13207 ? S 0:02 \_ /usr/sbin/apache2 -k start
13208 ? D 0:02 \_ /usr/sbin/apache2 -k start
13209 ? D 0:02 \_ /usr/sbin/apache2 -k start
13210 ? D 0:02 \_ /usr/sbin/apache2 -k start
13211 ? D 0:02 \_ /usr/sbin/apache2 -k start
13212 ? D 0:02 \_ /usr/sbin/apache2 -k start
13213 ? S 0:02 \_ /usr/sbin/apache2 -k start
13214 ? D 0:02 \_ /usr/sbin/apache2 -k start
13215 ? S 0:02 \_ /usr/sbin/apache2 -k start
13228 ? D 0:02 \_ /usr/sbin/apache2 -k start
13232 ? D 0:02 \_ /usr/sbin/apache2 -k start
13233 ? D 0:02 \_ /usr/sbin/apache2 -k start
13236 ? D 0:02 \_ /usr/sbin/apache2 -k start
13237 ? D 0:02 \_ /usr/sbin/apache2 -k start
13238 ? D 0:02 \_ /usr/sbin/apache2 -k start
13239 ? D 0:02 \_ /usr/sbin/apache2 -k start
13240 ? D 0:02 \_ /usr/sbin/apache2 -k start
13241 ? D 0:02 \_ /usr/sbin/apache2 -k start
13242 ? D 0:02 \_ /usr/sbin/apache2 -k start
13253 ? D 0:02 \_ /usr/sbin/apache2 -k start
13256 ? D 0:02 \_ /usr/sbin/apache2 -k start
13257 ? D 0:02 \_ /usr/sbin/apache2 -k start
13259 ? D 0:02 \_ /usr/sbin/apache2 -k start
13260 ? S 0:02 \_ /usr/sbin/apache2 -k start
12277 ? S 0:00 /bin/sh /usr/bin/mysqld_safe
12388 ? Sl 7:23 \_ /usr/sbin/mysqld --basedir=/usr
--datadir=/var/lib/mysql --user=mysql
--pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
12389 ? S 0:00 \_ logger -t mysqld -p daemon.error
12742 ? Sl 0:30 /usr/bin/tor --defaults-torrc
/usr/share/tor/tor-service-defaults-torrc --hush
12766 ? S 0:00 /bin/sh
/home/etherpad-lite/pad/etherpad-lite/bin/safeRun.sh
/var/log/etherpad-lite/etherpad-lite.log
12768 ? S 0:00 \_ /bin/sh bin/run.sh
12791 ? Dl 0:15 \_ node node_modules/ep_etherpad-lite
Output of this script (I called it „whatswaps.sh“):
http://northernmost.org/blog/find-out-what-is-using-your-swap/index.html
PID= 1 - Swap used: 108 - (init )
PID= 2 - Swap used: 0 - (kthreadd )
PID= 3 - Swap used: 0 - (migration/0 )
PID= 4 - Swap used: 0 - (ksoftirqd/0 )
PID= 5 - Swap used: 0 - (watchdog/0 )
PID= 20 - Swap used: 0 - (kondemand/0 )
PID= 23 - Swap used: 0 - (khungtaskd )
PID= 24 - Swap used: 0 - (kswapd0 )
PID= 25 - Swap used: 0 - (ksmd )
PID= 26 - Swap used: 0 - (aio/0 )
PID= 27 - Swap used: 0 - (crypto/0 )
PID= 208 - Swap used: 0 - (ata/0 )
PID= 762 - Swap used: 0 - (kjournald )
PID= 763 - Swap used: 0 - (kjournald )
PID= 764 - Swap used: 0 - (kjournald )
PID= 765 - Swap used: 0 - (kjournald )
PID= 766 - Swap used: 0 - (kjournald )
PID=12791 - Swap used: 14880 - (node )
PID=13140 - Swap used: 28596 - (apache2 )
PID=13148 - Swap used: 21208 - (apache2 )
PID=13160 - Swap used: 24972 - (apache2 )
PID=13172 - Swap used: 26560 - (apache2 )
PID=13179 - Swap used: 23452 - (apache2 )
PID=13180 - Swap used: 21548 - (apache2 )
PID=13181 - Swap used: 21868 - (apache2 )
PID=13182 - Swap used: 20056 - (apache2 )
PID=13183 - Swap used: 21132 - (apache2 )
PID=13184 - Swap used: 24016 - (apache2 )
PID=13185 - Swap used: 22820 - (apache2 )
PID=13186 - Swap used: 26772 - (apache2 )
PID=13187 - Swap used: 26260 - (apache2 )
PID=13188 - Swap used: 35144 - (apache2 )
PID=13189 - Swap used: 23172 - (apache2 )
PID=13190 - Swap used: 22472 - (apache2 )
PID=13191 - Swap used: 27792 - (apache2 )
PID=13192 - Swap used: 20328 - (apache2 )
PID=13200 - Swap used: 22056 - (apache2 )
PID=13201 - Swap used: 23860 - (apache2 )
PID=13202 - Swap used: 19908 - (apache2 )
PID=13203 - Swap used: 23300 - (apache2 )
PID=13204 - Swap used: 23496 - (apache2 )
PID=13205 - Swap used: 27804 - (apache2 )
PID=13206 - Swap used: 21108 - (apache2 )
PID=13207 - Swap used: 35468 - (apache2 )
PID=13208 - Swap used: 25552 - (apache2 )
PID=13209 - Swap used: 25700 - (apache2 )
PID=15329 - Swap used: 0 - ( )
PID=15331 - Swap used: 0 - ( )
PID=20704 - Swap used: 604 - (icecast2 )
PID=20718 - Swap used: 308 - (dovecot )
PID=20721 - Swap used: 736 - (dovecot-auth )
PID=21927 - Swap used: 20624 - (eggdrop )
Overall swap used: 1342904
From apache2.conf:
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 3
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 500
</IfModule>
As it seems it will happen again I ask for advice which data to obtain
before I stop the apache2 process. Load of 58 is heavy, but although it
sometimes needs up to half an hour, I can still get in when it happens.
There is one connection I see: It didn't happen until the last kernel
update in Debian Squeeze. That machine is running since 10/2010 and
never saw that problem before.
--
http://www.atari-frosch.de/ | http://blog.atari-frosch.de/
PGP encryption welcome! Key-ID: 0xCC0AEF3E @ usual places.
Und außerdem bin ich der Meinung, daß Deutschland kein Rechtsstaat ist.
Manchmal möchte ich 'ne Bombe sein und einfach explodier'n ...
Reply to: