Re: Memory problems

On Wed, Jun 2, 2010 at 5:26 PM, Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:

On Wed, Jun 02, 2010 at 05:06:30PM +0200, Jonatan Soto wrote:
> Hi list,
>
> I'm facing a problem with lenny regarding to memory usage.
>
> I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a
> few days and top command shows a very high amount of memory consumption for
> each server. I have a little knowledge of how linux (kernel 2.6) manages
> memory. A nice resource I found is this:
> http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm
>
> So, I understand that cached memory may be free if some application requires
> it but I don't understand why lenny is consuming 2GB of physical memory.
> It's worth to mention that all the systems are running with only the
> standard package installed and few additional daemons for each server.
>
> I post what top command shows in order to provide better clues of what's
> going on:
>
> Server1:
> top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00
> Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie
> Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers
> Swap: 578300k total, 0k used, 578300k free, 592736k cached
> additional daemons -> apache2, bind9, sshd
>
> Server2:
> top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00
> Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie
> Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers
> Swap: 915664k total, 0k used, 915664k free, 155112k cached
> additional daemons -> apache2, sshd
>
> Server3:
> top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00
> Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie
> Mem: 3097764k total, 2263200k used, 834564k free, 49152k buffers
> Swap: 2928632k total, 0k used, 2928632k free, 107700k cached
> additional daemons -> lvm, sshd
>
> Server4:
> top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00
> Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie
> Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers
> Swap: 1341388k total, 0k used, 1341388k free, 53196k cached
> additional daemons -> bind9, sldap, samba, sshd
>
> --------------------
>
> Note that I've recently rebooted Server4 and it has lower memory consumption
> rather than the others and it is running more daemons.
> May be this issue is a misconfiguration of my servers or a memory leak?
> Should I tweak something in order to improve memory management?
>
> Any help would be much appreciated.
>
> PD: Apache2 is installed using default configuration of the Apache2 official
> Debian release.

Well I would run top, hit 'M' to sort by memory usage, and see which
processes are using the ram.

Well, that's what I did and that's why I decided to consult here because I thought it might be a problem since the memory usage shown in top doesn't match the amount of memory consumption of each process.

The following corresponds to Server3, I understand that these processes cannot consume 2GB of physical memory....

12027 root      20   0 89732 3764 2988 R    0 0.1   0:00.30 sshd
22502 root      20   0 163m 2752 1560 S    0 0.1   0:42.68 nscd
3608 root      20   0 77784 2164 1668 S    0 0.1   0:00.66 login
3009 root      20   0 119m 2080 1004 S    0 0.1   0:20.07 rsyslogd
28878 root      20   0 10576 1768 1312 S    0 0.1   0:00.36 bash
12030 root      20   0 10528 1676 1264 S    0 0.1   0:00.00 bash
3280 root      20   0 23896 1436 1132 S    0 0.0   7:36.22 vmware-guestd
28889 root      20   0 42808 1144 672 S    0 0.0   0:01.12 sshd
1262 root      16 -4 16912 1124 484 S    0 0.0   0:00.04 udevd
12034 root      20   0 10624 1100 848 R    0 0.0   0:04.90 top
3553 Debian-e 20   0 42716 1016 612 S    0 0.0   0:07.40 exim4
3591 root      20   0 19804 844 652 S    0 0.0   0:05.16 cron
2793 statd     20   0 10136 760 632 S    0 0.0   0:00.00 rpc.statd
    1 root      20   0 10312 756 620 S    0 0.0   0:11.76 init
3020 root      20   0 3796 600 476 S    0 0.0   0:00.00 acpid
3616 root      20   0 3796 584 484 S    0 0.0   0:00.00 getty
3612 root      20   0 3796 580 484 S    0 0.0   0:00.00 getty
3617 root      20   0 3796 580 484 S    0 0.0   0:00.00 getty
3609 root      20   0 3796 576 484 S    0 0.0   0:00.00 getty
3613 root      20   0 3796 576 484 S    0 0.0   0:00.00 getty
2782 daemon    20   0 8020 536 416 S    0 0.0   0:00.00 portmap
3571 daemon    20   0 16356 444 296 S    0 0.0   0:02.40 atd

For example:

top - 11:22:12 up 12 days, 23:25, 11 users, load average: 1.52, 1.51, 1.67
Tasks: 257 total, 1 running, 256 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.5%us, 0.3%sy, 0.0%ni, 86.4%id, 9.8%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16473836k total, 16311892k used, 161944k free, 5624832k buffers
Swap: 16777208k total, 168k used, 16777040k free, 6619416k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8998 bacula 20 0 193m 123m 1792 S 28 0.8 1611:17 bacula-dir
21000 snmp 20 0 41976 34m 980 S 0 0.2 2:30.73 snmpd
20123 bind 20 0 117m 27m 860 S 0 0.2 0:00.22 named
20768 mysql 20 0 124m 12m 1060 S 0 0.1 1:26.30 mysqld
23350 root 20 0 34336 9988 7404 S 0 0.1 0:00.76 bat
21302 root 20 0 23564 6772 2312 S 0 0.0 0:02.66 apache2
8829 root 20 0 44808 5892 1276 S 0 0.0 69:42.16 bacula-fd
9077 www-data 20 0 23564 5728 1160 S 0 0.0 0:00.39 apache2
20085 www-data 20 0 23700 5684 1072 S 0 0.0 0:01.12 apache2
22824 www-data 20 0 23564 5684 1120 S 0 0.0 0:00.46 apache2
1128 www-data 20 0 23564 5676 1160 S 0 0.0 0:00.34 apache2
22821 www-data 20 0 23564 5676 1112 S 0 0.0 0:00.40 apache2
9079 www-data 20 0 23564 5672 1108 S 0 0.0 0:01.33 apache2
20606 www-data 20 0 23564 5668 1104 S 0 0.0 0:00.39 apache2
20468 www-data 20 0 23564 5644 1080 S 0 0.0 0:00.80 apache2
20082 www-data 20 0 23564 5640 1072 S 0 0.0 0:00.98 apache2
9078 www-data 20 0 23564 5624 1044 S 0 0.0 0:00.61 apache2

So in this case my bacula tape backup is using about 200MB, snmpd is
using 34M, bind (named) is using 27M, mysql is using 12M. not much
going on. Lots of apache2 processes each using a bit under 6M.

The RES is how much is resident in memory. VIRT is the whole process
memory space but that includes shared libraries and memory mapped files
and other things, and not just the memory used.

So I understand there's nothing to worry. I should look at field RES so the sum of all the processes is the correct physical memory usage . Is that right?

Another way to see how much memory is consuming the process could be ps -ef, but it's a percentage. Taking this value, probably matches the REST value of top command. Please correct me if I'm wrong.

Of course the kernel can also have network buffers and other things
using some memory, which won't show in top. I do expect in your case
something obvious to be using the memory though.

--
Len Sorensen