Re: sys cpu usage

[Michael Loftis <mloftis@modwest.com>]

--On August 26, 2008 8:14:50 PM +0200 Marek Podmaka 
<marki@marki-online.net> wrote:

> Hello,
>
> Tuesday, August 26, 2008, 17:18:01, Michael Loftis wrote:
> > their primary filesystem is on NFS, the NICs do UDP/TCP checksum
> > offloading, and they mostly push large packets I see up to about 15%
> > or so on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so
> > four CPUs). But that's peak, average is more like 7-8% at around 70
> > hits/second, and we've a pretty high overhead too. So 25%+ sounds
>
> How is TCP checksum offload affecting the SYS usage? It is the chip on
> network card which is doing the stuff and not kernel, right? Or does
> it show in the sys cpu usage? (On this server, only checksumming
> offload is supported.)

Depends on what is supported on your card, use ethtool -k <dev> to find out 
and to change settings.  On our systems we're doing checksum and TSO.  The 
offload doesn't show at all in the CPU usage stats.  The CPU basically 
skips computing checksums, and for TSO, segmentation, and just sends big 
packets to the DMA ring buffers, the card then picks it up, does any 
segmentation (replicating/affixing IP headers), computes checksums, and 
pushes it out over the wire.  This frees up a decent amount of CPU at high 
packet rates or high line rates.  The result is lower %sys CPU, thus more 
CPU available for userspace (and system) operations.  UDP checksum 
offloading is actually pretty important for us since we're using NFS as 
primary storage.

What kernel are you running BTW?  And debian version?  Kernels prior to 2.6 
didn't do well on high interrupt loads P4, Xeon, and Core2's because of 
large stack frames.  Somewhere in 2.6 they made a change that allows the 
kernel to export an optimal path for entering/leaving the kernel routines 
without necessarily saving the entire stack frame (sysenter/sysexit calling 
convention), I can't recall if that shows up as %sys, or just high overall 
CPU usage.  Try using vmstat too, it'll tell you your number of context 
switches.

Someone else mentioned oprofile to profile kernel CPU usage too.

> Tuesday, August 26, 2008, 17:48:35, blorb wrote:
> > well, if it's within the mysql, try mytop but if it's something
> > within the apahce or php i'm not sure myself...
> > but try logging while working
>
> Tuesday, August 26, 2008, 17:52:00, Joel Merrick wrote:
> > Take a look at apache-top
> > You need the server-status page enabled and it has to be apache 2.
> > Also the mysql slow query log can be useful
>
> Thanks for the tips, but I'm already using all this (mytop, slow log
> and apache's status page) for performance tuning, but that wouldn't
> help in case I need to track the system cpu usage.
>
>
> Tuesday, August 26, 2008, 18:51:43, Marcin Sochacki wrote:
> > oprofile is a Linux tool for kernel profiling.
> > Regarding the original issue with high load, it's worth noting that USER
> > CPU usage is also very high and corresponding to the SYS usage. I think
> > that simply the web application you're using is not optimised enough, or
> > you just have a lot of traffic.
>
> Oh yes, I was playing with oprofile some time ago, so I will try it.
>
> The total CPU load is high now and mysql will be migrated to new
> server next month. I just published the graph so that someone will be
> able to tell if the SYS usage is too high compared to USER or not. And
> yes, there's lot of traffic - about 40 hits/s for apache/php and another
> 100-120 hits/s for images using lighttpd (but 95% of them are served
> from mem-cache.
>
>
> --
>   bYE, Marki
>
>
> --
> To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster@lists.debian.org



--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting