Re: Finding the Bottleneck
Hi Bertrand,
Thanks for your insightful email!
Just so you know, this server is an:
AMD K6-2 500Mhz, 128M-133Mhz, 2 UDMA100 drives (IBM), 10M bandwidth.
The server runs Apache, Qmail, vpopmail (for pop3). The webserver is not
primary (doesn't have to have the fastest response time), as this is
mainly for the mailing lists. The 2 hard disks are on 2 different IDE
channels, as putting both disks on the same cable would drastically reduce
performance of both disks.
The way it is organized is that the mail spool/queue is on the 2nd disk,
while the OS and programs are on disk 1. Logging is also performed on disk
1, so that writing to the mail log won't interfere with the mail queue (as
they commonly both occur simultaneously).
>From MY understanding, the "load average" shows how many programs are
running, and not really how "stressed" the CPU is. I'm not sure exactly
sure how this works (please correct me if i'm wrong) but 1 program taking
80% CPU might have load average of 2, while 100 programs taking 0.5% each
would take 50% CPU and have load average of 8. Is that correct thinking?
The reason I'm saying that is because qmail spawns a program called
"qmail-remote" for EACH email to be sent. So if 200 concurrent emails are
being sent at one time, then 200 qmail-remotes are created. Granted...
each of these takes up tiny amounts of ram and CPU time, but I suspect (if
the above statements are correct) that the load average would be
artificially inflated because of it.
We don't use NFS on this server. NFS on linux, as you said, is pretty
crummy and should be avoided if possible. We simply put the mail queue on
a seperate hard disk.
pop3 load is extremely minimal. Its mainly an outgoing mail server (mail
list server). People essentially use the website to send mail to the
mailing list, so load of the pop3 server won't be an issue.
About the HK job market, do you have any official qualifications (eg. a
degree, diploma, cert., etc.)? In HK bosses like to see that... even more
than in some other countries like Australia (not sure bout US).
BTW. was your mother headmistress of St. Paul before?
Sincerely,
Jason
----- Original Message -----
From: "schemerz" <schemerz@localhost.penguinpowered.com>
To: "Jason Lim" <maillist@jasonlim.com>
Sent: Wednesday, June 06, 2001 3:57 PM
Subject: Re: Finding the Bottleneck
On Wed, Jun 06, 2001 at 11:53:22AM +0800, Jason Lim wrote:
> Hi all,
>
> I was wondering if there is a way to find out what/where the bottleneck
of
> a large mail server is.
>
> A client is running a huge mail server that we set up for them (running
> qmail), but performance seems to be limited somewhere. Qmail has already
> been optimized as far as it can go (big-todo patch, large concurrency
> patch, etc.).
>
> We're thinking one of the Hard Disks may be the main bottleneck (the
mail
> queue is already on a seperate disk on a seperate IDE channel from other
> disks). Is there any way to find out how "utilized" the IDE channel/hard
> disk is, or how hard it is going? Seems that right now the only way we
> really know is by looking at the light on the server case (how technical
> ;-) ). Must be a better way...
>
> The bottleneck wouldn't be bandwidth... it is definately with the
server.
> Perhaps the CPU or kernel is the bottleneck (load average: 4.84, 3.94,
> 3.88, going up to 5 or 6 during heavy mailing)? Is that normal for a
large
> mail server? We haven't run such a large mail server before (anywhere
> between 500k to 1M per day so far, increasing each day), so ANY tips and
> pointers would be greatly appreciated. We've already been playing around
> with hdparm to see if we can tweak the disks, but doesn't seem to help
> much. Maybe some cache settings we can fiddle with? Maybe the mail queue
> disk could use a different file cache setting (each email being from 1K
to
> 10K on average)?
>
> Thanks in advance!
>
> Sincerely,
> Jason
>
Jason,
I am a lurker on the list. I don't run linux anymore but recently a
friend of mine encountered a similar load problem. Granted, he was
running sendmail, but his main bottleneck wasn't the mta at all. I will
explain his situation and see if you will find any similarites with
yours...
The discussion pertains to the mailserver at sunflower.com. When my
friend took over the previous admin left the place in a mess. One of the
machines he inherited was a dual 400 mhz penta 2 with about 256 megs of
ram. It would be quite adequate for serving about 5000 accounts except...
1) The server was running on a box using the promise IDE raid
controllers. IDE for a small server would work fine, but this box was
getting hit alot. 2) This server was also the pop server. One of the
reasons of design to run pop and sendmail on the same box was because
nfs-exporting mail spools on linux was unsafe (nfs still sucks and WILL
suck for ad infinitum, BUT linux was and still is probably the worst
platform to employ nfs in a production system). So nfs exporting the mail
spools to a seperate pop3 boxes were out of the question. Another local
ISP, grapevine.net tried to do this with large sun boxes to no avail...
Because this is a cable modem ISP, people configured their pop3 clients to
pop their mail every 10 minutes or so... that times 5000... well you
get my point.
3) Sendmail was configured out of the box from redhat, which works well
for a 700 person institution. 5000 ? No...
So when my friend got the box, he was looking at loads of 5-6 on any given
day.
Now, your choice to go with qmail is indicative of a few things. First of
all you are probably nfs-exporting your maildirs, which is safe to do
instead of plain spool flat files. second of all you aren't forking one
large binary with sendmail, you are forking several smaller ones with
qmail, which is far less load with the virtual manager (you can configure
sendmail to use qmail's local delivery agent, i.e. sendmail writing to
maildirs, but you aren't using sendmail so that can't be the problem).
You are probably exporting your maildirs to another box to do pop/imap
from, so those servers can't be the problem... (if the servers are
running on the same box as the mta, check in top to see if that is the
problem. qpopper may be notorious for security flaws, but it is
internally multithreaded, so loads drop like a rock. There is nothing an
openwall patch can't fix temporaily... check www.openwall.com.) Lastly,
you might want to use scsi disks instead. Granted udma 100 is fast for
burst rates, but it is still extremely cpu intensive. My friend went with
ultra 2 disks and on tests using postal (another mail benchmarking util)
performance soared. I don't know what else to think of, but those are my
shots in the dark.
On another topic, I am from Hong Kong and am currently investigating the
job market back home to see how things are. I reside in the US until I
get my permanent residence, of which I will return to Hong Kong to work if
I can. My expertise is mostly BSD or Linux (I am flexible with BSD
flavors, not so much with linux distros, but hey...) I'm also a fluent
cantonese speaker and my english isn't shabby. I have references here and
job experience (give or take 2 years). What should I shoot for back home
if that option were to be made available to me ?
yours truly,
Bertrand Kotewall
(schemerz@usa.net... please mail me here, I don't have a static IP quite
yet)
Reply to: