[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Finding the Bottleneck

I agree with you that splitting the mail queue to another server wouldn't
help, especially since you've seen the top results, and know that it isn't
very heavily loaded with other jobs in the first place. So I think you are
very correct in saying that the hard disk is the limit here.

Today I played around with hdparm to see if I could tweak some additional
performance out of the existing drives, and it helped by about 10% (not a
huge jump, but anything helps!).

Specifically, I set /sbin/hdparm -a4 -c3 -d1 -m16 -u1 /dev/hdc:

       -a     Get/set sector  count  for  filesystem  read-ahead.
              This  is  used to improve performance in sequential
              reads of large  files,  by  prefetching  additional
              blocks  in anticipation of them being needed by the
              running  task.   In  the  current  kernel   version
              (2.0.10)  this  has  a default setting of 8 sectors
              (4KB).  This value seems good  for  most  purposes,
              but in a system where most file accesses are random
              seeks, a smaller setting might provide better  per­
              formance.   Also, many IDE drives also have a sepa­
              rate built-in read-ahead function, which alleviates
              the need for a filesystem read-ahead in many situa­
(Since most the emails are small and randomly placed around, I thought
maybe 2KB read-ahead might make more sense. Tell me if I'm wrong...
because the performance jump may not be due to this setting)

       -c     Query/enable  (E)IDE 32-bit I/O support.  A numeric
              parameter can be used to enable/disable 32-bit  I/O
              support:  Currently  supported  values include 0 to
              disable 32-bit I/O support, 1 to enable 32-bit data
              transfers,  and  3  to enable 32-bit data transfers
              with a  special  sync  sequence  required  by  many
              chipsets.  The value 3 works with nearly all 32-bit
              IDE chipsets, but incurs  slightly  more  overhead.
              Note  that "32-bit" refers to data transfers across
              a PCI or VLB bus to the interface  card  only;  all
              (E)IDE  drives  still have only a 16-bit connection
              over the ribbon cable from the interface card.

(Couldn't hurt to have it going 32 bit rather than 16 bit)

       -d     Disable/enable the "using_dma" flag for this drive.
              This  option  only works with a few combinations of
              drives and interfaces which support DMA  and  which
              are known to the IDE driver (and with all supported
              XT interfaces).  In particular,  the  Intel  Triton
              chipset is supported for bus-mastered DMA operation
              with many drives (experimental).  It is also a good
              idea to use the -X34 option in combination with -d1
              to ensure that the drive itself is  programmed  for
              multiword  DMA mode2.  Using DMA does not necessar­
              ily provide any improvement in throughput or system
              performance,  but  many  folks  swear  by it.  Your
              mileage may vary.
(this is a dma100 7200 drive so setting this couldn't hurt either. Didn't
see much performance increase with this though)

       -m     Get/set sector count for multiple sector I/O on the
              drive.  A setting of 0 disables this feature.  Mul­
              tiple  sector  mode (aka IDE Block Mode), is a fea­
              ture of most modern IDE hard drives, permitting the
              transfer  of  multiple  sectors  per I/O interrupt,
              rather than the usual  one  sector  per  interrupt.
              When  this feature is enabled, it typically reduces
              operating system overhead for disk I/O  by  30-50%.
              On  many  systems,  it also provides increased data
              throughput  of  anywhere  from  5%  to  50%.   Some
              drives,   however   (most  notably  the  WD  Caviar

              series), seem to  run  slower  with  multiple  mode
              enabled.   Your mileage may vary.  Most drives sup­
              port the minimum settings of 2, 4, 8, or  16  (sec­
              tors).   Larger  settings  may  also  be  possible,
              depending on the drive.  A  setting  of  16  or  32
              seems  optimal  on  many  systems.  Western Digital
              recommends lower settings of 4  to  8  on  many  of
              their  drives,  due  tiny  (32kB) drive buffers and
              non-optimized buffering algorithms.   The  -i  flag
              can  be  used to find the maximum setting supported
              by an installed drive (look for MaxMultSect in  the
              output).   Some  drives  claim  to support multiple
              mode, but lose data at some settings.   Under  rare
              circumstances,  such failures can result in massive
              filesystem corruption.
(I set it to 16... do you think 32 would make more sense?)

       -u     Get/set interrupt-unmask flag  for  the  drive.   A
              setting  of  1  permits  the driver to unmask other
              interrupts during processing of a  disk  interrupt,
              which  greatly  improves Linux's responsiveness and
              eliminates "serial port overrun" errors.  Use  this
              feature  with caution: some drive/controller combi­
              nations do not tolerate the increased I/O latencies
              possible when this feature is enabled, resulting in
              massive  filesystem  corruption.   In   particular,
              CMD-640B  and RZ1000 (E)IDE interfaces can be unre­
              liable (due to a hardware flaw) when this option is
              used  with  kernel  versions  earlier  than 2.0.13.
              Disabling the IDE prefetch feature of these  inter­
              faces (usually a BIOS/CMOS setting) provides a safe
              fix for the problem for use with earlier kernels.
(this seem to have the largest performance boost)

Anyway... there it is. Maybe someone else could use these results to get a
free 10% increase as well. I stupidly set write_cache on as well, which
ended up trashing a bunch of stuff. Thank goodness at that time the server
was not being used, and I immediately rebuilt the mail queue.

Does anyone have any better configs than above, or some utility that could
further boost performance?


----- Original Message -----
From: "Russell Coker" <russell@coker.com.au>
To: "Jason Lim" <maillist@jasonlim.com>; "Brian May"
Cc: <debian-isp@lists.debian.org>
Sent: Friday, June 08, 2001 7:17 PM
Subject: Re: Finding the Bottleneck

On Friday 08 June 2001 12:25, Jason Lim wrote:
> The network is connected via 100Mb to a switch, so server to server
> connections would be at that limit. Even 10Mb wouldn't be a problem as
> I don't think that much data would be crossing the cable.. would it?

10Mb shouldn't be a problem for DNS.  Of course there's the issue of what
else is on the same cable.

There will of course be a few extra milli-seconds latency, but you are
correct that it shouldn't make a difference.

> As for the "single machine" issue, that would depend. If you're talking
> about either getting a couple of SCSI disks, putting them on a hardware
> raid, or getting an additional small server just for the queue, then I
> think the cost would end up approximately the same. This client doesn't
> have the cash for a huge upgrade, but something moderate would be okay.

However getting an extra server will not make things faster, in fact it
will probably make things slower (maybe a lot slower).  Faster hard
drives is what you need!

> BTW, just to clarify for people who are not familar with qmail, qmail
> stores outgoing email in a special queue, not in Maildir. Only incoming
> mail is stored in Maildir. The Maildirs are actually stored on Disk 1
> (along with the operating system and everything else except the queue).
> I know Maildir can be put in a NFS disk... BUT i've never heard of
> anyone putting the mail queue on NFS, so I'm not sure if the file
> locking issues you mention would pertain to that as well.

For the queue, Qmail creates file names that match Inode numbers (NFS
doesn't have Inodes).  Qmail also relies on certain link operations being
atomic and reliable, while on NFS they aren't guaranteed to be atomic,
and packet loss can cause big reliability problems.

Consider "ln file file2", when an NFS packet is sent to the server the
server will create the link and return success, if the return packet is
lost due to packet corruption then the client will re-send the request.
The server will notice that file2 exists and return an error message.
The result is that the operation succeeded but the client thinks it

There are many other issues with NFS for this type of thing.  NFS is only
good for data that has simple access patterns (read-only files and simple
operations like mounting a home directory and editing a file with "vi"),
and for applications which have carefully been written to work with NFS
(Maildir programs).

http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

To UNSUBSCRIBE, email to debian-isp-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact

Reply to: