Re: High volume mail handling architecture
RAM is always not the answer with 32Bit machines. You can cause bounce buffers with too much RAM.
The sweet spot for Linux on a 32Bit platform seems to be 4GB of RAM. I had 10GB of RAM in a Courier IMAP
server and the server had problems releasing swap after a week. The kernel was compiled for 64GB of RAM.
When I reduced the RAM to 4GB and recompiled for a 4GB machine these problems disappeared.
On 10/09/04 03:40 +1000, Russell Coker wrote:
> On Thu, 9 Sep 2004 18:44, Marcin Owsiany <porridge@debian.org> wrote:
> > On Thu, Sep 09, 2004 at 06:03:20AM +1000, Russell Coker wrote:
> > > You have to either be doing something very intensive or very wrong to
> > > need more than one server for 20K users. Last time I did this I got 250K
> > > users per server, and I believe that I could have easily doubled that if
> > > I was allowed to choose the hardware.
> >
> > We have a little over 10K users, and the disk subsystem seems to be the
> > bottleneck. When we reach about 600 read transactions + 150 write
> > transactions per second (as reported by sar -b), the load average starts
> > to grow expotentially instead of proportionally. There are about 20K
> > sectors read, and 3K written per second. (That was before I turned noatime
> > on. After that we had about 2K sector writes and 70 write transactions
> > less, and load average dropped to a more sane value - about 3, instead
> > of 20.)
>
> Last time I was doing this I had some Dell 2U servers (2650 from memory) with
> 4 * 10K U160 disks in a RAID-5 (5th disk was hot-spare) and something like 4G
> of RAM. The machines had almost no read access to the drives, something less
> than 10% of disk access was for read because the cache worked really well
> (the accounts that receive the most mail are the ones that have clients
> checking them most often - in some cases people leave their email client on
> 24*7 checking every 5 mins).
>
> The write bottleneck was just under 3M/s, I don't recall how many transactions
> that was.
>
> To give better performance you may want to look at getting more RAM. RAM is
> cheap and you can eliminate most read bottlenecks by caching lots of stuff.
>
> 3K sectors written per second isn't too good, but I guess that's because of
> the 20K sectors read. Get some more cache and things should improve a lot.
>
> Also if using a typical Unix mail server (Postfix, Sendmail, etc) then the
> data is written synchronously somewhere under /var before being read from
> there and written to the destination. If you use a NVRAM card from UMEM
> http://www.umem.com/ for /var/spool then you could possibly double mail
> delivery performance. If you use data=journal and put the journal for the
> mail store file system on the umem device you could probably double
> performance again.
>
> > Also, did you implement virus/spam scanning on that box?
>
> No! Virus/spam scanning was on the front-end machines. It was believed that
> the mail store machines were busy enough with doing the most basic work
> without virus scanning (also the number of licences for the anti-virus
> program didn't match the number of store machines that were planned).
>
> You want to do as much work away from the mail store as possible. Mail store
> machines can not be replaced without major inconvenience to everyone
> (customers, staff, management). Front-end anti-virus machines are
> disposable, if you have a traffic balancing device (such as a Cisco
> LocalDirector or IPVS) in front of a cluster of anti-virus machines then an
> anti-virus machine can go down for a few days without anyone bothering.
>
> If (hypothetically) anti-virus was to take 10% of the performance from a mail
> store then it could require another mail store machine (if you have 5-10
> machines) and therefore that's one more machine which can break and cause
> massive pain to everyone.
>
> Another thing, a mail store machine should require almost no CPU power. Give
> it a single CPU that's not the fastest available. It sucks when you have two
> almost unused CPUs which are both fast and hot and then one breaks down
> killing the machine.
>
> --
> http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages
> http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
> http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
> http://www.coker.com.au/~russell/ My home page
>
>
> --
> To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
>
--
------------------------------------------
Ted Knab
Chester, Maryland 21619 USA
------------------------------------------
[In War] Conquest is easy. Control is not.
-- Kirk, "Mirror, Mirror", stardate unknown
Reply to: