Re: Deploying Hotmail-like service with crazy requirements
On Sat, 25 Mar 2000 dan@cg619985-a.adubn1.nj.home.com wrote:
> Wonder if anyone has experience with very high-volume web-to-mail
> services, or just have any recommendations at all. We are going to be
> working on a 100,000 user web-to-mail gateway system similar to Hotmail.
> Every user will have a 50 MB mailbox. At this point this is a one-person
> job, and quite frankly I don't know where to start since so little
> information on these types of systems is available. First, can this be
> done with Linux at all (meaning will it scale well)? How can this be
> done? What software/hardware needs to be employed. The ISP's RADIUS
> database is stored in MS SQL 7.0 database, so the users will have to be
> authenticated against it in order to be allowed to send/receive mail via
> a web-client running on Linux box/boxes. No local accounts on the mail
> cluster will be permitted. Please let me know if you have any clue what
> so ever on these issues. Anything will be greately appreciated. Sorry if
> this message is a little off-topic, but since it has to be deployed on
> Linux I hope this will prove to be a nice thread.
>
> Regards.
Firstly, make sure you have the network resources to support this kind of
traffic. Remember that many of these messages are going to contain
attachments in the order of megabytes so allow for it. A lot of ISP's have
some pretty crappy networks. Make sure they have connectivity to more than
one provider and are announcing their network to all their providers.
Sendmail will work for your inbound mail ... you will probably need a bank
of them ... at least 5. I would not use sendmail for OUTBOUND mail. It's
queuing sucks and requires a lot of scripting and babysitting when some
popular destination goes unreachable. It backs up the entire queue. Use
exim or qmail for outbound mail but be careful in setting limits. I know
Exim can send so much mail at once as to overload system RAM, file and
process resources. There are config items to set limits ... so I would use
them to limit the number of concurrent deliveries and queue running
processes.
For web, I would probably use Zeus on Solaris 64-bit machines or something
on Alpha's ... the AlphaServer DS10's and 20's are nice machines. The 10's
go for under $5K each. Again you are going to need a bank of them ...
probably 5-10 machines.
You are probably going to want to load balance stuff. Look at the Accelar
series of load balancers from Nortel ... they can also do SSL in
hardware to offload that stuff from the web server CPU. www.phobos.com
used to have some good stuff for Linux but I am not sure if they decided
to continue with it or not. The Nortel boxes even examine the content of
web server responses and if it sees an error, it will intercept the data
and retry the query to another server and the customer will not know an
error occured. Altion and F5 (Big IP) also make these things. They all are
price competitive ... figure $30K and up per box.
The important thing to remember is that email is the most sensative of all
net services. When people can not get or loose email, they become VERY
irate VERY quickly.
In a nutshell:
You need a reliable network
You need multiple servers to provide high-availability.
You need a lot of throughput on your servers. Intel machines will probably
bog particularly if you are doing https.
You might be able to pull it off for under a couple of million bucks.
(Cisco quotes about $250K for just the networking pieces to support an
operation like this)
If you really want to do it with Linux/Intel, you will likely end up
spending about the same because you will need more machines that will take
up more space and need more admin resources.
Again .... people WILL NOT TOLERATE slow, lost, intermittant or otherwise
unreliable email.
Reply to: