Re: Building an IMAP server
On Wed, Feb 05, 2003 at 05:34:16PM -0800, nate wrote:
> > What's the better way to go when building a new server? Should I start
> > with 2.x or stay at 1.5?
>
> If it were me, I would use 1.5. See my other posts with the maintainer
> of the cyrus 2 packages for debian for why. It really depends on your
> requirements. cyrus 1.5 is VERY VERY old and does not have near the
> feature set that cyrus 2 has(e.g. sieve filtering for server-side
> filtering). But the flip side, is it is "tried and true".
As far as I've seen yet, cyrus 1.5 will fullfill the current
requirements exceptionally well :) By making use of ACLs, I'll even be
able to get around the need of server-side filtering.
What bothers me is having to upgrade to 2.x at some time. Upgrading
may be either so difficult that it is better to start with 2.x from
the very beginning, or it may be no problem at all. I just don't know.
However, since 1.5 comes with Woody, I suppose it to make the least
problems, if any at all. Once you get a bit into it, it's fairly easy
to setup and to use. It has good performance and offers some features
that will be actually very useful. Such a thing that flawlessly works,
more or less just out of the box without any hassles, is what I
want. (In fact, to have things that work is one of the reasons to use
Debian distributions, thanks to all the developers and package
maintainers!)
> > But how do users authenticate when they're not local users? I'm
>
> If you do use LDAP, and you use libnss-ldap(to lookup account info
> in the LDAP database for stuff like finger, ssh, etc) you cannot
> use cyrus 2.x.
Well, I'd like to use LDAP to have a global address book for users, as
a first step. If I only could get it to work, LDAP could be used to
authenticate mail-users.
But lacking something else, I would set up users with adduser, though
not create home directories and have /bin/false as their shell. This
would result in plain text authentication, which is not exactly
secure.
> There's a library conflict w/sasl which totally hoses the
> system. Which is one reason why I won't be using cyrus 2 anytime
> soon. I'd expect this issue to be eventually resolved perhaps in the
> comming year or 2, especially as more users start to use this new
> sasl library, LDAP authentication is becomming more common.
Does SASL use LDAP?
> > Hm, courier is fairly easy to setup, but it's slow on larger
> > mailboxes. It's ok with only a few users, but nonetheless you'll
> > probably not be happy with it on larger mailboxes.
>
> thats good to know, I haven't tried it myself, I migrated my last
> company from UW IMAP to Cyrus(upgraded hardware at the same time),
> my boss did some testing and noticed a near 20x improvement in
> performance for large mailboxes(10k+ messages).
The testing I did/do was/is on the mesages of debian-users, about
38k/155MB in the folder, on ext3fs, IDE disc, 700 MHz Athlon. Courier
becomes, hmm, acceptable for a single user when the FS is mounted with
the noatime option. Cyrus is still happy with it, even with webmail
clients; it might be even better when used on a partition mounted with
noatime, but I can't test this here because I won't mount /var with
noatime. Maybe it won't do any harm, but I don't know.
> Since i use webmail I need to keep folder sizes small(folders I
> routinely access I try to keep under 500 messages, my archive
> folders have 1500+),
This won't be a problem with courier. When testing, I created a folder
and copied over about 8000 messages from debian-users into it. Courier
should be ok, for a single user, up to about 10k messages in a
folder. However, cyrus is much faster in any case.
> just so that folder access is near instantaneous. If you use a mail
> client which caches data such as netscape, mozilla, and I think even
> outlook caches data, response time will be near immediate for even
> huge mailboxes.
Ja, besides squirrelmail and imp, I'm using the mozilla client for
testing. The mozilla client is really fast.
But I'll stay with mutt ... :)
> Keep in mind to use a good file system or at least tune your
> filesystem if you plan to have tens/hundreds of thousands of small
> files. I hear reiserfs is good for this.
The server will need some RAID setup to have the data mirrored, either
software RAID or hardware RAID. Unfortunately, it will have IDE discs
to provide sufficent storage capacity at reasonable costs. My idea is
to eventually use a fast SCSI disk to put the more actively used mail
folders on it and to create an archives.* hierarchy on the IDE
disc. Users will be forced to move their older mail to their folders
under archives.* by setting quotas accordingly. Thanks to cyrus, this
can be set up transparently.
As for the number of files that need to be handled, I've no good idea
of how many it will be. Here at home, my mail folders currently keep
1.2 GB in 158675 files (including directories). Each user will
probably have a quota of 1 GB, but the mails they get are often much
larger than those I receive because they mail around more attachments.
Hm, with 100 users, the server may end up with about 15 million
files in the worst case. hmmmm ...
> >> > + What's the best way to do backups and restores?
> >>
> >> just tar up the user's mail folder(/var/spool/cyrus/mail/user/$USER).
> >
> > Can exim be suspended somehow so that it keeps incoming mails in the queue
> > instead of delivering it during backup or recovery operations?
>
> not sure, haven't used exim myself, when I did migrations I just stopped
> the SMTP server, if you time it right(e.g. with scripts) you can migrate
> a mail box in maybe 10 seconds, hardly noticable. When I was doing real
> time migrations I would use rsync, and have it update everything then
> run the cyrus commands to rebuild, could sync 20 mailboxes in ~25-30
> seconds(via T1 link over VPN to the other side of the U.S.)
Whow! :)
The migration we'll have to do is another issue. It has to be done
step by step, and the current server is a black-box router. Once the
new server is running, I'll set up a user to migrate next on the
server, set up fetchmail to fetch mail for him from his current
account and go to his workstation and set up access to the new
account. Eventually, I'll have to copy his old mail over to the new
server. Then I'll do the same for the next user. When all users are
migrated and everything runs fine, the old server will eventually be
reconfigured as to directly hand over all mail to the new server. At
some later time, the old server will get fully disabled ... It'll be
much work and take quite some time.
> > Uh! What are you doing with so many accounts? Isn't it easier to have
> > server side filtering to direct mail into appropriate folders?
>
> cyrus 1.5 has no such support. And even if it did, there really is
> no way to determine what address the email was sent to. especially in
> the case of spam. Newer mail servers add something like a Delivered-To:
> header(perhaps exim does, I use postfix)
All my incoming mail has an Envelope-sender: header set. This comes
probably from exim, but I'm not sure.
> but my current installation does not provide such information, so I
> cannot rely upon something like procmail or sieve to filter mail. I
> personally find it useful to see what email addresses recieve what
> spam, and by having each email box have it's own account, it makes
> it impossible(?) for incorrect filtering.
Hm, I'm using only one address. Maildrop and the exim ~/.forward
filter take care to put all mail into the designated
folders. Spamassassin does quite a good job in filtering for SPAM, but
there's only about 1 mail in 1000 that is actually SPAM. The SPAM is
automatically bounced, except when it comes from a mailing list
(otherwise, the bounce would go the list-owner, what is not what I
want).
So I've really no need to have several addresses/accounts.
> and as an added bonus I can recieve email without seeing it in my
> client(some email addresses I check only a few times a year).
Sounds very peculiar to me :)
> also keep in mind cyrus is generally a one way trip, I haven't seen
> any tools that allow for easy migration away from it. when I
> migrated from UW imap to cyrus I had users manually copy their
> email(using their IMAP clients) to the new system, it worked well.
You could move away from cyrus the same way, I'd think. The only thing
you need is a client that can access both cyrus and the new system.
> I gave them about a 2 month grace period. If you have a lot of users
> this may not be an easy solution(my company had ~50 employees at the
> time).
Ja, see above ... For those users who want to take over their mail to
the new server, I'll have to copy their outlook.pst to some place on a
file server from where I can access it by a specially prepared
client. From that client, I'll copy over the mail to the new
server. Another thing to deal with are users' addressbooks ...
> migration manually. Cyrus did not accept messages with null bytes in
> them so some mail had to be lost(or printed), but maybe 1 in 20,000
> messages had a null byte, wasn't an issue for any of the users. I
> think there was only 2 or 3 messages that were affected accross the
> entire userbase I had.
Hm, I hope I'll not run into troubles when copying over the old
mail. I know that Outlook can, sometimes, a little bit, with some
effort and some luck, do IMAP very slowly ...
It will be a great delight to finally free the clients from Outlook,
as it tends to make ever more trouble over time.
GH
Reply to: