[Rant] The Endless Search for a Mail Client That Doesn't Suck
It's 2004. Email has been around for what, 30+ years? Is there an
email client out there that, after 30 years, still doesn't suck?
That's a rhetorical question. Anyone who blurts out "Mutt!" gets a
swift kick in the genitals. And if anyone even tries to utter "gnus", I
will beat them with a rolled up newspaper until they say, "No gnus is
*good* gnus" and mean it.
Here's the background to my situation:
I check my email from at least two different computers (a desktop and a
laptop computer) so I use a local IMAP server to access my mail. I've
been using this setup for around 4 years. If the world wasn't such a
cruel place, this would be the ideal setup for the situation.
Back when I started, I was a relatively modest email user. I had fewer
than 10 IMAP folders and was only subscribed to a couple mailing lists.
I started off using mutt, but I immediately discovered that mutt's IMAP
support was laughable, to put it nicely. It was extraordinarily slow,
and had absolutely no support for offline mode, which made it pretty
much useless for my laptop. Nonetheless, I put up with it for a few
months--long enough to learn how to customize it and become comfortable
with it. I was never really happy with mutt, and since I would fire up
emacs to compose mails anyway, I figured I'd give gnus a try.
I discovered gnus had several advantages over mutt. It was
significantly faster at loading folders (since it cheats and only
displays unread mails by default) and had rudimentary offline support.
Also, I didn't have to wait for emacs to load every time I wanted to
compose a mail because it was already there.
Over the years, my email usage has grown enormously. I now have 70 IMAP
folders, am subscribed to around 50 mailing lists, and receive over 2000
emails per day.
Gnus scaled, hmm, OK with this usage. It worked adequately when the
IMAP server was on a local network, but was pretty painful when I was on
the road. I would often read mail too quickly so that gnus couldn't
display the articles. And, due to the single-threaded nature of elisp,
any time gnus was busy doing something like checking for new mail in 70
folders or expiring messages in a very large folder, it would be
completely unusable, often for several minutes. Or, if the wireless
connection on my laptop dropped, gnus would be completely unusable for
10-15 minutes until the connection finally dropped.
Furthermore, I ran into problems with offline synchronization. Even on
a local network, it would take *FOREVER*, skip messages, time out, and
generally fuck up. It was pretty much useless.
I finally became fed up with gnus' IMAP support when I went to Brazil
for Debconf this spring. With my IMAP server something like 10,000
miles away, gnus was completely unusable. Synchronizing folders was
impossible; just trying to read mail in my inbox was a joke. About 95%
of the time, gnus was sitting there unresponsive, busily doing whatever
it does in elisp.
Despite the latency, offlineimap seemed to work OK from Brazil, so I
decided to make gnus switch to using a local Maildir folder. This
seemed like the ideal setup. No more of gnus just sitting there
pondering the meaning of life or whatever when there was a lot of
latency. No more completely broken offline synchronization. No more
retardation when my wireless signal flaked out. I thought I'd *finally*
have an email setup that wasn't constantly pissing me off.
Wrong... So very wrong...
Gnus uses the backend "nnmaildir" for its Maildir support. I can't say
enough bad things about nnmaildir. But, I'd like to try.
First problem: startup time. You might think gnus would be able to
startup pretty damn quickly if it only had to read a local Maildir
directly, right? Well... it takes 5-10 minutes for nnmaildir to
startup, the entire time during which the hard drive is furiously
scribbling its ass off. Really fucking lovely to happen on a laptop,
especially if I'm running on a battery. Here, nnmaildir, have 1 hour of
my battery life. I don't need it *that* badly.
Second problem: nnmaildir's memory usage makes OpenOffice.org look light
and fluffy. On my setup, it takes up, oh, a couple hundred megabytes of
memory. My laptop once had 256MB of memory, so if I started gnus, I was
automatically hitting the swap file pretty hard. I had to buy an
additional 256MB just to be able to do anything else while my mail
client was open. That is seriously fucking disturbing.
The next problem, and this one's a doozy, was its completely
non-standard use of message flags. Instead of appending flags like
",RS" to the message filename (to mark the message as "seen" and
"replied" in this case) like every other fucking Maildir-supporting
email client on the planet, it uses its own, errr, "system". In each
Maildir directory, it creates a ".nnmaildir" directory, which in turn
contains a "marks" directory, which in turn contains directories like
"read", "reply", and "ticked", which in turn contain hard links to the
original message files.
For example, if a mail was marked as seen and replied, you would find a
hard link in .nnmaildir/marks/read/1234 and a hard link in
.nnmaildir/marks/reply/1234, both of which point to cur/1234.
What the fucking shit? That bears repeating: What the fucking shit?
No, seriously: What the fucking shit?
This thoroughly broke synchronization with offlineimap. Offlineimap
would never know a message was read since gnus would never modify the
message flags, so the IMAP server would never know, and thus using
offlineimap from two different computers was useless. Not to mention
checking my mail through webmail when I didn't have my laptop with me...
However, the nnmaildir author insists it's easy to write a script to
convert the .nnmaildir/marks hard links to standard Maildir message
flags. So, I wrote such a script. Here it is, in all its glory:
#!/usr/bin/perl -w
# Maildir flags are:
# D (draft)
# F (flagged)
# R (replied)
# S (seen)
# T (trashed)
# and must occur in ASCII order.
#
# flagmatchre = re.compile(':.*2,([A-Z]+)')
#
# filename:2,F => .nnmaildir/marks/tick/filename
# filename:2,R => .nnmaildir/marks/reply/filename
# filename:2,S => .nnmaildir/marks/read/filename
use strict;
use File::Basename;
use Getopt::Long;
$Getopt::Long::ignorecase = 0;
my $from_gnus = 0;
my $from_maildir = 0;
my $dir = "~/.Maildir";
GetOptions('-g' => \$from_gnus,
'-m' => \$from_maildir,
'-d=s' => \$dir);
if (! ($from_gnus ^ $from_maildir)) {
die "Usage: sync_nnmaildir -g [-f]\n or: sync_nnmaildir -m [-v -f]\n";
}
for (glob "$dir/*") {
my $mb = $_;
mkdir "$mb/.nnmaildir";
mkdir "$mb/.nnmaildir/marks";
for (glob "$mb/cur/*") {
my $file = $_;
/(.*)\/cur\/(.*?):.*2,(.*)$/;
my $path = $1;
my $message = $2;
my $flags = $3;
if ($from_maildir) {
# Sync ticked flags
if ($flags =~ /F/) {
mkdir "$path/.nnmaildir/marks/tick";
my $dst = "$path/.nnmaildir/marks/tick/$message";
link "$file","$dst"
and print "Added mail in $mb to nnmaildir ticks\n";
} else {
my $dst = "$path/.nnmaildir/marks/tick/$message";
unlink "$dst"
and print "Removed mail in $mb from nnmaildir ticks\n";
}
# Sync replied flags
if ($flags =~ /R/) {
mkdir "$path/.nnmaildir/marks/reply";
my $dst = "$path/.nnmaildir/marks/reply/$message";
link "$file","$dst"
and print "Added mail in $mb to nnmaildir replies\n";
} else {
my $dst = "$path/.nnmaildir/marks/reply/$message";
unlink "$dst"
and print "Removed mail in $mb from nnmaildir replies\n";
}
# Sync read flags
if ($flags =~ /S/) {
mkdir "$path/.nnmaildir/marks/read";
my $dst = "$path/.nnmaildir/marks/read/$message";
link "$file","$dst"
and print "Added mail in $mb to nnmaildir seen\n";
} else {
my $dst = "$path/.nnmaildir/marks/read/$message";
unlink "$dst"
and print "Removed mail in $mb from nnmaildir seen\n";
}
} elsif ($from_gnus) {
my $new_flags = '';
if (-e "$path/.nnmaildir/marks/tick/$message") {
$new_flags = $new_flags . 'F';
}
if (-e "$path/.nnmaildir/marks/reply/$message") {
$new_flags = $new_flags . 'R';
}
if (-e "$path/.nnmaildir/marks/read/$message") {
$new_flags = $new_flags . 'S';
}
if ($new_flags ne $flags) {
rename "$file", "$path/cur/$message:2,$new_flags"
and print "Marked mail in $mb as $new_flags\n";
}
}
}
}
The script works, as well as it can anyway. So now, if I want to read
my mail, I startup offlineimap, let it download mail to completion, run
my script with "sync_nnmaildir -m" to make the nnmaildir flag system
match the Maildir, and then startup gnus.
20 minutes after I've begun, I can start reading my mail. Weeeeee...
When I'm finished, I quit gnus, run "sync_nnmaildir -g" to make the
Maildir match the nnmaildir flag system, and then re-run offlineimap to
completion. Then I can move over to my desktop system and repeat the
process, ad nauseum...
Overall, it's all a pretty disgusting setup, but I was willing to live
with it for the time being. Sure, at times it felt like a hot poker was
being shoved up my ass, but I have a pretty high tolerance for pain.
Today, nnmaildir finally put itself out of its own misery. And really,
who could blame it?
When I was, ehem, "happily" using gnus today, I got the message,
Adding new name: too many
links,/home/nelson/.Maildir/INBOX.lists.debian-bugs-dist/.nnmaildir/num/:,
/home/nelson/.Maildir/INBOX.lists.debian-bugs-dist/.nnmaildir/num/7d00
Here's the bug report: http://bugs.debian.org/263514
No matter what I did, I got that message. Why is it adding a new name
to a directly called "num"? Shouldn't it be adding a new number? And
what's with the name 7d00? Is that a dirty word you can type on a
calculate and turn upside down to giggle at?
Thoroughly annoyed, I tried to track down the problem. So I did:
$ ls /home/nelson/.Maildir/INBOX.lists.debian-bugs-dist/.nnmaildir/num/ | wc -l
32000
Hmmm, OK... I suppose this is another one of those mystery directories
full of hard links? However, every one of those files is empty. Does
it serve a purpose, other than eating up inodes like pac-man eats white
pills? Who the fuck knows?
I wondered if I really had 32,000 emails in that folder. It wouldn't be
surprising if you tracked debian-bugs-dist. That list receives every
since mail sent to the Debian BTS.
$ ls /home/nelson/.Maildir/INBOX.lists.debian-bugs-dist/cur/ | wc -l
6145
Nope, nnmaildir is just totally out of its fucking mind. Apparently it
likes to drop empty files into a .nnmaildir/num/ directory until it
can't create any more. And then it cries. Which in turn makes me cry.
I removed all 32,000 files, but apparently they did serve a purpose
because the number of unread mails went from 1600 to 150. Hope those
mails weren't important...
So now I'm seriously considering going back to mutt, but I just can't
get into it.
I have a few modest requirements that a mail client must meet:
1. I must be able to customize the order folders, and I don't want to
give them retardedly ugly names to force a correct alphabetic order.
This is very important when you have 70 freaking folders like I do.
2. I must be able to read the folders sequentially in the order I
specify with minimal pain. Obviously, I read from highest priority
to lowest, since I may not have time to read every single folder. I
don't want to be interrupted while in the sequence, because old mails
in lower priority folders are more important to me than brand new
mails in higher priority ones.
3. I must have a powerful editor. Emacs is generally my first choice,
but I'd be willing to learn another as long as it could do everything
I do in emacs.
4. It must support Maildir, and must play nicely with offlineimap.
5. It must have a decent summary view. I want to be able to see all of
my folders, the amount of mail in each, and be able to quickly choose
one to view the mail.
6. It must have a decent expiry system.
7. It must not be dog slow. I have big folders and I don't want to wait
5 minutes to load them.
Mutt fails several of these.
For (1), you can specify the order, sort of. It doesn't like a
directory full of Maildir subdirectories like what offlineimap uses, so
I use the following in .offlineimaprc to write the folders to a file
that mutt can read:
[mbnames]
enabled = yes
filename = ~/.mutt/mailboxes
header = "mailboxes "
peritem = "+%(foldername)s"
sep = " "
footer = "\n"
However, I have no ability to customize the order the folders are
written to the file, so (1) fails, but at least it came close
Mutt thoroughly fails (2). If I'm reading mail in a folder and a
previous folder receives mail, when I press 'c', mutt always wants to go
to the previous folder. That's especially annoying if I'm reading any
folder that comes after debian-bugs-dist. Since bugs-dist receives a
new mail about once per minute, it *always* wants to go back there.
There are workarounds like making the new mail checks very infrequent,
but that's also annoying since it makes the summary view useless.
Mutt does satisfy (3) and (4). Nice job. F+ for you.
(5): Hahahahahahaha. Here's a snip of what the summary view currently
looks like for me in mutt:
23 drwx------ 6 nelson nelson 4096 Jun 05 19:02 =INBOX.lists.debian-
24 drwx------ 6 nelson nelson 4096 Jun 05 19:02 =INBOX.lists.debian-
25 drwx------ 6 nelson nelson 4096 Jun 05 19:02 =INBOX.lists.debian-
26 N drwx------ 6 nelson nelson 4096 Jun 05 19:02 =INBOX.lists.debian-
Hey, I guess that last one has new mail in it. That's good to know.
Now if I only knew how many mails, or even the rest of the folder's
fucking name...
OK, so I could hack it to drop the redundant "INBOX." stuff and get back
6 characters. Yippee. And I guess that view is customizable, but it
doesn't seem possible to get what I want anyway. Or at least it never
worked when I tried. It's also extremely awkward getting to and from
the summary view. When I quit reading a folder, I want to go to the
summary view, not exit mutt, dammit.
Not surprisingly, (6) fails too. In the past, I've used the following
hook as a poor man's expiry:
folder-hook "lists" 'push "<delete-pattern>~d >2w<enter>"'
That deletes all mail over 2 weeks old. Really, all of it. Even stuff
that's flagged or unread, both of which I damn well don't want deleted.
I guess it's somehow possible to do what I want, but why does it have to
be so difficult?
Finally, (7) fails. Mutt is the slowest damn email client. Caching?
Indexing? Naaahhhh... Not without patching anyway.
Unfortunately, mutt and gnus are generally considered the best "power
user's" email clients, so there's no where to go but down...
kmail? Horrible editor, barfs on offlineimap's Maildirs...
sylpheed? Don't recall it supporting Maildir, horrible editor,
generally shitty...
thunderbird? Never tried it, doesn't support mailing lists from what
I've heard. Most likely also suffers from the horrible editor syndrome.
evolution? You must be shitting me.
mozilla-mail? Let's just stop right there.
I guess there's no end for my misery in sight. I hear Donald Knuth quit
using email 14 years ago. With that kind of foresight, he really *is* a
genius.
--
You win again, gravity!
Reply to: