Re: Good mail management techniques?

[Ross Boylan <RossBoylan@stanfordalumni.org>]

On Sun, Sep 02, 2001 at 03:32:02PM +1000, Andrew Pollock wrote:
> Hi,
> 
> I'm just wondering how people manage their email...
> 
....
> The main problem I have is that each mailbox/folder/whatever you want to call
> it, grows without bounds. I wouldn't mind something to automatically shoved mail
> in a folder for each month or something like that, but I don't think that
> IMAP/Pine etc support multilevel folders, or do they?
> 
> Anyway, I'm just interested in seeing how other people do it, and what is
> considered "best practise"
> 
> Andrew
> 

First the blue-sky dreams, then the reality:

DREAMS
I think the real solution is to put mail in a real database; I've read
some discussion that Sylpheed (maybe) and one other program (whose
name I've forgotten) can do this.

I've long thought that the folder metaphor for mail is inadequate; I
would prefer a more flexible scheme for classifying mail.  It would
have two properties: first, each piece of mail could belong to several
classifications, so that, for example, mail concerning computers and
politics could be classified under both.  Mail on a mailing list that
referred specifically to my questions or interests could be on the
list and under personal.  And so on.

Second, I'd like to see hieararchical classifications, e.g., computers
includes debian includes debian-user.  Then if I did a search for
topic x it would automatically include all subtopics.  We can do
hierarchies with folders now, but we can't do the folder and all
subfolders logic.


REALITY
In the meantime,  I use exim to filter my mail into boxes and read it
with mutt.  When the individual boxes get so big that performance is
painful, I run the following python script to hack the big ones up and
put them in the archive subdirectory.  I make no claim this is best
practice!

To use this, I switch to the directory with my mailboxes, and
(assuming the script is executable) do
./chopy.py box1 box2 .... boxN

This will ensure that each box is no more than sizeLeft (see below) bytes.
The excess stuff gets turned into things like box1.date.bz2, where
date is an indication of the date of the last message in the archive.
Unless your box is perfectly sorted, you shouldn't take the date too
literally.  Then I move the bz2 files into an archive.

#! /usr/bin/python
# Released under GPL.  (c) 2001 Ross Boylan
import fcntl,os,re,stat,sys
import pdb


sizeLeft = 15*1024*1024  # Size to leave
pattern = re.compile(r"^From \S+ (?P<weekday>\w\w\w) (?P<month>\w\w\w)"\
                     r" (?P<day>[0-9]{1,2}) \d\d:\d\d:\d\d (?P<year>\d{2,4})$")
patt2 = re.compile(r"^Received: from")
blockSize = 1024*1024  # size to transfer

def handleFile(file):
    "Process a single file"
    if os.stat(file)[stat.ST_SIZE] < sizeLeft :
        print "%s is not long enough to chop"%file
        return
    fh = open(file, "r+")
    fh.seek(-sizeLeft, 2)
    line = fh.readline()  # read to end of partial line
    while 1:
        pos = fh.tell()
        line = fh.readline()
        if not line:
            break       # hit EOF
        if line[:5] == "From " :
            #pdb.set_trace()
            pass
        m = pattern.match(line)
        if m:
            line = fh.readline()
            if patt2.match(line):
                dateString = m.group('year')+"-"+m.group('month')+\
                             "-"+m.group('day')
                chopFile(fh, file, pos, "."+dateString)
                return
            else:
                print "Odd.  Failed to match on 2nd line.  Continuing search."
                # note we skip over the 2nd line

    print "Could not find message start in last %d characters of %s"%(
        sizeLeft, file)
    fh.close()

    



def chopFile(fh, name, pos, decoration):
    """Chops file name, open with fh, at pos.  Add decoration to name of
    new first part"""
    # fh should be open for mod on entry.  It will be closed
    # by this function
    status = fcntl.flock(fh.fileno(), fcntl.LOCK_EX)
    if status :
        print "Couldn't lock ", file
        fh.close()
        return
    try:
        _chopFile(fh, name, pos, decoration)
    finally:
        # off with their locks
        fh = open(file,"r")
        status = fcntl.flock(fh.fileno(), fcntl.LOCK_UN)
        if status:
            print "Failed to remove lock for %s"%file
        fh.close()


def _chopFile(fh, name, pos, decoration):
    "Chops without worrying about locking"
        
    # copy tail
    fhshort = open(file+".short", "w")
    fh.seek(pos)
    block = fh.read(blockSize)
    while block:
        fhshort.write(block)
        block = fh.read(blockSize)
    fhshort.close()

    # and create long start
    fh.truncate(pos)
    fh.close()

    # assign final names
    archiveName = file+decoration
    os.rename(file, archiveName)
    os.rename(file+".short", file)
    os.system("bzip2 "+archiveName)
    print "Chopped start of %s to %s"%(file, archiveName+".bz2")


# executable part
for file in sys.argv[1:]:
    handleFile(file)