[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: How to improve on this Python code. Was: Re: OT: Why is C so popular?

Mark Roach wrote:

On Sat, 2003-08-30 at 02:21, Jacob Anawalt wrote:
Ron Johnson wrote:

But, of course, that's not an issue in The Clearly Superior Language,
is it?

Ok, if this thread has accomplished little else, it seems to have gotten a couple people, including myself to play around with Python.

I have a simple little perl program at work. It parses a mailbox format file and extracts email addresses from it. The mailbox file is a bunch of bounced email notifications. The program uses a couple regular expressions to extract the data and a hash to hold a unique list of the emails. It then connects to a database to see if those people have already been flagged as having bad emails.

I made a copy of this script, removed the database stuff, and stripped it down to the basic process of read the file, find emails excluding some admin type addresses and then print out the unique bounced addresses. Then I wrote a Python version. It didn't take very long to code, about the same time to write the perl code if I take out the time reading howto's and the language reference.
real    0m0.056s
user    0m0.051s
sys     0m0.004s
real    0m0.839s
user    0m0.818s
sys     0m0.008s
What did I do wrong to make the python code take over ten times as long?

Sure, use perl's #1 optimized, built-in feature for your test case ;-)

I didn't realize I was until I got some emails about it :)

I wasn't trying to do a performance test, just a readability. I liked that they were about the same length, and I'll see how well I understand them when I look at them in January. I was suprised by the difference in speed doing what I do all the time with my Perl code so I wanted to ask about what I could have done better. I think that question has been answered here and in a few off-list emails.

I think, that in this case it is probably safe to say "who cares?" do
you really care about .06 seconds vs .8 seconds? is your real data large
enough for the difference to matter?. (if it is, btw, you might want to
use a while loop and 'line = fi.readline()' instead of putting the whole
file in memory)

The code "for line in fi.readlines()" reads the whole file into a tuple or something instead of iterating lines? Well that was a big woops.

Sometimes I have 100MB mail files, so 10x as long is a difference (and I don't want to slurp the thing into memory - thanks for pointing out that I was).

On the other hand I don't process these every day and can do something else while it's running. In any case, I'm trying to find where I might use Python from day to day. I'll try CGI next, especially some longer and more complex work where I normaly write a few Perl modules. I don't do GUI work on Linux very often, so I probably won't dabble in that.

No one has said (that I know of) that python is the fastest language
ever. The only thing that I have heard is that it is "fast enough" and
that the benefits outweigh the addition (.8 seconds of) run time. In
some cases it might not, you might not like it, you might have code that
already works just fine, right tool for the right job and all that.

in terms of a very simple optimization though, try adding this just
before the "email_mo = email_pattern.search(line)" line:

if not '@' in line:

this prevents every line from having regex used when there is no chance
of it matching. It also cut the runtime for my test case (a 1.3 meg
mailbox) from .380 to .117

So my mistake was using regexp instead of find to ignore ignore lines that didn't have email addresses in them. Someone else also pointed out using find instead of the regex to limit the matches.

or if you want to really cheat, replace you file open and main for loop
with this:
lines = os.popen(r"egrep '[a-zA-Z_\.\-]+\@[a-zA-Z\.\-]+' bademail.mai").read()

matches = email_pattern.findall(lines)
for match in matches:
   if not ignore_pattern.search(match):
       badHash[match.lower()] = 'bad'
yes, it _is_ evil, but it works ;-)

real    0m0.069s
user    0m0.060s
sys     0m0.010s


Well, that's a way to limit the lines matched :) I'm not sure what the rest of the code does exactly, but if it turns:

The email address Jacob <jacob@CacheValley.com> was not valid.


"jacob@cachevalley.com" stored in match, then it's a solution I'll have to keep in mind if I decide to try Python for another file search and database script.


Reply to: