Re: Slow Script
On Tue, Feb 03, 2009 at 06:14:48PM +0100, Gorka wrote:
> Hi! I've got a perl script with this for:
>
> for (my $j=0;$j<=$#fichero1;$j++)
> {
> if (@fichero1[$j] eq $valor1)
> {
> $token = 1;
> }
> }
>
> The problem is that fichero1 has 32 millions of records and moreover I've
> got to repeat this for several millions times, so this way it would take
> years to finish.
> Does anybody know a way to optimize this script? Is there any other linux
> programing language I could make this more quickly whith?
> Thank you!
Although the Perl could definitely be optimized (and you've already been
shown one way to do so), your core issue is that you're doing several
million passes over 32 million records. That's not going to be fast in
any language. (Even if you can check a million records per second,
that's 32 seconds per pass, or about 6 hours for 1,000 passes, or just
over a year for a million passes.)
Given the small piece of code that you posted and the magnitude of the
numbers you've stated, I strongly suspect that you probably want to use
a database for this, since it's exactly the kind of thing they're
designed to handle, but we can't say much more than that without
additional information about what you're actually trying to accomplish.
--
Dave Sherohman
NomadNet, Inc.
http://nomadnetinc.com/
Reply to: