On Tue, Aug 26, 2003 at 08:03:35PM -0500, Ron Johnson wrote:
> On Tue, 2003-08-26 at 18:23, Bijan Soleymani wrote:
> > On Tue, Aug 26, 2003 at 02:30:57PM -0500, Ron Johnson wrote:
> > > On Tue, 2003-08-26 at 13:29, David Turetsky wrote:
> > > > > On Tue, 2003-08-26 at 10:05, Kirk Strauser wrote:
> > > > > From: Ron Johnson [mailto:ron.l.johnson@cox.net]
> > > > > For example, COBOL has intrinsic constructs for easily handling
> > > > > ISAM files in a variety of manners. Likewise, there is a very
> > > > > powerful intrinsic SORT verb.
> > > > >
> > > >
> > > > Yes, but how does that compare with similarly powerful features in Perl?
> > >
> > > I *knew* someone would ask about the Programmable Extraction and
> > > Reporting Language...
> > >
> > > Please don't think that I am implying that Perl or C are bad languages.
> > > I certainly wouldn't write a logfile analyzer in COBOL.
> > >
> > > For my knowledge, how would Perl sort the contents of a file,
> > > allowing the programmer to use a complex algorithm as an
> > > input filter, and then take the output stream, processing it
> > > 1 record at a time, without needing to write to and then read
> > > from temporary files with all of the extra SLOC that that entails?
> >
> > 1) Read from records from file into an array.
> > 2) Order them with any perl code you want and store them in an array.
> > 3) Use a nice foreach loop to process them.
> > --- Outline of Perl code ---
> > #!/usr/bin/perl
> > @records = read_records_function("records.txt");
> > #either
> > @sorted = sort @records; #to put things in alphabetical order
> > #or
> > @sorted = sort function @records; #to sort using a function
> > # or even
> > @sorted = sort {sort-function-code} @records; #to have it in-line
> > foreach $record (@sorted)
> > {
> > # code to process records
> > }
>
> That's great for in-memory stuff.
>
> What about when there are, say, 10 or 40M records to process?
>
> And what if you only need to SORT a fraction of those 40M records,
> and the winnowing algorithm is very complicated, possibly needing
> to access other files or database tables in the process?
Well then you do it differently:
open(INPUT,"records.txt");
LOOP: while($record = <INPUT>)
{
if(!input_filter($record))
{
next LOOP;
}
else
{
#code to process record
}
}
close(INPUT);
Where input_filter() is the winnowing function you define.
Note: This program assumes that records are seperated by newline. But
you could set the input record seperator to whatever you want.
Bijan
--
Bijan Soleymani <bijan@psq.com>
http://www.crasseux.com
Attachment:
pgpJR1mVTgbY_.pgp
Description: PGP signature