Re: scripting lynx
Hi,
I tried to install LWP and had no success. Is there a
way to install it w/o using the tar ball?
Kris
--- Craig Sanders <cas@taz.net.au> wrote:
> On Wed, Aug 01, 2001 at 06:08:29PM +0200, Russell
> Coker wrote:
> > I tried both of them with no difference.
>
> interesing. either should have worked.
>
> > > or use the LWP modules to make yourself a
> web-bot.
> >
> > I may have to do that. Thanks for the
> suggestions.
>
> you may need to set the Referer: header in the HTTP
> request. some cgi
> scripts check the referer...(yes, that's pointless
> and stupid, but it's
> quite common).
>
> and set the user agent to something like:
>
> $ua->agent('Mozilla/4.51 (Macintosh; I; PPC)');
>
> i generally use netscape on mac as my user-agent in
> web robots because:
>
> a) moronic sites generally don't block netscape on
> macintosh
> (i have seen some sites that block netscape on
> linux with a stupid
> message like "sorry, we don't support your
> browser/operating-system".
> unfortunately, brain-dead web design is not yet a
> capital crime)
>
> b) said moronic sites generally wont output moronic
> IE-specific junk
> if they detect netscape. sometimes. if you're
> lucky.
>
>
>
> btw, the perl HTML::TokeParser module is excellent
> for extracting stuff
> from web pages. i used this (plus LWP::UserAgent,
> HTTP::Cookies, and
> HTTP::Request) to write a wrapper script for
> searching the Melbourne
> Trading Post site, which is one of the most
> brain-dead cretinous sites
> i've ever had the misfortune of having to use.
>
> there's also HTML::TableExtract for getting data out
> of html tables.
>
> these modules are all packaged for debian.
>
>
> craig
>
> --
> craig sanders <cas@taz.net.au>
>
> Fabricati Diem, PVNC.
> -- motto of the Ankh-Morpork City Watch
>
>
> --
> To UNSUBSCRIBE, email to
> debian-isp-request@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster@lists.debian.org
>
__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/
Reply to: