Re: Request: Ticker-2-Mail
On Sun, Jul 11, 1999 at 06:11:01AM -0400, Justin Maurer wrote:
> > Given the growing numbers of ticker and news services, don't people
> > think it is time to establish a system that will convert the most
> > recent items into mail sending to the user? One would then only
> > have to add a cronjob and read n more mails instead of leeching to
> > all the news services. The mail should, of course, not contain the
> > menues, buttons, ads etc., thus lynx -dump is not the proper tool.
heh. i've been working on something like this this weekend and came up
with two perl programs.
the first parses the daily freshmeat-news mail message, and injects the
records into a local postgres database (uses dbi so it should work with
any sql database). i've got this running now out of my .procmailrc.
seems to work fine so far....i fed it my archive of old freshmeat news
and it swallowed most of them (the msg format appears to have changed
several times in the last few months)
the second is a shell script which snarfs
http://freshmeat.net/backend/annnouncements.txt and compares it to the
last one downloaded. if it is different, then it feeds it into a perl
script which parses the records and injects it into a postgres table
called "appindex". i run this out of cron once every hour.
the reason i did this was because i often need to search for programs
but really hate having to use a web interface - it's too clumsy and
way too slow. i'd rather have the db residing on my own disk so i can
construct my own queries and get the output as plain text rather than
html.
while writing these, it occurred to me that it's entirely the wrong way
to go about it. it seems to me that it's the data which is important,
not the web pages. there should be a distribution network for this data
so that anyone and everyone can have their own local copy if they want
to. simplest way to do this would be to use email as the transport
- records in the local db could be added, deleted, and updated in a
batch via compressed pgp signed mail attachments. everyone could then
do whatever they needed to with their own local copy (e.g. do their
own custom searches, write their own custom front-end, or neat sql
things like "delete from appindex where lower(license) ~ 'commercial' or
lower(license) ~ 'shareware'".
ideally i'd like to convince the freshmeat guys that it would be cool to
be able to subscribe to a daily machine-parsable (XML perhaps, or just
plain old key=val with \n\n to separate records) update which could then
be fed into your own 'localmeat' :-) database.
craig
ps: if anyone wants a copy of these scripts, email me. be warned,
though, they work but they're a first draft and not pretty. they're
completely undocumented and unsupported. the code should be pretty easy
to read though.
--
craig sanders
Reply to: