Re: robots.txt contents.
> > It shows, IMO, a misunderstanding of what the web is. Why would we disallow
> > access to e.g. /Packages/ or /Lists-Archives/ ?
> >
> > Could we change this?
> >
> I don't know why Lists-Archives is in there, but I wouldn't put Packages in
> there. Those pages are updated frequently enough that I see little reason
> in having them indexed by outsiders.
It doesn't matter that pages are updated often. What it does matter is that
URL seldom change. So search engines will take users to the proper url, and
uptodate information will be shown. If I remember well, I've had put the
meta information in the pages so that they show well in the search engines.
It's really a pitty to exclude them.
Reply to: