Bug#67637: marked as done (www.debian.org: robots.txt has too many entries)
Your message dated Wed, 25 Oct 2000 20:09:26 +0200
with message-id <[🔎] 20001025200926.A8317@cibalia.gkvk.hr>
and subject line robots.txt on www.debian.org
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Darren Benham
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 24 Jul 2000 03:37:04 +0000
>From nick@debian.org Sun Jul 23 22:37:04 2000
Return-path: <nick@debian.org>
Received: from postino4.prima.com.ar [200.42.0.162]
by master.debian.org with esmtp (Exim 3.12 2 (Debian))
id 13GZ38-0007Wz-00; Sun, 23 Jul 2000 22:37:03 -0500
Received: from newton (host028250.ciudad.com.ar [200.42.28.250])
by postino4.prima.com.ar (8.9.3/8.9.1) with ESMTP id AAA78952
for <submit@bugs.debian.org>; Mon, 24 Jul 2000 00:36:55 -0300 (ART)
Received: from nick by newton with local (Exim 3.12 #1 (Debian GNU/Linux))
id 13GZ3T-0000pQ-00; Mon, 24 Jul 2000 00:37:23 -0300
From: Nicolás Lichtmaier <nick@debian.org>
Subject: www.debian.org: robots.txt has too many entries
To: submit@bugs.debian.org
X-Mailer: bug 3.3.4
Message-Id: <E13GZ3T-0000pQ-00@newton>
Date: Mon, 24 Jul 2000 00:37:23 -0300
Delivered-To: submit@bugs.debian.org
Package: www.debian.org
Version: 20000724
Severity: normal
Current /robots.txt prohibits indexing of many resources that should be
indexed.
User-agent: *
Disallow: /Bugs/
Disallow: /Lists-Archives/
Disallow: /Packages/
Disallow: /security/
Disallow: /news.html
Disallow: /consultants.html
Disallow: /consultant_info/
Disallow: /people.html
Disallow:
I don't see any reason to have *any* of these entries. But even so:
/Packages/ and /Lists-Archives/ are completelly out of place here (perpetual
URLs pointing to useful, indexable content).
---------------------------------------
Received: (at 67637-close) by bugs.debian.org; 25 Oct 2000 18:05:03 +0000
>From joy@cibalia.gkvk.hr Wed Oct 25 13:05:03 2000
Return-path: <joy@cibalia.gkvk.hr>
Received: from cibalia.gkvk.hr [::ffff:161.53.211.3]
by master.debian.org with esmtp (Exim 3.12 1 (Debian))
id 13oUv7-00016p-00; Wed, 25 Oct 2000 13:05:02 -0500
Received: from joy by cibalia.gkvk.hr with local (Exim 3.12 #1 (Debian))
id 13oUzO-0002BQ-00; Wed, 25 Oct 2000 20:09:26 +0200
Date: Wed, 25 Oct 2000 20:09:26 +0200
From: Josip Rodin <joy@cibalia.gkvk.hr>
To: 67637-close@bugs.debian.org, 67637@bugs.debian.org
Subject: robots.txt on www.debian.org
Message-ID: <[🔎] 20001025200926.A8317@cibalia.gkvk.hr>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
User-Agent: Mutt/1.0.1i
Delivered-To: 67637-close@bugs.debian.org
Hi,
Nicolas Lichtmaier wrote:
> > > Current /robots.txt prohibits indexing of many resources that should
> > > be indexed.
> > >
> > > /Packages/ and /Lists-Archives/ are completelly out of place here
> > > (perpetual URLs pointing to useful, indexable content).
> >
> > Probably because search engines would overload www.debian.org otherwise.
>
> This can be easily checked.. are there any log analisis that has shown
> this?
A few minutes ago master suffered a DoS (sort of, the load was >80 and you
couldn't do anything) by googlebot which was accessing all the bug reports
and stuff, because the robots.txt file was missing on klecker (it was
forgotten during the move).
I've put the file back on klecker, and removed the obsolete entries (i.e.
the files that don't exist), but I'm definitely leaving Bugs/ and Packages/
in there so that stuff like this doesn't happen anymore.
--
Digital Electronic Being Intended for Assassination and Nullification
Reply to: