On 05/23/2019 05:36 AM, Matt Owen wrote: >> -----Original Message----- >> From: Cameron Kaiser <spectre@floodgap.com> >> Sent: 23 May 2019 01:29
Speaking only for V-2, and not for any other crawlers.I think it would be a good idea for an informal RFC type document specifying how gopher crawlers should work - that way everyone can design to the same standard. A sitemap file could just be a list of selector URIs, one per line. Keep it simple.
I think that is a very good idea. I've been toying with the idea of making a Gopher crawler of my own and it would be wonderful to have a robots.txt specification reference document (no matter how informal) to refer to. Likewise for sitemaps.
-Dave Gauer