On Tue, Jan 20, 2026 at 09:47:34PM +1100, Svetlana Tkachenko wrote: > > but I suppose you need to compromise with LLM > > robots going wild. > > Are they not required to follow do_not_track http headers or robots.txt ? If LLM robots do not obey these instructions, they should be probably reported to their hosting provider. They don't always care. Their hosting provider isn't always in position to care. There's enough betting money in this pool to motivate actors to break and/or creatively bend the rules. Recommended reading: https://medium.com/@kolla.gopi/the-cloudflare-perplexity-standoff-why-robots-txt-is-broken-for-the-ai-era-1b9d309bdc2b Cheers -- t
Attachment:
signature.asc
Description: PGP signature