Bug#771121: security-tracker: often returns "502 Proxy Error"
* Francesco Poli:
>> The fix is only temporary because at a certain point, broken bots
>> scraping information from the HTML pages may overload the server
>> again. There are several of them requesting the same CVE page again
>> and again, but it's difficult tell what's actually going because of
>> the privacy-enhanced logging.
>
> One of the bots scraping information from the tracker HTML pages is
> actually a script that I manually run once a day (or less)... :-/
> It just requests three tracker pages and does everything else locally,
> hence I hope it does not cause any significant overload.
No, these bots are doing something completely different, they downlaod
specific CVE pages, mostly older ones which do not seem in any way
particularly relevant.
As a side effect of my fix, the request rate has increased further,
but I expect that the current system can keep up with a sustained rate
of around 10 requests per second at least (assuming there are no
threading bugs in the code), maybe even more, but beyond that, we need
caching. We are currently at roughly 1 request per second (up from
0.5). The deployment already uses mod_proxy, but the web application
does not set proper caching headers.
Reply to: