Re: Load on udd.d.o
On 24/10/11 at 14:16 +0200, Martin Zobel-Helas wrote:
> Hi,
>
> [Please note that this is my very personal opinion!]
>
> On Mon Oct 24, 2011 at 10:06:11 +0800, Paul Wise wrote:
> > On Sun, Oct 23, 2011 at 7:46 PM, Peter Palfrader wrote:
> > > On Sun, 23 Oct 2011, Lucas Nussbaum wrote:
> > >
> > >> But the main problem, anyway, is that samosa is a bit low on RAM (only 6
> > >> GB), and a bit slow on I/O. In the long term, it would be useful to move
> > >> UDD to a faster box...
> > >
> > > We probably don't have anything like that tho.
> >
> > Is this something that the publicity team could help with; putting out
> > a call for donations like was done for snapshot.debian.org?
>
> Throwing money/newer hardware on this will not help for something that is
> literally broken by design. You can not expect, if everyone[1] can do any
> SQL query on UDD, this will perform nicely. You will never be able to do
> proper indexing for every possible query. Users are and ever will be
> insane. If persons really want to do special queries, UDD offers the
> SQL dump for download so they could run this on their local machine.
>
> While investing in new hardware might make UDD a bit faster, it will
> not solve the underlying problem and performance issues will either not
> go away at all or return very quickly.
>
> If Debian wants to invest some of its funds in renewing our gear there
> probably are more important services to us and our users that could
> benefit from new HW. For instance putting bugs.debian.org on SSDs might
> be a really nice idea. Having it on machines covered by warranty would
> certainly not be wrong either.
I'm not going to argue the respective importances of the BTS and UDD.
But saying that UDD is broken by design, thus should not be given faster
hardware, and then saying that the BTS should get faster hardware, is a
bit strange. The BTS uses a myriad of small files to store the bugs
data, and using a proper database could probably remove the need for
faster hardware, as well as bring a lot of new features (proper
multi-criteria search, for example). Now, I understand that this
requires quite large changes in the BTS, and that we don't have anybody
willing to work on them.
Anyway, I agree that UDD cannot be fast by design. We just need to make
it sufficiently fast to be useful. One way to move forward could be to
have two instances of UDD:
- one hosted on samosa, that only does the importing and the execution
of "official" services that run reasonable queries for which UDD has
been optimized (by adding indexes, etc).
- one on another machine (alioth?), that would be available to everyone
to hack on UDD-based tools. That one could have a strict query
duration limit.
Also, wouldn't it be possible to just add more RAM to samosa ? It only
has 6 GB, and RAM isn't that expensive nowadays. With <$1000, we could
probably get a significant performance increase.
Note that the recent load problems were apparently caused by a lack of
VACUUMing in the DB. I'm not sure why auto-vacuum doesn't work as
expected, but I'm going to cron a VACUUM FULL once per month.
This doesn't solve the root issue, though.
(Andreas, could you confirm that the situation improved for you too?)
Lucas
Reply to: