Re: Currently web sentinel on Alioth is not up to date (Was: UDD scripts hammering the DB?)

[Andreas Tille <andreas@an3as.eu>]

Hi,

after some weeks of basically SQL coding I can come up with the first
part of the good news:  The tasks pages of the web sentinel are more or
less functional and regularly updated again.  The code rewrite has
reduced the number of SQL queries of the order of 10000 per Blend to the
order of 100 per Blend and it became only dependant from the number of
tasks independantly how many dependencies are inside the task (because
these are queried in one rush).  This has speeded up the rendering of
tasks pages by a factor of 4 on blends.debian.net but much more
importantly at about the same factor on Alioth - especially for larger
Blends which were causing the longest delay formerly.  

When working on the new code I detected some issues in the old code which
are fixed now but there are also some minor issues left in the new code.
These are:

  - For developer names with non-ASCII characters sometimes the encoding
    is broken (needs to be fixed)
  - In case there is more than one screenshot per package provide these
    are in no specific order (but they should rather ordered according to
    the version of package they are belonging to)
  - Not a real problem but I want to mention it for those who wonder:
    The sorting is a bit different (SQL seems to have a different handling
    of '-' compared to sorting in Python; I will leave this as is)
  - Some translations are not displayed because formerly I took any
    translation but now the translation is used which fits the version of
    the English page and this might be missing.  I will probably add a
    button "Update old translation" which will reflect this issue but
    currently the I18n / DDTP people have serios problems so I'll wait a
    bit here.

If you might detect further problems on the tasks pages or something looks
differently from what you expect just tell me.

I need to work on the bugs pages now after I have pushed the task pages
back into operation again and then I will come back to an issue I
planned to realise for a longer time where I want to send mails about
outdated packages to the packaging lists.  This was close to be finished
before the Alioth service had to be stopped.

BTW, here are some stats: old/new = time in seconds to render the tasks
pages with old/new code running on Alioth, ratio is the quotient of
old/new, blends.d.o is the new job running on blends.debian.org with the
local copy of UDD.  Behind ratio I marked not so good performance of new
code with '-' and good performance with '+' / '++':

                         old    new     ratio	  blends.d.o
brdesktop                1200   1126    0,94 - 	 	 82
debian-accessibility      447    965    2,16 --          45
debian-edu              17267   2453    0,14 ++         411
debian-ezgo              1894    777    0,41 +           73
debian-gis                803    299    0,37 +           34
debian-imaging            534    207    0,39 +           32
debian-junior            1136    632    0,56 +           79
debian-lex                173    230    1,33 -           18
debian-med               3570   1122    0,31 ++         121
debian-multimedia        1522    386    0,25 ++          78
debian-science           7044   1271    0,18 ++         257
debichem                  326   216     0,66             29

all                     35916   9684    0,27

Luckily the larger Blends (edu, med, science) enhanced in speed mostly
drastically.  I think the database query performance is fine now the lag
behind a run of the code compared to a local UDD (as in blends.d.o) is
because of intercontinental data transfer and a probably way lower
performance of alioth (which has a lot more to do than just rendering
tasks pages).

Kind regards

     Andreas.

-- 
http://fam-tille.de

On Wed, Dec 29, 2010 at 08:23:57AM +0100, Andreas Tille wrote:
> [for those on the list who might wonder why the web sentinel pages on
>  {blends,debian-med}.alioth.debian.org are not updated any more: the
>  pages generating code was hammering UDD with a lot of queries and was
>  disabled by DSA.]
> 
> On Tue, Dec 28, 2010 at 05:00:30PM -0500, Yaroslav Halchenko wrote:
> >
> > On Tue, 28 Dec 2010, Andreas Tille wrote:
> > > > I just wondered if there was any progress in bringing Debian science
> > > > blends "online" since last generation date is 12th of Dec.
> > > I'm working on reducing the number of queries in a limited time frame
> > > (to avoid conflicts with family holidays) and can confirm that I'm
> > > making some progress with the UDD queries.  I'm now more or less able to
> > > make one large query per task which reduces the number of queries by
> > > about two orders of magnitude.  So the work is promissing but
> > > non-functional and thus I did not yet commited the code to SVN.
> > sounds great!
> >
> > > If you really need updated Debian Science pages you might either look at
> > > blends.debian.net/science, rsync these pages to alioth (I did not
> > ha -- I didn't know that my unspoken desire is somewhat dealt with
> > already -- I could have all blends/tasks pages available from somewhat
> > more uniform location, e.g.:
> > 
> > http://blends.debian.net/science/tasks/
> > http://blends.debian.net/debian-med/tasks/
> > instead of 
> > http://debian-med.alioth.debian.org/tasks/
> 
> Yes.  This was announced at some point in time on the relevant mainling
> lists but I did not made to much noise about it because this is the
> place where I'm testing all my code and thus it might be broken for some
> intermediate period.
>  
> > cool ;)  so, are those @ blends.debian.net always up-to-date or that is just
> > temporarily?  Why those locations are not primary?
> 
> The page should be online always.  However:
>   - It is not maintained by DSA (thus there is higher potential that the
>     machine might be broken).
>   - UDD is updated only once a day and not twice a day so the resulting
>     pages might show some delay in certain circumstances
>   - Once I'm testing the new code the pages might be broken or at least
>     different from what should be shown
>  
> > > realised this plan to rather work on the new code) or simply redirect
> > > from Alioth to blends.debian.net temporarily.  We just need to stop
> > > redirection / rsyncing once I start testing the new code (hopefully
> > > at beginning of new year).
> > 
> > I will better take a passive side -- I can direct people to
> > blends.debian.net for now whenever that comes desired/necessary
> 
> As long as you tell them explicitely that these are not the official
> pages and there is no guarantee that these will be broken this is fine.
> 
> Kind regards
> 
>     Andreas.
> 
> -- 
> http://fam-tille.de
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-blends-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/20101229072357.GA15408@an3as.eu
> 
> 

-- 
http://fam-tille.de