Developer.php performance (+patch)

[Jeroen van Wolffelaar <jeroen@wolffelaar.nl>]

Hi,

developer.php is quite slow sometimes, especially on people with much
packages, you can notice delays, it even timeouts sometimes.

The main performance hog is a linear (O(n)) search in bugs.txt for every
row==package.

On my tests, a linear search in a 8000 line (number of source packages)
bugs.txt would take 160ms on average (for every package), while a db2
lookup, including opening and such, takes 0.09ms on average. The huge
timehit for the linear search is also parially because of two perl
regex's for every line.

cvs:qa/data/cronjobs/ddpo.bug is out of sync with reality unfortunately.

Anyway, after a new bugs.txt is generated, execute this php script
(should take about 200ms with 8000 packages, so isn't much):

<SCRIPT>
#!/usr/bin/php4
<?php
// Copyright 2004 Jeroen van Wolffelaar <jeroen@wolffelaar.nl>
// This script may be redistributed under the conditions of the GPL
// 17 feb 2004

unlink('bugs.db.new');
$db = dba_open('bugs.db.new', 'cwdt', 'db2')
        or die("Opening bugs.db.new.db failed");
$fp = fopen("bugs.txt", "r");
 
while ($line = fgets($fp, 512)) {
        list($package, $bugdata) = preg_split('/\\s+/', $line, 2);
        $bugdata = trim($bugdata);
		// optionally: do sanity checking here, so developer.php can
		// rely on it. But my guess is that sanity checking is unneeded
 
        dba_insert($package, $bugdata, $db)
                or die("Insert for $package failed");
}
 
dba_optimize($db)
        or die("Optimize failed");
 
dba_close($db);

move('bugs.db.new', 'bugs.db');

</SCRIPT>

and then modify developer.wml as such:

diff -u -r1.49 developer.wml
--- developer.wml       20 Nov 2003 14:09:21 -0000      1.49
+++ developer.wml       17 Feb 2004 14:24:07 -0000
@@ -410,18 +410,12 @@
   $priority = $results[6];
   $section = $results[7];
  
-  $fp = fopen("$prefix/bugs.txt", "r");
-  $bug_list = explode(" ", "0(0) 0(0) 0(0) 0(0)");
-  $temp = preg_replace("/\+/", "_", $package);
-  while(!feof($fp)) {
-       $line = fgets($fp, "512");
-       $line = preg_replace("/\+/", "_", $line);
-       if(preg_match("/^$temp\s+(\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\))/", $line, $results)) {
-               $bug_list = explode(" ", $results[1]);
-               break;
-       }
+  $db = dba_open('bugs.db', 'rd', 'db2');
+  if (!($bug_list = dba_fetch($package, $db))) {
+         $bug_list = explode(' ', '0(0) 0(0) 0(0) 0(0)');
   }
-  fclose($fp);
+  dba_close($db);
+  $bug_list = explode(' ', $buglist);
  
   if(strcmp($version_stable, "-") == 0 || !debcheckavailable('stable', $package)) {
     $stable_dep = "<small>-</small>";




Not tested of course, as I'm not able to, but this is a really
straightforward fix.

--Jeroen

-- 
Jeroen van Wolffelaar
Jeroen@wolffelaar.nl (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl