[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Developer.php performance (+patch)



Package: qa.debian.org
Severity: minor
Tags: patch

On Tue, Feb 17, 2004 at 03:29:56PM +0100, Jeroen van Wolffelaar wrote:
> Hi,
> 
> developer.php is quite slow sometimes, especially on people with much
> packages, you can notice delays, it even timeouts sometimes.

Thanks to Martin Michlmayr, I was able to test developer.php myself.
Find attached a patch with these changes:
- Fix all PHP warning that were in the script (but surpressed by the
  server)
- Read bug information using a tree lookup (db2) rather than linear
  search
- Fix a off-by-one mistake and a missing $, causing respectively a
  warning and wrong comaintainer-behaviour (These were easy to catch
  with warnings enabled)
- Add a 'Page generated in' item at the bottom

And a 'convert-bugs-to-db' script, to be run (after fixing paths)
immediately after bugs.txt is updated (new db is moved over old, no race
conditions).

Note: if for some reason qa.debian.org isn't running woody (where I
tested it on), you might need to change db2 to db3 or db4, I worked
around the current bug in php4 (but you do need an uptodate php4 for it
to work).

While with the current implementation the developer.php page for
packages@qa.debian.org fails with a 30sec timelimit timeout, that same
page takes only 0.1 seconds with this updated implementation.
Test URL: http://jeroen.a-eskwadraat.nl/misc/qa/developer.php
(not updated with new info, unless someone can get me new extract &
bugs.txt files on the required intervals).

A developer.php page with freetext developer name still takes about
700ms constant time, I plan to look after that later, but currently other
things on my TODO list have more priority.

An improvement that shouldn't be hard, is making developer.php able to
generate a package-table based on other keys that the maintainer, like
based on task, or on a certain package with all his dependencies, stuff
like that. IMHO quite useful tracking down stuff.

--Jeroen

-- 
Jeroen van Wolffelaar
Jeroen@wolffelaar.nl (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl
Index: developer.wml
===================================================================
RCS file: /cvs/qa/wml/developer.wml,v
retrieving revision 1.49
diff -u -r1.49 developer.wml
--- developer.wml	20 Nov 2003 14:09:21 -0000	1.49
+++ developer.wml	18 Feb 2004 18:11:49 -0000
@@ -28,6 +28,7 @@
 </style>
 
 <?
+$time = microtime();
 
 $branch = array ("main", 
 	       "contrib", 
@@ -322,7 +323,7 @@
          }
      }
      $global_count -= $result;
-  }elseif(comaint == "only") {
+  }elseif($comaint == "only") {
      $result = 0;
      for ($i = 0; $i < strlen($packages); $i++) {
          if ($packages[$i] == '#') {
@@ -410,18 +411,15 @@
   $priority = $results[6];
   $section = $results[7];
 
-  $fp = fopen("$prefix/bugs.txt", "r");
-  $bug_list = explode(" ", "0(0) 0(0) 0(0) 0(0)");
-  $temp = preg_replace("/\+/", "_", $package);
-  while(!feof($fp)) {
-  	$line = fgets($fp, "512");
-	$line = preg_replace("/\+/", "_", $line);
-	if(preg_match("/^$temp\s+(\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\))/", $line, $results)) {
-		$bug_list = explode(" ", $results[1]);
-		break;
-	}
+  // make $bugsdb link persistent during this pagerequest, we don't lock, so
+  // this doesn't matter at all
+  static $bugsdb;
+  if (!$bugsdb) $bugsdb = dba_open("$prefix/bugs.db", 'r-', 'db2');
+
+  if (!($bug_list = dba_fetch($package, $bugsdb))) {
+	  $bug_list = '0(0) 0(0) 0(0) 0(0)';
   }
-  fclose($fp);
+  $bug_list = explode(' ', $bug_list);
 
   if(strcmp($version_stable, "-") == 0 || !debcheckavailable('stable', $package)) {
     $stable_dep = "<small>-</small>";
@@ -499,12 +497,12 @@
     }
   }
 
-  if ($background == 1) {
+  if (@$background) {
     print '<tr align="center">';
-    $background=0;
+    $background=false;
   }else{
     print '<tr align="center" bgcolor="#dcdcdc">';
-    $background=1;
+    $background=true;
   }
 ?>
   <td width="15%">
@@ -529,8 +527,8 @@
   $bugs_all = "http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=";. rawurlencode($package);
      $bug_count = $bug_list[0] + $bug_list[1] + $bug_list[2] + $bug_list[3];
      $real_bug_count = 0;
-     for ($i = 0; $i <= count($bug_list); $i++) {
-        preg_match("/\d+\((\d+)\)/", $bug_list[$i], $count_array);
+     foreach ($bug_list as $bug_item) {
+        preg_match("/\d+\((\d+)\)/", $bug_item, $count_array);
 	$real_bug_count += $count_array[1];
 }
 	if($bug_count == 0) { ?>
@@ -607,7 +605,7 @@
   <td>
   <small>
 <?
-  if($all == 1 || preg_match("/non-free/", $branch) || (strcmp($version_unstable, "-") == 0)) {
+  if(@$all || preg_match("/non-free/", $branch) || (strcmp($version_unstable, "-") == 0)) {
     print "-";
     $all = 0;
   }else{
@@ -670,10 +668,10 @@
 <?
     }
 
-if($comaint == "") {
+if(!@$comaint) {
   $comaint = "yes";
 }
-if($excuse != "") {
+if(@$excuse) {
 	$char = substr($excuse, 0, 1);
 	$excuse = preg_replace("/ /", "+", $excuse);
 	if(file_exists("$prefix/excuses/$char/$excuse")) {
@@ -683,7 +681,7 @@
 	}else{
 		print "<h2>No excuse for $excuse</h2>";
 	}
-}else if($gpg_key != "") {
+}else if (@$gpg_key) {
   $contents = file("$prefix/extract");
 
   $printed = 0;
@@ -715,7 +713,7 @@
   }
   
   printBackLink();
-}else if($login != "") {
+}else if(@$login) {
   $fp = fopen("$prefix/extract", "r");
   $contents = fread($fp, filesize ("$prefix/extract"));
 
@@ -753,7 +751,7 @@
   }
   fclose($fp);
   printBackLink();
-}else if ($package != "") {
+}else if (@$package) {
   $fp = fopen("$prefix/extract", "r");
   $found = 0;
   if(preg_match("/\+\+/", $package)) {
@@ -796,7 +794,7 @@
   printBackLink();
 }else{
   print_form();
-  if($all) {
+  if(@$all) {
     print_all();
   }else{
     print_query_all();
@@ -818,5 +816,10 @@
 General information: <? print $general; ?><br>
 Bug information: <? print $bugs; ?><br>
 General information is updated once a day.<br>
-Bug information is updated every 15 minutes.
+Bug information is updated every 15 minutes.<br>
+<?
+list($micro,$sec) = explode(' ', microtime());
+list($smicro,$ssec) = explode(' ', $time);
+echo "Time needed to generate page: ".($sec-$ssec+$micro-$smicro)."s";
+?>
 </small></p>
#!/usr/bin/php4 -q
<?

$db = dba_open('bugs.db.new', 'n-', 'db2')
	or die("Opening bugs.db failed");
$fp = fopen("bugs.txt", "r");

while ($line = fgets($fp, 512)) {
	list($package, $bugdata) = preg_split('/\\s+/', $line, 2);
	$bugdata = trim($bugdata);

	dba_insert($package, $bugdata, $db)
		or die("Insert for $package failed");
}

dba_optimize($db)
	or die("Optimize failed");

dba_close($db);

rename('bugs.db.new', 'bugs.db');


Reply to: