Bug#233564: marked as done (Developer.php performance (+patch))
Your message dated Sun, 22 Feb 2004 16:26:43 +0100
with message-id <[🔎] 20040222152643.GD2314@tass.genibel.org>
and subject line Developer.php performance (+patch)
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Debian bug tracking system administrator
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 18 Feb 2004 18:30:12 +0000
>From jeroen@wolffelaar.nl Wed Feb 18 10:30:12 2004
Return-path: <jeroen@wolffelaar.nl>
Received: from a-eskwadraat.nl [131.211.34.218] (mail)
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1AtWSZ-0008HK-00; Wed, 18 Feb 2004 10:30:11 -0800
Received: from jeroen by A-Eskwadraat.nl with local (Exim 3.35 #1 (Debian))
id 1AtWST-0007bi-00; Wed, 18 Feb 2004 19:30:05 +0100
Date: Wed, 18 Feb 2004 19:30:05 +0100
To: Jeroen van Wolffelaar <jeroen@wolffelaar.nl>
Cc: debian-qa@lists.debian.org, submit@bugs.debian.org
Subject: Re: Developer.php performance (+patch)
Message-ID: <[🔎] 20040218183005.GH2680@A-Eskwadraat.nl>
References: <[🔎] 20040217142956.GF2680@A-Eskwadraat.nl>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="tcC6YSqBgqqkz7Sb"
Content-Disposition: inline
In-Reply-To: <[🔎] 20040217142956.GF2680@A-Eskwadraat.nl>
User-Agent: Mutt/1.3.28i
X-Habeas-SWE-1: winter into spring
X-Habeas-SWE-2: brightly anticipated
X-Habeas-SWE-3: like Habeas SWE (tm)
X-Habeas-SWE-4: Copyright 2002 Habeas (tm)
X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this
X-Habeas-SWE-6: email in exchange for a license for this Habeas
X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant
X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this
X-Habeas-SWE-9: mark in spam to <http://www.habeas.com/report/>.
From: Jeroen van Wolffelaar <jeroen@wolffelaar.nl>
Delivered-To: submit@bugs.debian.org
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_02_18
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-7.8 required=4.0 tests=HABEAS_SWE,HAS_PACKAGE,
HTML_10_20,HTML_MESSAGE autolearn=no
version=2.60-bugs.debian.org_2004_02_18
X-Spam-Level:
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Package: qa.debian.org
Severity: minor
Tags: patch
On Tue, Feb 17, 2004 at 03:29:56PM +0100, Jeroen van Wolffelaar wrote:
> Hi,
>
> developer.php is quite slow sometimes, especially on people with much
> packages, you can notice delays, it even timeouts sometimes.
Thanks to Martin Michlmayr, I was able to test developer.php myself.
Find attached a patch with these changes:
- Fix all PHP warning that were in the script (but surpressed by the
server)
- Read bug information using a tree lookup (db2) rather than linear
search
- Fix a off-by-one mistake and a missing $, causing respectively a
warning and wrong comaintainer-behaviour (These were easy to catch
with warnings enabled)
- Add a 'Page generated in' item at the bottom
And a 'convert-bugs-to-db' script, to be run (after fixing paths)
immediately after bugs.txt is updated (new db is moved over old, no race
conditions).
Note: if for some reason qa.debian.org isn't running woody (where I
tested it on), you might need to change db2 to db3 or db4, I worked
around the current bug in php4 (but you do need an uptodate php4 for it
to work).
While with the current implementation the developer.php page for
packages@qa.debian.org fails with a 30sec timelimit timeout, that same
page takes only 0.1 seconds with this updated implementation.
Test URL: http://jeroen.a-eskwadraat.nl/misc/qa/developer.php
(not updated with new info, unless someone can get me new extract &
bugs.txt files on the required intervals).
A developer.php page with freetext developer name still takes about
700ms constant time, I plan to look after that later, but currently other
things on my TODO list have more priority.
An improvement that shouldn't be hard, is making developer.php able to
generate a package-table based on other keys that the maintainer, like
based on task, or on a certain package with all his dependencies, stuff
like that. IMHO quite useful tracking down stuff.
--Jeroen
--
Jeroen van Wolffelaar
Jeroen@wolffelaar.nl (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="performance.patch"
Index: developer.wml
===================================================================
RCS file: /cvs/qa/wml/developer.wml,v
retrieving revision 1.49
diff -u -r1.49 developer.wml
--- developer.wml 20 Nov 2003 14:09:21 -0000 1.49
+++ developer.wml 18 Feb 2004 18:11:49 -0000
@@ -28,6 +28,7 @@
</style>
<?
+$time = microtime();
$branch = array ("main",
"contrib",
@@ -322,7 +323,7 @@
}
}
$global_count -= $result;
- }elseif(comaint == "only") {
+ }elseif($comaint == "only") {
$result = 0;
for ($i = 0; $i < strlen($packages); $i++) {
if ($packages[$i] == '#') {
@@ -410,18 +411,15 @@
$priority = $results[6];
$section = $results[7];
- $fp = fopen("$prefix/bugs.txt", "r");
- $bug_list = explode(" ", "0(0) 0(0) 0(0) 0(0)");
- $temp = preg_replace("/\+/", "_", $package);
- while(!feof($fp)) {
- $line = fgets($fp, "512");
- $line = preg_replace("/\+/", "_", $line);
- if(preg_match("/^$temp\s+(\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\))/", $line, $results)) {
- $bug_list = explode(" ", $results[1]);
- break;
- }
+ // make $bugsdb link persistent during this pagerequest, we don't lock, so
+ // this doesn't matter at all
+ static $bugsdb;
+ if (!$bugsdb) $bugsdb = dba_open("$prefix/bugs.db", 'r-', 'db2');
+
+ if (!($bug_list = dba_fetch($package, $bugsdb))) {
+ $bug_list = '0(0) 0(0) 0(0) 0(0)';
}
- fclose($fp);
+ $bug_list = explode(' ', $bug_list);
if(strcmp($version_stable, "-") == 0 || !debcheckavailable('stable', $package)) {
$stable_dep = "<small>-</small>";
@@ -499,12 +497,12 @@
}
}
- if ($background == 1) {
+ if (@$background) {
print '<tr align="center">';
- $background=0;
+ $background=false;
}else{
print '<tr align="center" bgcolor="#dcdcdc">';
- $background=1;
+ $background=true;
}
?>
<td width="15%">
@@ -529,8 +527,8 @@
$bugs_all = "http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=";. rawurlencode($package);
$bug_count = $bug_list[0] + $bug_list[1] + $bug_list[2] + $bug_list[3];
$real_bug_count = 0;
- for ($i = 0; $i <= count($bug_list); $i++) {
- preg_match("/\d+\((\d+)\)/", $bug_list[$i], $count_array);
+ foreach ($bug_list as $bug_item) {
+ preg_match("/\d+\((\d+)\)/", $bug_item, $count_array);
$real_bug_count += $count_array[1];
}
if($bug_count == 0) { ?>
@@ -607,7 +605,7 @@
<td>
<small>
<?
- if($all == 1 || preg_match("/non-free/", $branch) || (strcmp($version_unstable, "-") == 0)) {
+ if(@$all || preg_match("/non-free/", $branch) || (strcmp($version_unstable, "-") == 0)) {
print "-";
$all = 0;
}else{
@@ -670,10 +668,10 @@
<?
}
-if($comaint == "") {
+if(!@$comaint) {
$comaint = "yes";
}
-if($excuse != "") {
+if(@$excuse) {
$char = substr($excuse, 0, 1);
$excuse = preg_replace("/ /", "+", $excuse);
if(file_exists("$prefix/excuses/$char/$excuse")) {
@@ -683,7 +681,7 @@
}else{
print "<h2>No excuse for $excuse</h2>";
}
-}else if($gpg_key != "") {
+}else if (@$gpg_key) {
$contents = file("$prefix/extract");
$printed = 0;
@@ -715,7 +713,7 @@
}
printBackLink();
-}else if($login != "") {
+}else if(@$login) {
$fp = fopen("$prefix/extract", "r");
$contents = fread($fp, filesize ("$prefix/extract"));
@@ -753,7 +751,7 @@
}
fclose($fp);
printBackLink();
-}else if ($package != "") {
+}else if (@$package) {
$fp = fopen("$prefix/extract", "r");
$found = 0;
if(preg_match("/\+\+/", $package)) {
@@ -796,7 +794,7 @@
printBackLink();
}else{
print_form();
- if($all) {
+ if(@$all) {
print_all();
}else{
print_query_all();
@@ -818,5 +816,10 @@
General information: <? print $general; ?><br>
Bug information: <? print $bugs; ?><br>
General information is updated once a day.<br>
-Bug information is updated every 15 minutes.
+Bug information is updated every 15 minutes.<br>
+<?
+list($micro,$sec) = explode(' ', microtime());
+list($smicro,$ssec) = explode(' ', $time);
+echo "Time needed to generate page: ".($sec-$ssec+$micro-$smicro)."s";
+?>
</small></p>
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=convert-bugs-to-db
#!/usr/bin/php4 -q
<?
$db = dba_open('bugs.db.new', 'n-', 'db2')
or die("Opening bugs.db failed");
$fp = fopen("bugs.txt", "r");
while ($line = fgets($fp, 512)) {
list($package, $bugdata) = preg_split('/\\s+/', $line, 2);
$bugdata = trim($bugdata);
dba_insert($package, $bugdata, $db)
or die("Insert for $package failed");
}
dba_optimize($db)
or die("Optimize failed");
dba_close($db);
rename('bugs.db.new', 'bugs.db');
--tcC6YSqBgqqkz7Sb--
---------------------------------------
Received: (at 233564-done) by bugs.debian.org; 22 Feb 2004 15:27:19 +0000
>From igor@genibel.org Sun Feb 22 07:27:19 2004
Return-path: <igor@genibel.org>
Received: from home.genibel.org (tass) [81.56.84.10]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1AuvVm-0001o2-00; Sun, 22 Feb 2004 07:27:19 -0800
Received: from igor by tass with local (Exim 4.30)
id 1AuvVD-0008DW-HK; Sun, 22 Feb 2004 16:26:43 +0100
Date: Sun, 22 Feb 2004 16:26:43 +0100
From: Igor Genibel <igor@genibel.org>
To: Jeroen van Wolffelaar <jeroen@wolffelaar.nl>
Cc: debian-qa@lists.debian.org, 233564-done@bugs.debian.org
Subject: Re: Developer.php performance (+patch)
Message-ID: <[🔎] 20040222152643.GD2314@tass.genibel.org>
Mail-Followup-To: Jeroen van Wolffelaar <jeroen@wolffelaar.nl>,
debian-qa@lists.debian.org, 233564-done@bugs.debian.org
References: <[🔎] 20040217142956.GF2680@A-Eskwadraat.nl> <[🔎] 20040218183005.GH2680@A-Eskwadraat.nl>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature"; boundary="3uo+9/B/ebqu+fSQ"
Content-Disposition: inline
In-Reply-To: <[🔎] 20040218183005.GH2680@A-Eskwadraat.nl>
Organisation: TuxFamily free hosting for free people
X-URL: http:/www.genibel.org/
User-Agent: Mutt/1.5.5.1+cvs20040105i
Delivered-To: 233564-done@bugs.debian.org
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_02_22
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=0.0 required=4.0 tests=none autolearn=no
version=2.60-bugs.debian.org_2004_02_22
X-Spam-Level:
--3uo+9/B/ebqu+fSQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
* Jeroen van Wolffelaar <jeroen@wolffelaar.nl> [2004-02-18 19:30:05 +0100]:
I applied the patch and modified backend scripts in order to have these
functionalities into DDPO.
This really increase the performances and permits users that have a lot
of package to display correctly their page without timeouts.
Thanks a lot Jeroen ;)
--=20
Igor Genibel=20
http://www.answare.fr/ igor.genibel@eds.com
http://www.tuxfamily.org/ igor@tuxfamily.net
http://people.debian.org/~igenibel/ igenibel@debian.org
GPG: 1024D/1648F6C8
--3uo+9/B/ebqu+fSQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAOMoz+xgdMBZI9sgRAsYZAKCTLaZVbL2ct9kFD7oWqZSidwtQrgCfWl0F
0Qq6t7miFdus9mdVs0v/yaw=
=OuE8
-----END PGP SIGNATURE-----
--3uo+9/B/ebqu+fSQ--
Reply to: