--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: Orphaned packages list is outdated (with patch)
- From: Arno Töll <debian@toell.net>
- Date: Wed, 29 Jun 2011 17:45:18 +0200
- Message-id: <4E0B488E.7030607@toell.net>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Package: qa.debian.org
Severity: important
Hello,
I noticed the list of orphaned packages with wrong maintainers is
outdated [1]. jwilk was kindly enough to tell me, this is because the
LDAP server used to gather bug data is not accessible (anymore?). Output
of the script right now is:
$ /srv/qa.debian.org/data/wnpp/gen-orphaned
Cannot connect to LDAP server
For the records: the script tries to connect to bts2ldap.debian.net [2]
which seems not to respond anymore. Therefore he suggested to switch to
SOAP (or use UDD) instead. This is what I did and I hence reimplemented
te gen-orphan script [3] to access the public SOAP interface.
It is still loosely based on Michael's original script but it is
essentially a complete rewrite, therefore the diff is not particularly
useful. For the sake of completeness I attached it nonetheless. However
I attached the full source as well which may be better used instead.
Note I wasn't able to test the script on the destination site, but on my
local system only. Let me know if it does not work. I used syntax and
format of [4] as maintainer file, but I had no chance to test whether
the resulting output file parses as WML input.
Finally please note, the WNPP "library" the script used before is in use
for several other scripts as well, e.g. wnpp-lint. Those may be still
broken.
[1] http://qa.debian.org/orphaned.html
[2] http://anonscm.debian.org/viewvc/qa/trunk/data/wnpp/wnpp.py?view=markup
[3]
http://anonscm.debian.org/viewvc/qa/trunk/data/wnpp/gen-orphaned?view=markup
[4] ftp://ftp.debian.org/debian/indices/Maintainers
- --
with kind regards,
Arno Töll
IRC: daemonkeeper on Freenode/OFTC
GnuPG Key-ID: 0x9D80F36D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJOC0iOAAoJEMcrUe6dgPNtpPUP/ibhaYZs5s4x4jJwMpuI9e9S
AzibOFkVZ8AZybjBs6CYtFsJSioRFlGuv2Orr/Ept89N6zVa/v3ADvPXWaxaDau1
HLygjlVTA+nBLijQJ3To4hSgXlbu9Fb1tNR37/QmJ9EtbsNoeB9CQ9P7Zk0rmaze
BEHbB+l365etevXmq3FH7O+I2GFKKRiSnF7edrERBwJMDJy0XbAaFGbm+6MNEkXO
Ce/1KLAVbN1hc3wtPawZpZ1cb1Jq1lodvJoFsLIdkiqq2zqGxbx8ykg72C6dTzGG
lvohJ0BzGj74WuC1sxGa9Kg5Syumy1y/LJAWNvA7vVqsef9pUFb2AELrOR8ub0cg
RKmQ5yMbW7ki1dS9Kddo6N5ZWEiu0T9+5gqg/hpGA2+2P7F1H8L006k9jJdT7T0S
5KQmkOYgP2cyK3hAYcwdQLool0J+P5D7GWe9RB+2hEZhTCEPltTd+mJNu2gobm4q
zV8/lidob68LkrHTuUPhPQgt4BtmlNydsrVBGwPKCIiXsuCpiBIaBAnUggzsCWkM
H7muryfup3zlRE9/1hPKOOCbsh5MqP2ohde1Ed3BSgRRf71wbE1hB+EZz0P9fwrb
pM42kG8t5DP+xb3hW9qmuu4CP8ChyN+uc1jOjciZ6AycDE7V2STdOM4+CBiBRS63
UoHFaVNoCWAlH/cGlTRK
=GD1E
-----END PGP SIGNATURE-----
--- wnpp/gen-orphaned 2011-06-29 17:26:23.945892014 +0200
+++ gen-orphaned/src/gen-orphaned.py 2011-06-29 17:27:56.782352372 +0200
@@ -1,9 +1,22 @@
#!/usr/bin/env python
+# -*- coding: utf-8 -*-
-# Generate the orphan.txt file used by orphaned.wml
-# Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <tbm@cyrius.com>
-# $Id: gen-orphaned 761 2004-04-21 13:46:12Z tbm $
-
+"""
+ Generates the orphan.txt file used by orphaned.wml.
+ This version is basically a complete rewrite from the previous version
+ and was changed to use the SOAP interface to query the BTS instead of the LDAP
+ bridge.
+ It requires the following Debian packages:
+
+ * Python 2.6
+ * python-soappy
+
+ Please change paths configured below if necessary
+"""
+
+# Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <tbm@cyrius.com>
+# 2011 Arno Töll <debian@toell.net>
+#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
@@ -11,102 +24,127 @@
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston,
+# MA 02110-1301, USA.
-import string
+from __future__ import print_function
+
import sys
import re
-import wnpp
+import string
+import SOAPpy
+import time
+
+
+MAINTAINER_FILE = '/org/qa.debian.org/data/ftp/Maintainers'
+OUTPUT_FILE = '/org/qa.debian.org/data/orphaned.txt'
+URL = 'http://bugs.debian.org/cgi-bin/soap.cgi'
+NAMESPACE = 'Debbugs/SOAP'
+
+###
+### Nothing else should be changed below
+###
+
+class OrphanedPackage( object ):
+ def __repr__( self ):
+ return( "%s\n%s\n%d\n%s\n%s\n%d\n\n"
+ % ( self._package, self._maintainer, self._bugid, self._title, self._reporter, self._age ) )
+
+ def __init__( self, package, maintainer, bugid, title, reporter, age ):
+ self._package = package
+ self._maintainer = maintainer
+ self._bugid = bugid
+ self._title = title
+ self._reporter = reporter
+ self._age = age
try:
- input = open('/org/qa.debian.org/data/ftp/Maintainers', 'r')
-except (IOError, OSError):
- print 'Maintainers file not found'
- sys.exit(1)
+ input = open( MAINTAINER_FILE, 'r' )
+except ( IOError, OSError ) as e:
+ print( 'Couldn\'t access maintainer file `%s\': %s' % ( MAINTAINER_FILE, e ) )
+ sys.exit( 1 )
+
maintainers = { }
+orphaned = []
+
for line in input.readlines():
- # Python 1.5 ignores maxsplit in re.split() so this won't work:
- ## package, maintainer = re.split('\s+', line, 2)
- # Thus using the following solution:
- line = string.rstrip(re.sub('\s+', ' ', line))
- package, maintainer = string.split(line, ' ', 1)
- maintainers[package] = maintainer
-input.close()
+ line = string.rstrip( re.sub( '\s+', ' ', line ) )
+ package, maintainer = string.split( line, ' ', 1 )
+ maintainers[package] = maintainer
+input.close()
-ldap_result = wnpp.query([ 'debbugsID', 'debbugsTitle', 'debbugsState', 'debbugsDate', 'debbugsSubmitter' ])
+try:
+ server = SOAPpy.SOAPProxy( URL, NAMESPACE, simplify_objects = 1 )
+ raw_bugs = server.get_status( server.get_bugs( "package", "wnpp" ) )
+except ( Exception ) as e:
+ print( 'Can\'t access SOAP interface on %s (ns: %s): %s' % ( URL, NAMESPACE, e ) )
+ sys.exit( 1 )
+
+if ( not raw_bugs or 'item' not in raw_bugs ):
+ print( "Could not retrieve any bugs (but no apparent error occurred)\n" )
+ sys.exit( 0 )
+
+# Force argument to be a list, SOAPpy returns a dictionary instead of a dictionary list
+# if a single bug was found. Not this were likely after all :>
+if ( not isinstance( raw_bugs['item'], list ) ):
+ raw_bugs['item'] = ( raw_bugs['item'], )
+
+for bugs in raw_bugs['item']:
+ current_bug = bugs['value']
+
+ # Skip packages with illegal subject line
+ parsed_subject = re.match( "(\w+):\s+(\S+)", current_bug['subject'] )
+ if ( not parsed_subject ):
+ continue
+
+ # Skip Non-O bugs
+ #if ( parsed_subject.group( 1 ) not in ( "O", "RFA" ) ):
+ if ( parsed_subject.group( 1 ) != "O" ):
+ continue
+
+ package = parsed_subject.group( 2 ).lower()
+
+ if ( not len( package ) ):
+ continue
+
+ # Skip packages where the maintainer is not listed in the maintainers file
+ if ( package not in maintainers ):
+ continue
+
+ # Skip packages already owned by the QA team
+ if ( string.count( maintainers[package], 'debian-qa@lists.debian.org' ) or
+ string.count( maintainers[package], 'packages@qa.debian.org' ) ):
+ continue
+
+ orphaned.append( OrphanedPackage( package = package,
+ maintainer = maintainers[package],
+ bugid = current_bug['id'],
+ title = current_bug['subject'],
+ reporter = current_bug['originator'],
+ age = int( time.time() - current_bug['date'] ) / 86400
+ ) )
-if not ldap_result:
- print 'Problem with LDAP search'
- sys.exit(1)
-else:
- orphaned = { }
- ages = { }
- for _, dict in ldap_result:
- if not dict:
- continue
-
- if dict['debbugsState'][0] == 'done':
- continue
-
- bugid = int(dict['debbugsID'][0])
- subject = dict['debbugsTitle'][0]
- date = dict['debbugsDate'][0]
- originater = dict['debbugsSubmitter'][0]
-
- result = wnpp.weak_subject.search(subject)
- if result:
- tag = result.group(1)
- package = string.lower(result.group(2))
- else:
- continue
- if tag != 'O':
- continue
- # Package not in Maintainers file. Package name has a typo or
- # the package was removed. Ignore for now and let wnpp-handle it.
- if not maintainers.has_key(package):
- continue
-
- if (string.count(maintainers[package], 'debian-qa@lists.') or
- string.count(maintainers[package], 'packages@qa.')):
- continue
-
- age = wnpp.date2days(date)
- if orphaned.has_key(package):
- continue
- orphaned[package] = [ maintainers[package], str(bugid), subject, originater ]
-
- try:
- ages[age].append(package)
- except KeyError:
- ages[age] = [ package ]
-
-lages = ages.keys()
-lages.sort()
-lages.reverse()
try:
- output = open('/org/qa.debian.org/data/orphaned.txt', 'w')
-except (IOError, OSError):
- print 'Cannot write output file'
- sys.exit(1)
-
-output.write(str(len(orphaned)) + '\n\n')
-for age in lages:
- ages[age].sort()
- for package in ages[age]:
- output.write(package + '\n')
- for i in range(4):
- output.write(orphaned[package][i] + '\n')
- output.write(str(age) + '\n\n')
-output.close()
+ output = open( OUTPUT_FILE, 'w' )
+except ( IOError, OSError ) as e:
+ print( 'Output file not accessible `%s\': %s' % ( MAINTAINER_FILE, e ) )
+ sys.exit( 1 )
+
+
+output.write( str( len( orphaned ) ) + "\n\n" )
+for orphaned_package in sorted( orphaned, key = lambda x: x._age ): # sort by age:
+ output.write( str( orphaned_package ) )
+
+print( "%s: %d entries written\n" % ( OUTPUT_FILE, len( orphaned ) ) )
# vim: ts=4:expandtab:shiftwidth=4:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Generates the orphan.txt file used by orphaned.wml.
This version is basically a complete rewrite from the previous version
and was changed to use the SOAP interface to query the BTS instead of the LDAP
bridge.
It requires the following Debian packages:
* Python 2.6
* python-soappy
Please change paths configured below if necessary
"""
# Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <tbm@cyrius.com>
# 2011 Arno Töll <debian@toell.net>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston,
# MA 02110-1301, USA.
from __future__ import print_function
import sys
import re
import string
import SOAPpy
import time
MAINTAINER_FILE = '/org/qa.debian.org/data/ftp/Maintainers'
OUTPUT_FILE = '/org/qa.debian.org/data/orphaned.txt'
URL = 'http://bugs.debian.org/cgi-bin/soap.cgi'
NAMESPACE = 'Debbugs/SOAP'
###
### Nothing else should be changed below
###
class OrphanedPackage( object ):
def __repr__( self ):
return( "%s\n%s\n%d\n%s\n%s\n%d\n\n"
% ( self._package, self._maintainer, self._bugid, self._title, self._reporter, self._age ) )
def __init__( self, package, maintainer, bugid, title, reporter, age ):
self._package = package
self._maintainer = maintainer
self._bugid = bugid
self._title = title
self._reporter = reporter
self._age = age
try:
input = open( MAINTAINER_FILE, 'r' )
except ( IOError, OSError ) as e:
print( 'Couldn\'t access maintainer file `%s\': %s' % ( MAINTAINER_FILE, e ) )
sys.exit( 1 )
maintainers = { }
orphaned = []
for line in input.readlines():
line = string.rstrip( re.sub( '\s+', ' ', line ) )
package, maintainer = string.split( line, ' ', 1 )
maintainers[package] = maintainer
input.close()
try:
server = SOAPpy.SOAPProxy( URL, NAMESPACE, simplify_objects = 1 )
raw_bugs = server.get_status( server.get_bugs( "package", "wnpp" ) )
except ( Exception ) as e:
print( 'Can\'t access SOAP interface on %s (ns: %s): %s' % ( URL, NAMESPACE, e ) )
sys.exit( 1 )
if ( not raw_bugs or 'item' not in raw_bugs ):
print( "Could not retrieve any bugs (but no apparent error occurred)\n" )
sys.exit( 0 )
# Force argument to be a list, SOAPpy returns a dictionary instead of a dictionary list
# if a single bug was found. Not this were likely after all :>
if ( not isinstance( raw_bugs['item'], list ) ):
raw_bugs['item'] = ( raw_bugs['item'], )
for bugs in raw_bugs['item']:
current_bug = bugs['value']
# Skip packages with illegal subject line
parsed_subject = re.match( "(\w+):\s+(\S+)", current_bug['subject'] )
if ( not parsed_subject ):
continue
# Skip Non-O bugs
#if ( parsed_subject.group( 1 ) not in ( "O", "RFA" ) ):
if ( parsed_subject.group( 1 ) != "O" ):
continue
package = parsed_subject.group( 2 ).lower()
if ( not len( package ) ):
continue
# Skip packages where the maintainer is not listed in the maintainers file
if ( package not in maintainers ):
continue
# Skip packages already owned by the QA team
if ( string.count( maintainers[package], 'debian-qa@lists.debian.org' ) or
string.count( maintainers[package], 'packages@qa.debian.org' ) ):
continue
orphaned.append( OrphanedPackage( package = package,
maintainer = maintainers[package],
bugid = current_bug['id'],
title = current_bug['subject'],
reporter = current_bug['originator'],
age = int( time.time() - current_bug['date'] ) / 86400
) )
try:
output = open( OUTPUT_FILE, 'w' )
except ( IOError, OSError ) as e:
print( 'Output file not accessible `%s\': %s' % ( MAINTAINER_FILE, e ) )
sys.exit( 1 )
output.write( str( len( orphaned ) ) + "\n\n" )
for orphaned_package in sorted( orphaned, key = lambda x: x._age ): # sort by age:
output.write( str( orphaned_package ) )
print( "%s: %d entries written\n" % ( OUTPUT_FILE, len( orphaned ) ) )
# vim: ts=4:expandtab:shiftwidth=4:
--- End Message ---