Bug#698057: britney: Move/refactor parts into a sep. module

[Niels Thykier <niels@thykier.net>]

On 2013-05-11 16:49, Adam D. Barratt wrote:
> [...]
> 
> Thanks for these, and sorry for not looking over them properly earlier.
> 
> On the whole, they look good; thanks. At least one patch would have been
> easier to review if a "-w" version had been provided as well. :) A few
> queries / comments:
> 
> Firstly, I'm assuming that the resulting britney produces the same
> result from the testsuite.
> 
> +        same_source is an opt to avoid "load global".
> 
> Could we avoid using shorthand here? I'm assuming "optimisation" and
> something along the lines of "the overhead of forcing a lookup of a
> function from the global namespace"; the expansion of the latter is
> probably too far in the other direction though. :(
> 
> +        This method returns the full dependency tree for the package
> +        `pkg`, inside the `arch` architecture for the suite `suite`
> +        flatterned as an iterable.
> 
> I was going to grumble about the quoting style here, but then realised
> it's already in use in other parts of the code; I still don't really
> like it though. We appear to have at least three different quoting
> styles in use in various parts of the code. :-(
> 
> +    If given an iterable it returns a filtered iterator, otherwise it
> +    returns a function to generate filtered iterators.  The latter is
> +    useful if the same filter has to be (re-)used on multiple
> +    iterators that are not known on before hand.
> 
> "beforehand" is a single word; there's another occurrence in the same
> file.
> 
> +    The tree (or grapH) is returned as an iterable of (package, arch)
> +    tuples and the iterable will contain (`pkg`, `arch`) if it is
> +    available on that architecture.
> 
> s/grapH/graph/
> 
> Regards,
> 
> Adam
> 
> 

I have attached the patch series rebased against the current master.
All the comments should be fixed (these changes are visible in the
attached interdiffs[1]).

The patches series produces the same results as the current master
branch in the test suite.

~Niels

[1] Note: to reduce the noise, the interdiff was done after rebasing.
I.e. it is not an interdiff between the last patch, so changes caused by
rebasing is not included.


From 03ca0c81b74e163dbf795da776eab5ec714588a9 Mon Sep 17 00:00:00 2001
From: Niels Thykier

Date: Sat, 12 Jan 2013 14:47:51 +0100 Subject: [PATCH 01/18] Remove unused write_bugs The method was last invoked in 2008 (commit 3dc3be1c7). Signed-off-by: Niels Thykier --- britney.py | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/britney.py b/britney.py index a8af658..a5eadfa 100755 --- a/britney.py +++ b/britney.py @@ -656,21 +656,6 @@ class Britney(object): bugs[pkg] += l[1].split(",") return bugs - def write_bugs(self, basedir, bugs): - """Write the release critical bug summary to the specified directory - - For a more detailed explanation of the format, please check the method - read_bugs. - """ - filename = os.path.join(basedir, "BugsV") - self.__log("Writing RC bugs data to %s" % filename) - f = open(filename, 'w') - for pkg in sorted(bugs.keys()): - if not bugs[pkg]: - continue - f.write("%s %s\n" % (pkg, ','.join(bugs[pkg]))) - f.close() - def __maxver(self, pkg, dist): """Return the maximum version for a given package name -- 1.7.10.4
<<< text/html; name="0002-Move-same_source-to-separate-module.patch": Unrecognized >>>
From 0ca1261898961aa57d7c6aa072075516105d93e0 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 10:58:34 +0100 Subject: [PATCH 03/18] Move "constants" to a new consts module Signed-off-by: Niels Thykier --- britney.py | 24 +++--------------------- consts.py | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 21 deletions(-) create mode 100644 consts.py diff --git a/britney.py b/britney.py index 7aa9f95..6988284 100755 --- a/britney.py +++ b/britney.py @@ -213,31 +213,13 @@ from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem from britney_util import same_source +from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, + SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, + PROVIDES, RDEPENDS, RCONFLICTS) __author__ = 'Fabio Tranchitella and the Debian Release Team' __version__ = '2.0' -# source package -VERSION = 0 -SECTION = 1 -BINARIES = 2 -MAINTAINER = 3 -FAKESRC = 4 - -# binary package -SOURCE = 2 -SOURCEVER = 3 -ARCHITECTURE = 4 -# PREDEPENDS = 5 - No longer used by the python code -# - The C-code needs it for alignment reasons and still check it -# but ignore it if it is None (so keep it None). -DEPENDS = 6 -CONFLICTS = 7 -PROVIDES = 8 -RDEPENDS = 9 -RCONFLICTS = 10 - - class Britney(object): """Britney, the Debian testing updater script diff --git a/consts.py b/consts.py new file mode 100644 index 0000000..47670b2 --- /dev/null +++ b/consts.py @@ -0,0 +1,39 @@ +# -*- coding: utf-8 -*- + +# Constants from britney.py +# +# Assuming constants are copyrightable, then they are: +# Copyright (C) 2001-2008 Anthony Towns +# Andreas Barth +# Fabio Tranchitella +# Copyright (C) 2010-2012 Adam D. Barratt + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# source package +VERSION = 0 +SECTION = 1 +BINARIES = 2 +MAINTAINER = 3 +FAKESRC = 4 + +# binary package +SOURCE = 2 +SOURCEVER = 3 +ARCHITECTURE = 4 +# PREDEPENDS = 5 - No longer used by the python code +# - The C-code needs it for alignment reasons and still check it +# but ignore it if it is None (so keep it None). +DEPENDS = 6 +CONFLICTS = 7 +PROVIDES = 8 +RDEPENDS = 9 +RCONFLICTS = 10 -- 1.7.10.4 From 436414f5d840e47f88d3238e81ec066c4f82851b Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 11:32:03 +0100 Subject: [PATCH 04/18] Move "undo_changes" to britney_util Signed-off-by: Niels Thykier --- britney.py | 74 +++---------------------------------------------------- britney_util.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 71 deletions(-) diff --git a/britney.py b/britney.py index 6988284..5e47f80 100755 --- a/britney.py +++ b/britney.py @@ -212,7 +212,7 @@ from excuse import Excuse from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem -from britney_util import same_source +from britney_util import same_source, undo_changes from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -2176,7 +2176,7 @@ class Britney(object): skipped.append(pkg) single_undo = [(undo, item)] # (local-scope) binaries is actually self.binaries["testing"] so we cannot use it here. - self.undo_changes(single_undo, systems, sources, self.binaries) + undo_changes(single_undo, systems, sources, self.binaries) # if we are processing hints, return now if hint: @@ -2277,75 +2277,7 @@ class Britney(object): if not lundo: return lundo.reverse() - self.undo_changes(lundo, self.systems, self.sources, self.binaries) - - - def undo_changes(self, lundo, systems, sources, binaries): - """Undoes one or more changes to testing - - * lundo is a list of (undo, item)-tuples - * systems is the britney-py.c system - * sources is the table of all source packages for all suites - * binaries is the table of all binary packages for all suites - and architectures - """ - - # We do the undo process in "4 steps" and each step must be - # fully completed for each undo-item before starting on the - # next. - # - # see commit:ef71f0e33a7c3d8ef223ec9ad5e9843777e68133 and - # #624716 for the issues we had when we did not do this. - - - # STEP 1 - # undo all the changes for sources - for (undo, item) in lundo: - for k in undo['sources'].keys(): - if k[0] == '-': - del sources["testing"][k[1:]] - else: - sources["testing"][k] = undo['sources'][k] - - # STEP 2 - # undo all new binaries (consequence of the above) - for (undo, item) in lundo: - if not item.is_removal and item.package in sources[item.suite]: - for p in sources[item.suite][item.package][BINARIES]: - binary, arch = p.split("/") - if item.architecture in ['source', arch]: - del binaries["testing"][arch][0][binary] - systems[arch].remove_binary(binary) - - - # STEP 3 - # undo all other binary package changes (except virtual packages) - for (undo, item) in lundo: - for p in undo['binaries'].keys(): - binary, arch = p.split("/") - if binary[0] == "-": - del binaries['testing'][arch][0][binary[1:]] - systems[arch].remove_binary(binary[1:]) - else: - binaries_t_a = binaries['testing'][arch][0] - binaries_t_a[binary] = undo['binaries'][p] - systems[arch].remove_binary(binary) - systems[arch].add_binary(binary, binaries_t_a[binary][:PROVIDES] + \ - [", ".join(binaries_t_a[binary][PROVIDES]) or None]) - - # STEP 4 - # undo all changes to virtual packages - for (undo, item) in lundo: - for p in undo['nvirtual']: - j, arch = p.split("/") - del binaries['testing'][arch][1][j] - for p in undo['virtual']: - j, arch = p.split("/") - if j[0] == '-': - del binaries['testing'][arch][1][j[1:]] - else: - binaries['testing'][arch][1][j] = undo['virtual'][p] - + undo_changes(lundo, self.systems, self.sources, self.binaries) diff --git a/britney_util.py b/britney_util.py index 8343359..52dc9fa 100644 --- a/britney_util.py +++ b/britney_util.py @@ -17,6 +17,7 @@ # GNU General Public License for more details. import re +from consts import BINARIES, PROVIDES binnmu_re = re.compile(r'^(.*)\+b\d+$') @@ -41,3 +42,74 @@ def same_source(sv1, sv2, binnmu_re=binnmu_re): return 1 return 0 + + +def undo_changes(lundo, systems, sources, binaries, + BINARIES=BINARIES, PROVIDES=PROVIDES): + """Undoes one or more changes to testing + + * lundo is a list of (undo, item)-tuples + * systems is the britney-py.c system + * sources is the table of all source packages for all suites + * binaries is the table of all binary packages for all suites + and architectures + + The "X=X" parameters are optimizations to avoid "load global" + in loops. + """ + + # We do the undo process in "4 steps" and each step must be + # fully completed for each undo-item before starting on the + # next. + # + # see commit:ef71f0e33a7c3d8ef223ec9ad5e9843777e68133 and + # #624716 for the issues we had when we did not do this. + + + # STEP 1 + # undo all the changes for sources + for (undo, item) in lundo: + for k in undo['sources']: + if k[0] == '-': + del sources["testing"][k[1:]] + else: + sources["testing"][k] = undo['sources'][k] + + # STEP 2 + # undo all new binaries (consequence of the above) + for (undo, item) in lundo: + if not item.is_removal and item.package in sources[item.suite]: + for p in sources[item.suite][item.package][BINARIES]: + binary, arch = p.split("/") + if item.architecture in ['source', arch]: + del binaries["testing"][arch][0][binary] + systems[arch].remove_binary(binary) + + + # STEP 3 + # undo all other binary package changes (except virtual packages) + for (undo, item) in lundo: + for p in undo['binaries']: + binary, arch = p.split("/") + if binary[0] == "-": + del binaries['testing'][arch][0][binary[1:]] + systems[arch].remove_binary(binary[1:]) + else: + binaries_t_a = binaries['testing'][arch][0] + binaries_t_a[binary] = undo['binaries'][p] + systems[arch].remove_binary(binary) + systems[arch].add_binary(binary, binaries_t_a[binary][:PROVIDES] + \ + [", ".join(binaries_t_a[binary][PROVIDES]) or None]) + + # STEP 4 + # undo all changes to virtual packages + for (undo, item) in lundo: + for p in undo['nvirtual']: + j, arch = p.split("/") + del binaries['testing'][arch][1][j] + for p in undo['virtual']: + j, arch = p.split("/") + if j[0] == '-': + del binaries['testing'][arch][1][j[1:]] + else: + binaries['testing'][arch][1][j] = undo['virtual'][p] -- 1.7.10.4 From b397bc769964e338cb0d0f416ecfbea03598621b Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 11:37:53 +0100 Subject: [PATCH 05/18] Move old_libraries_format to britney_util Signed-off-by: Niels Thykier --- britney.py | 18 +++--------------- britney_util.py | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/britney.py b/britney.py index 5e47f80..2a16a7a 100755 --- a/britney.py +++ b/britney.py @@ -212,7 +212,7 @@ from excuse import Excuse from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem -from britney_util import same_source, undo_changes +from britney_util import old_libraries_format, same_source, undo_changes from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -2370,14 +2370,14 @@ class Britney(object): removals = self.old_libraries() if len(removals) > 0: self.output_write("Removing packages left in testing for smooth updates (%d):\n%s" % \ - (len(removals), self.old_libraries_format(removals))) + (len(removals), old_libraries_format(removals))) self.do_all(actions=[ MigrationItem(x) for x in removals ]) removals = self.old_libraries() else: removals = () self.output_write("List of old libraries in testing (%d):\n%s" % \ - (len(removals), self.old_libraries_format(removals))) + (len(removals), old_libraries_format(removals))) # output files if not self.options.dry_run: @@ -2641,18 +2641,6 @@ class Britney(object): removals.append("-" + pkg_name + "/" + arch) return removals - def old_libraries_format(self, libs): - """Format old libraries in a smart table""" - libraries = {} - for i in libs: - pkg, arch = i.split("/") - pkg = pkg[1:] - if pkg in libraries: - libraries[pkg].append(arch) - else: - libraries[pkg] = [arch] - return "\n".join([" " + k + ": " + " ".join(libraries[k]) for k in libraries]) + "\n" - def nuninst_arch_report(self, nuninst, arch): """Print a report of uninstallable packages for one architecture.""" all = {} diff --git a/britney_util.py b/britney_util.py index 52dc9fa..5ce6ad0 100644 --- a/britney_util.py +++ b/britney_util.py @@ -113,3 +113,17 @@ def undo_changes(lundo, systems, sources, binaries, del binaries['testing'][arch][1][j[1:]] else: binaries['testing'][arch][1][j] = undo['virtual'][p] + + +def old_libraries_format(libs): + """Format old libraries in a smart table""" + libraries = {} + for i in libs: + pkg, arch = i.split("/") + pkg = pkg[1:] + if pkg in libraries: + libraries[pkg].append(arch) + else: + libraries[pkg] = [arch] + return "\n".join(" " + k + ": " + " ".join(libraries[k]) for k in libraries) + "\n" + -- 1.7.10.4 From 165a41625b65faf40bc2430f53b9e0205d8366b6 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Wed, 30 May 2012 09:24:34 +0200 Subject: [PATCH 06/18] Merge get_reverse_tree and get_full_tree Beside some "minor differences" they were computing the same "tree" (read: "graph"), so merge them into one (get_reverse_tree) and properly document return value and special cases. Signed-off-by: Niels Thykier --- britney.py | 51 ++++++++++++++++++++++----------------------------- 1 file changed, 22 insertions(+), 29 deletions(-) diff --git a/britney.py b/britney.py index 2a16a7a..95e050f 100755 --- a/britney.py +++ b/britney.py @@ -190,7 +190,7 @@ import urllib import apt_pkg from functools import reduce, partial -from itertools import chain, repeat +from itertools import chain, repeat, izip from operator import attrgetter if __name__ == '__main__': @@ -1930,11 +1930,7 @@ class Britney(object): affected.update(self.get_reverse_tree(binary, parch, 'testing')) # all the reverse conflicts and their dependency tree are affected by the change for j in binaries[parch][0][binary][RCONFLICTS]: - key = (j, parch) - if key not in affected: affected.add(key) - for p in self.get_full_tree(j, parch, 'testing'): - key = (p, parch) - if key not in affected: affected.add(key) + affected.update(self.get_reverse_tree(j, parch, 'testing')) self.systems[parch].remove_binary(binary) else: # the binary isn't in testing, but it may have been at @@ -1952,7 +1948,6 @@ class Britney(object): if p in tundo['binaries']: for rdep in tundo['binaries'][p][RDEPENDS]: if rdep in binaries[parch][0] and rdep not in source[BINARIES]: - affected.add( (rdep, parch) ) affected.update(self.get_reverse_tree(rdep, parch, 'testing')) # add/update the binary package binaries[parch][0][binary] = self.binaries[item.suite][parch][0][binary] @@ -1984,10 +1979,27 @@ class Britney(object): return (item, affected, undo) def get_reverse_tree(self, pkg, arch, suite): - binaries = self.binaries[suite][arch][0] + """Calculate the full dependency tree for the given package + + This method returns the full dependency tree for the package + `pkg`, inside the `arch` architecture for the suite `suite` + flatterned as an iterable. + + The tree is returned as an iterable of (package, arch) tuples + and the iterable will contain (`pkg`, `arch`) if it is + available on that architecture. + If `pkg` is not available on that architecture in that suite, + this returns an empty iterable. + + The method does not promise any ordering of the returned + elements and the iterable is not reusable nor mutable. + """ + binaries = self.binaries[suite][arch][0] + if pkg not in binaries: + return frozenset() rev_deps = set(binaries[pkg][RDEPENDS]) - seen = set() + seen = set([pkg]) while len(rev_deps) > 0: # mark all of the current iteration of packages as affected seen |= rev_deps @@ -1999,26 +2011,7 @@ class Britney(object): # in the process rev_deps = set([package for package in chain.from_iterable(new_rev_deps) \ if package not in seen ]) - return zip(seen, repeat(arch)) - - def get_full_tree(self, pkg, arch, suite): - """Calculate the full dependency tree for the given package - - This method returns the full dependency tree for the package `pkg`, - inside the `arch` architecture for the suite `suite`. - """ - packages = [pkg] - binaries = self.binaries[suite][arch][0] - if pkg in binaries: - l = n = 0 - while len(packages) > l: - l = len(packages) - for p in packages[n:]: - packages.extend([x for x in binaries[p][RDEPENDS] if x not in packages and x in binaries]) - n = l - return packages - else: - return [] + return izip(seen, repeat(arch)) def _check_packages(self, binaries, systems, arch, affected, skip_archall, nuninst, pkg): broken = nuninst[arch + "+all"] -- 1.7.10.4 From 3c3aaa8df629fe72b489cac0947bd09a695dd060 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Thu, 5 Jul 2012 12:07:34 +0200 Subject: [PATCH 07/18] Add container filters and use it to refactor get_reverse_tree Signed-off-by: Niels Thykier --- britney.py | 16 ++++++++-------- britney_util.py | 30 ++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 8 deletions(-) diff --git a/britney.py b/britney.py index 95e050f..517d530 100755 --- a/britney.py +++ b/britney.py @@ -212,7 +212,8 @@ from excuse import Excuse from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem -from britney_util import old_libraries_format, same_source, undo_changes +from britney_util import (old_libraries_format, same_source, undo_changes, + ifilter_except, ifilter_only) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -2000,17 +2001,16 @@ class Britney(object): return frozenset() rev_deps = set(binaries[pkg][RDEPENDS]) seen = set([pkg]) - while len(rev_deps) > 0: + + binfilt = ifilter_only(binaries) + revfilt = ifilter_except(seen) + flatten = chain.from_iterable + while rev_deps: # mark all of the current iteration of packages as affected seen |= rev_deps # generate the next iteration, which is the reverse-dependencies of # the current iteration - new_rev_deps = [ binaries[x][RDEPENDS] for x in rev_deps \ - if x in binaries ] - # flatten the list-of-lists, filtering out already handled packages - # in the process - rev_deps = set([package for package in chain.from_iterable(new_rev_deps) \ - if package not in seen ]) + rev_deps = set(revfilt(flatten( binaries[x][RDEPENDS] for x in binfilt(rev_deps) ))) return izip(seen, repeat(arch)) def _check_packages(self, binaries, systems, arch, affected, skip_archall, nuninst, pkg): diff --git a/britney_util.py b/britney_util.py index 5ce6ad0..dee520b 100644 --- a/britney_util.py +++ b/britney_util.py @@ -5,6 +5,7 @@ # Andreas Barth # Fabio Tranchitella # Copyright (C) 2010-2012 Adam D. Barratt +# Copyright (C) 2012 Niels Thykier # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -16,6 +17,9 @@ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. + +from functools import partial +from itertools import ifilter, ifilterfalse import re from consts import BINARIES, PROVIDES @@ -44,6 +48,32 @@ def same_source(sv1, sv2, binnmu_re=binnmu_re): return 0 +def ifilter_except(container, iterable=None): + """Filter out elements in container + + If given an iterable it returns a filtered iterator, otherwise it + returns a function to generate filtered iterators. The latter is + useful if the same filter has to be (re-)used on multiple + iterators that are not known on beforehand. + """ + if iterable is not None: + return ifilterfalse(container.__contains__, iterable) + return partial(ifilterfalse, container.__contains__) + + +def ifilter_only(container, iterable=None): + """Filter out elements in which are not in container + + If given an iterable it returns a filtered iterator, otherwise it + returns a function to generate filtered iterators. The latter is + useful if the same filter has to be (re-)used on multiple + iterators that are not known on beforehand. + """ + if iterable is not None: + return ifilter(container.__contains__, iterable) + return partial(ifilter, container.__contains__) + + def undo_changes(lundo, systems, sources, binaries, BINARIES=BINARIES, PROVIDES=PROVIDES): """Undoes one or more changes to testing -- 1.7.10.4 From ea6506420a8a0dbc7eeec6083aba8c206de6737b Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 11:52:56 +0100 Subject: [PATCH 08/18] Move register_reverses to britney_util Signed-off-by: Niels Thykier --- britney.py | 44 ++------------------------------------------ britney_util.py | 49 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 43 deletions(-) diff --git a/britney.py b/britney.py index 517d530..c87aaa9 100755 --- a/britney.py +++ b/britney.py @@ -213,7 +213,7 @@ from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, - ifilter_except, ifilter_only) + register_reverses) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -566,52 +566,12 @@ class Britney(object): packages[pkg] = dpkg # loop again on the list of packages to register reverse dependencies and conflicts - register_reverses = self.register_reverses for pkg in packages: register_reverses(pkg, packages, provides, check_doubles=False) # return a tuple with the list of real and virtual packages return (packages, provides) - def register_reverses(self, pkg, packages, provides, check_doubles=True, parse_depends=apt_pkg.parse_depends): - """Register reverse dependencies and conflicts for the specified package - - This method registers the reverse dependencies and conflicts for - a given package using `packages` as the list of packages and `provides` - as the list of virtual packages. - - The method has an optional parameter parse_depends which is there - just for performance reasons and is not meant to be overwritten. - """ - # register the list of the dependencies for the depending packages - dependencies = [] - if packages[pkg][DEPENDS]: - dependencies.extend(parse_depends(packages[pkg][DEPENDS], False)) - # go through the list - for p in dependencies: - for a in p: - # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]): - packages[a[0]][RDEPENDS].append(pkg) - # also register packages which provide the package (if any) - if a[0] in provides: - for i in provides.get(a[0]): - if i not in packages: continue - if not check_doubles or pkg not in packages[i][RDEPENDS]: - packages[i][RDEPENDS].append(pkg) - # register the list of the conflicts for the conflicting packages - if packages[pkg][CONFLICTS]: - for p in parse_depends(packages[pkg][CONFLICTS], False): - for a in p: - # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]): - packages[a[0]][RCONFLICTS].append(pkg) - # also register packages which provide the package (if any) - if a[0] in provides: - for i in provides[a[0]]: - if i not in packages: continue - if not check_doubles or pkg not in packages[i][RCONFLICTS]: - packages[i][RCONFLICTS].append(pkg) def read_bugs(self, basedir): """Read the release critial bug summary from the specified directory @@ -1970,7 +1930,7 @@ class Britney(object): for p in source[BINARIES]: binary, parch = p.split("/") if item.architecture not in ['source', parch]: continue - self.register_reverses(binary, binaries[parch][0] , binaries[parch][1]) + register_reverses(binary, binaries[parch][0] , binaries[parch][1]) # add/update the source package if item.architecture == 'source': diff --git a/britney_util.py b/britney_util.py index dee520b..45b5ac1 100644 --- a/britney_util.py +++ b/britney_util.py @@ -18,10 +18,14 @@ # GNU General Public License for more details. +import apt_pkg from functools import partial from itertools import ifilter, ifilterfalse import re -from consts import BINARIES, PROVIDES + + +from consts import (BINARIES, PROVIDES, DEPENDS, CONFLICTS, + RDEPENDS, RCONFLICTS) binnmu_re = re.compile(r'^(.*)\+b\d+$') @@ -157,3 +161,46 @@ def old_libraries_format(libs): libraries[pkg] = [arch] return "\n".join(" " + k + ": " + " ".join(libraries[k]) for k in libraries) + "\n" + + +def register_reverses(pkg, packages, provides, check_doubles=True, + parse_depends=apt_pkg.parse_depends, + RDEPENDS=RDEPENDS, RCONFLICTS=RCONFLICTS): + """Register reverse dependencies and conflicts for the specified package + + This method registers the reverse dependencies and conflicts for + a given package using `packages` as the list of packages and `provides` + as the list of virtual packages. + + The "X=X" parameters are optimizations to avoid "load global" in + the loops. + """ + # register the list of the dependencies for the depending packages + dependencies = [] + if packages[pkg][DEPENDS]: + dependencies.extend(parse_depends(packages[pkg][DEPENDS], False)) + # go through the list + for p in dependencies: + for a in p: + # register real packages + if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]): + packages[a[0]][RDEPENDS].append(pkg) + # also register packages which provide the package (if any) + if a[0] in provides: + for i in provides.get(a[0]): + if i not in packages: continue + if not check_doubles or pkg not in packages[i][RDEPENDS]: + packages[i][RDEPENDS].append(pkg) + # register the list of the conflicts for the conflicting packages + if packages[pkg][CONFLICTS]: + for p in parse_depends(packages[pkg][CONFLICTS], False): + for a in p: + # register real packages + if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]): + packages[a[0]][RCONFLICTS].append(pkg) + # also register packages which provide the package (if any) + if a[0] in provides: + for i in provides[a[0]]: + if i not in packages: continue + if not check_doubles or pkg not in packages[i][RCONFLICTS]: + packages[i][RCONFLICTS].append(pkg) -- 1.7.10.4 From 56375343e9072a2bdebd2bca45473fc65a5b9ec2 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 12:14:36 +0100 Subject: [PATCH 09/18] Move the package loop into register_reverses By moving the package loop inside register_reverses, it will be invoked a lot less (reducing the overhead of invoking functions). Signed-off-by: Niels Thykier --- britney.py | 13 ++++++----- britney_util.py | 67 ++++++++++++++++++++++++++++++++----------------------- 2 files changed, 46 insertions(+), 34 deletions(-) diff --git a/britney.py b/britney.py index c87aaa9..4dd2176 100755 --- a/britney.py +++ b/britney.py @@ -566,8 +566,7 @@ class Britney(object): packages[pkg] = dpkg # loop again on the list of packages to register reverse dependencies and conflicts - for pkg in packages: - register_reverses(pkg, packages, provides, check_doubles=False) + register_reverses(packages, provides, check_doubles=False) # return a tuple with the list of real and virtual packages return (packages, provides) @@ -1927,10 +1926,12 @@ class Britney(object): affected.update(self.get_reverse_tree(binary, parch, 'testing')) # register reverse dependencies and conflicts for the new binary packages - for p in source[BINARIES]: - binary, parch = p.split("/") - if item.architecture not in ['source', parch]: continue - register_reverses(binary, binaries[parch][0] , binaries[parch][1]) + if item.architecture == 'source': + pkg_iter = (p.split("/")[0] for p in source[BINARIES]) + else: + ext = "/" + item.architecture + pkg_iter = (p.split("/")[0] for p in source[BINARIES] if p.endswith(ext)) + register_reverses(binaries[parch][0], binaries[parch][1], iterator=pkg_iter) # add/update the source package if item.architecture == 'source': diff --git a/britney_util.py b/britney_util.py index 45b5ac1..762bb34 100644 --- a/britney_util.py +++ b/britney_util.py @@ -163,44 +163,55 @@ def old_libraries_format(libs): -def register_reverses(pkg, packages, provides, check_doubles=True, +def register_reverses(packages, provides, check_doubles=True, iterator=None, parse_depends=apt_pkg.parse_depends, + DEPENDS=DEPENDS, CONFLICTS=CONFLICTS, RDEPENDS=RDEPENDS, RCONFLICTS=RCONFLICTS): - """Register reverse dependencies and conflicts for the specified package + """Register reverse dependencies and conflicts for a given + sequence of packages - This method registers the reverse dependencies and conflicts for - a given package using `packages` as the list of packages and `provides` - as the list of virtual packages. + This method registers the reverse dependencies and conflicts for a + given sequence of packages. "packages" is a table of real + packages and "provides" is a table of virtual packages. + + iterator is the sequence of packages for which the reverse + relations should be updated. The "X=X" parameters are optimizations to avoid "load global" in the loops. """ - # register the list of the dependencies for the depending packages - dependencies = [] - if packages[pkg][DEPENDS]: - dependencies.extend(parse_depends(packages[pkg][DEPENDS], False)) - # go through the list - for p in dependencies: - for a in p: - # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]): - packages[a[0]][RDEPENDS].append(pkg) - # also register packages which provide the package (if any) - if a[0] in provides: - for i in provides.get(a[0]): - if i not in packages: continue - if not check_doubles or pkg not in packages[i][RDEPENDS]: - packages[i][RDEPENDS].append(pkg) - # register the list of the conflicts for the conflicting packages - if packages[pkg][CONFLICTS]: - for p in parse_depends(packages[pkg][CONFLICTS], False): + if iterator is None: + iterator = packages.iterkeys() + else: + iterator = ifilter_only(packages, iterator) + + for pkg in iterator: + # register the list of the dependencies for the depending packages + dependencies = [] + if packages[pkg][DEPENDS]: + dependencies.extend(parse_depends(packages[pkg][DEPENDS], False)) + # go through the list + for p in dependencies: for a in p: # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]): - packages[a[0]][RCONFLICTS].append(pkg) + if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]): + packages[a[0]][RDEPENDS].append(pkg) # also register packages which provide the package (if any) if a[0] in provides: for i in provides[a[0]]: if i not in packages: continue - if not check_doubles or pkg not in packages[i][RCONFLICTS]: - packages[i][RCONFLICTS].append(pkg) + if not check_doubles or pkg not in packages[i][RDEPENDS]: + packages[i][RDEPENDS].append(pkg) + # register the list of the conflicts for the conflicting packages + if packages[pkg][CONFLICTS]: + for p in parse_depends(packages[pkg][CONFLICTS], False): + for a in p: + # register real packages + if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]): + packages[a[0]][RCONFLICTS].append(pkg) + # also register packages which provide the package (if any) + if a[0] in provides: + for i in provides[a[0]]: + if i not in packages: continue + if not check_doubles or pkg not in packages[i][RCONFLICTS]: + packages[i][RCONFLICTS].append(pkg) -- 1.7.10.4 From 354eded73dc6d4d1e13fe8c6ed1311382a9479e2 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 28 Dec 2012 13:32:04 +0100 Subject: [PATCH 10/18] register_reverses: factor out a[0] Signed-off-by: Niels Thykier --- britney_util.py | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/britney_util.py b/britney_util.py index 762bb34..4aa4252 100644 --- a/britney_util.py +++ b/britney_util.py @@ -188,30 +188,33 @@ def register_reverses(packages, provides, check_doubles=True, iterator=None, for pkg in iterator: # register the list of the dependencies for the depending packages dependencies = [] - if packages[pkg][DEPENDS]: - dependencies.extend(parse_depends(packages[pkg][DEPENDS], False)) + pkg_data = packages[pkg] + if pkg_data[DEPENDS]: + dependencies.extend(parse_depends(pkg_data[DEPENDS], False)) # go through the list for p in dependencies: for a in p: + dep = a[0] # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RDEPENDS]): - packages[a[0]][RDEPENDS].append(pkg) + if dep in packages and (not check_doubles or pkg not in packages[dep][RDEPENDS]): + packages[dep][RDEPENDS].append(pkg) # also register packages which provide the package (if any) - if a[0] in provides: - for i in provides[a[0]]: + if dep in provides: + for i in provides[dep]: if i not in packages: continue if not check_doubles or pkg not in packages[i][RDEPENDS]: packages[i][RDEPENDS].append(pkg) # register the list of the conflicts for the conflicting packages - if packages[pkg][CONFLICTS]: - for p in parse_depends(packages[pkg][CONFLICTS], False): + if pkg_data[CONFLICTS]: + for p in parse_depends(pkg_data[CONFLICTS], False): for a in p: + con = a[0] # register real packages - if a[0] in packages and (not check_doubles or pkg not in packages[a[0]][RCONFLICTS]): - packages[a[0]][RCONFLICTS].append(pkg) + if con in packages and (not check_doubles or pkg not in packages[con][RCONFLICTS]): + packages[con][RCONFLICTS].append(pkg) # also register packages which provide the package (if any) - if a[0] in provides: - for i in provides[a[0]]: + if con in provides: + for i in provides[con]: if i not in packages: continue if not check_doubles or pkg not in packages[i][RCONFLICTS]: packages[i][RCONFLICTS].append(pkg) -- 1.7.10.4 From d06b8dbeddd284b450308598320ebededb4335fc Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Sat, 12 Jan 2013 16:34:02 +0100 Subject: [PATCH 11/18] Move get_reverse_tree into britney_util Rename get_reverse_tree and move it to britney_util with slightly different arguments. Signed-off-by: Niels Thykier --- britney.py | 49 ++++++++----------------------------------------- britney_util.py | 43 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 42 deletions(-) diff --git a/britney.py b/britney.py index 4dd2176..21815a4 100755 --- a/britney.py +++ b/britney.py @@ -190,7 +190,6 @@ import urllib import apt_pkg from functools import reduce, partial -from itertools import chain, repeat, izip from operator import attrgetter if __name__ == '__main__': @@ -213,7 +212,7 @@ from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, - register_reverses) + register_reverses, compute_reverse_tree) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -1764,6 +1763,7 @@ class Britney(object): # local copies for better performances sources = self.sources binaries = self.binaries['testing'] + get_reverse_tree = partial(compute_reverse_tree, self.binaries["testing"]) # remove all binary packages (if the source already exists) if item.architecture == 'source' or not item.is_removal: if item.package in sources['testing']: @@ -1842,7 +1842,7 @@ class Britney(object): # save the old binary for undo undo['binaries'][p] = binaries[parch][0][binary] # all the reverse dependencies are affected by the change - affected.update(self.get_reverse_tree(binary, parch, 'testing')) + affected.update(get_reverse_tree(binary, parch)) # remove the provided virtual packages for j in binaries[parch][0][binary][PROVIDES]: key = j + "/" + parch @@ -1866,7 +1866,7 @@ class Britney(object): # updates but not supported as a manual hint elif item.package in binaries[item.architecture][0]: undo['binaries'][item.package + "/" + item.architecture] = binaries[item.architecture][0][item.package] - affected.update(self.get_reverse_tree(item.package, item.architecture, 'testing')) + affected.update(get_reverse_tree(item.package, item.architecture)) del binaries[item.architecture][0][item.package] self.systems[item.architecture].remove_binary(item.package) @@ -1887,10 +1887,10 @@ class Britney(object): # save the old binary package undo['binaries'][p] = binaries[parch][0][binary] # all the reverse dependencies are affected by the change - affected.update(self.get_reverse_tree(binary, parch, 'testing')) + affected.update(get_reverse_tree(binary, parch)) # all the reverse conflicts and their dependency tree are affected by the change for j in binaries[parch][0][binary][RCONFLICTS]: - affected.update(self.get_reverse_tree(j, parch, 'testing')) + affected.update(get_reverse_tree(j, parch)) self.systems[parch].remove_binary(binary) else: # the binary isn't in testing, but it may have been at @@ -1908,7 +1908,7 @@ class Britney(object): if p in tundo['binaries']: for rdep in tundo['binaries'][p][RDEPENDS]: if rdep in binaries[parch][0] and rdep not in source[BINARIES]: - affected.update(self.get_reverse_tree(rdep, parch, 'testing')) + affected.update(get_reverse_tree(rdep, parch)) # add/update the binary package binaries[parch][0][binary] = self.binaries[item.suite][parch][0][binary] self.systems[parch].add_binary(binary, binaries[parch][0][binary][:PROVIDES] + \ @@ -1923,7 +1923,7 @@ class Britney(object): undo['virtual'][key] = binaries[parch][1][j][:] binaries[parch][1][j].append(binary) # all the reverse dependencies are affected by the change - affected.update(self.get_reverse_tree(binary, parch, 'testing')) + affected.update(get_reverse_tree(binary, parch)) # register reverse dependencies and conflicts for the new binary packages if item.architecture == 'source': @@ -1940,39 +1940,6 @@ class Britney(object): # return the package name, the suite, the list of affected packages and the undo dictionary return (item, affected, undo) - def get_reverse_tree(self, pkg, arch, suite): - """Calculate the full dependency tree for the given package - - This method returns the full dependency tree for the package - `pkg`, inside the `arch` architecture for the suite `suite` - flatterned as an iterable. - - The tree is returned as an iterable of (package, arch) tuples - and the iterable will contain (`pkg`, `arch`) if it is - available on that architecture. - - If `pkg` is not available on that architecture in that suite, - this returns an empty iterable. - - The method does not promise any ordering of the returned - elements and the iterable is not reusable nor mutable. - """ - binaries = self.binaries[suite][arch][0] - if pkg not in binaries: - return frozenset() - rev_deps = set(binaries[pkg][RDEPENDS]) - seen = set([pkg]) - - binfilt = ifilter_only(binaries) - revfilt = ifilter_except(seen) - flatten = chain.from_iterable - while rev_deps: - # mark all of the current iteration of packages as affected - seen |= rev_deps - # generate the next iteration, which is the reverse-dependencies of - # the current iteration - rev_deps = set(revfilt(flatten( binaries[x][RDEPENDS] for x in binfilt(rev_deps) ))) - return izip(seen, repeat(arch)) def _check_packages(self, binaries, systems, arch, affected, skip_archall, nuninst, pkg): broken = nuninst[arch + "+all"] diff --git a/britney_util.py b/britney_util.py index 4aa4252..762c9fe 100644 --- a/britney_util.py +++ b/britney_util.py @@ -20,7 +20,7 @@ import apt_pkg from functools import partial -from itertools import ifilter, ifilterfalse +from itertools import chain, ifilter, ifilterfalse, izip, repeat import re @@ -218,3 +218,44 @@ def register_reverses(packages, provides, check_doubles=True, iterator=None, if i not in packages: continue if not check_doubles or pkg not in packages[i][RCONFLICTS]: packages[i][RCONFLICTS].append(pkg) + + +def compute_reverse_tree(packages_s, pkg, arch, + set=set, flatten=chain.from_iterable, + RDEPENDS=RDEPENDS): + """Calculate the full dependency tree for the given package + + This method returns the full dependency tree for the package + "pkg", inside the "arch" architecture for a given suite flattened + as an iterable. The first argument "packages_s" is the binary + package table for that given suite (e.g. Britney().binaries["testing"]). + + The tree (or graph) is returned as an iterable of (package, arch) + tuples and the iterable will contain ("pkg", "arch") if it is + available on that architecture. + + If "pkg" is not available on that architecture in that suite, + this returns an empty iterable. + + The method does not promise any ordering of the returned + elements and the iterable is not reusable. + + The flatten=... and the "X=X" parameters are optimizations to + avoid "load global" in the loops. + """ + binaries = packages_s[arch][0] + if pkg not in binaries: + return frozenset() + rev_deps = set(binaries[pkg][RDEPENDS]) + seen = set([pkg]) + + binfilt = ifilter_only(binaries) + revfilt = ifilter_except(seen) + + while rev_deps: + # mark all of the current iteration of packages as affected + seen |= rev_deps + # generate the next iteration, which is the reverse-dependencies of + # the current iteration + rev_deps = set(revfilt(flatten( binaries[x][RDEPENDS] for x in binfilt(rev_deps) ))) + return izip(seen, repeat(arch)) -- 1.7.10.4 From fe04d8166f55389fbf282c48922a657eb72be05e Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Sat, 12 Jan 2013 17:17:41 +0100 Subject: [PATCH 12/18] Move {read,write}_nuninst to britney_util Signed-off-by: Niels Thykier --- britney.py | 29 +++++++---------------------- britney_util.py | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 22 deletions(-) diff --git a/britney.py b/britney.py index 21815a4..a2276bc 100755 --- a/britney.py +++ b/britney.py @@ -212,7 +212,8 @@ from migrationitem import MigrationItem, HintItem from hints import HintCollection from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, - register_reverses, compute_reverse_tree) + register_reverses, compute_reverse_tree, + read_nuninst, write_nuninst) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -273,7 +274,7 @@ class Britney(object): if self.options.print_uninst: self.nuninst_arch_report(nuninst, arch) if not self.options.print_uninst: - self.write_nuninst(nuninst) + write_nuninst(self.options.noninst_status, nuninst) else: self.__log("Not building the list of non-installable packages, as requested", type="I") @@ -888,25 +889,6 @@ class Britney(object): f.write(output + "\n") f.close() - def write_nuninst(self, nuninst): - """Write the non-installable report""" - f = open(self.options.noninst_status, 'w') - f.write("Built on: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n") - f.write("Last update: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n\n") - f.write("".join([k + ": " + " ".join(nuninst[k]) + "\n" for k in nuninst])) - f.close() - - def read_nuninst(self): - """Read the non-installable report""" - f = open(self.options.noninst_status) - nuninst = {} - for r in f: - if ":" not in r: continue - arch, packages = r.strip().split(":", 1) - if arch.split("+", 1)[0] in self.options.architectures: - nuninst[arch] = set(packages.split()) - return nuninst - # Utility methods for package analysis # ------------------------------------ @@ -1652,10 +1634,13 @@ class Britney(object): It returns a dictionary with the architectures as keys and the list of uninstallable packages as values. + + NB: If build is False, requested_arch is ignored. """ # if we are not asked to build the nuninst, read it from the cache if not build: - return self.read_nuninst() + return read_nuninst(self.options.noninst_status, + self.options.architectures) nuninst = {} diff --git a/britney_util.py b/britney_util.py index 762c9fe..d4739e1 100644 --- a/britney_util.py +++ b/britney_util.py @@ -22,6 +22,7 @@ import apt_pkg from functools import partial from itertools import chain, ifilter, ifilterfalse, izip, repeat import re +import time from consts import (BINARIES, PROVIDES, DEPENDS, CONFLICTS, @@ -259,3 +260,35 @@ def compute_reverse_tree(packages_s, pkg, arch, # the current iteration rev_deps = set(revfilt(flatten( binaries[x][RDEPENDS] for x in binfilt(rev_deps) ))) return izip(seen, repeat(arch)) + + +def write_nuninst(filename, nuninst): + """Write the non-installable report + + Write the non-installable report derived from "nuninst" to the + file denoted by "filename". + """ + with open(filename, 'w') as f: + # Having two fields with (almost) identical dates seems a bit + # redundant. + f.write("Built on: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n") + f.write("Last update: " + time.strftime("%Y.%m.%d %H:%M:%S %z", time.gmtime(time.time())) + "\n\n") + f.write("".join([k + ": " + " ".join(nuninst[k]) + "\n" for k in nuninst])) + + +def read_nuninst(filename, architectures): + """Read the non-installable report + + Read the non-installable report from the file denoted by + "filename" and return it. Only architectures in "architectures" + will be included in the report. + """ + nuninst = {} + with open(filename) as f: + for r in f: + if ":" not in r: continue + arch, packages = r.strip().split(":", 1) + if arch.split("+", 1)[0] in architectures: + nuninst[arch] = set(packages.split()) + return nuninst + -- 1.7.10.4 From 2aa45713412e7eecdbd936666b982c4d4e991bce Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Sat, 12 Jan 2013 17:47:24 +0100 Subject: [PATCH 13/18] Move write_heidi to britney_util Signed-off-by: Niels Thykier --- britney.py | 39 ++++----------------------------------- britney_util.py | 41 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 43 insertions(+), 37 deletions(-) diff --git a/britney.py b/britney.py index a2276bc..58c354f 100755 --- a/britney.py +++ b/britney.py @@ -213,7 +213,7 @@ from hints import HintCollection from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, register_reverses, compute_reverse_tree, - read_nuninst, write_nuninst) + read_nuninst, write_nuninst, write_heidi) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -806,39 +806,6 @@ class Britney(object): return hints - def write_heidi(self, filename): - """Write the output HeidiResult - - This method write the output for Heidi, which contains all the - binary packages and the source packages in the form: - - - source - """ - self.__log("Writing Heidi results to %s" % filename) - f = open(filename, 'w') - - # local copies - sources = self.sources['testing'] - - # write binary packages - for arch in sorted(self.options.architectures): - binaries = self.binaries['testing'][arch][0] - for pkg_name in sorted(binaries): - pkg = binaries[pkg_name] - pkgv = pkg[VERSION] - pkgarch = pkg[ARCHITECTURE] or 'all' - pkgsec = pkg[SECTION] or 'faux' - f.write('%s %s %s %s\n' % (pkg_name, pkgv, pkgarch, pkgsec)) - - # write sources - for src_name in sorted(sources): - src = "" - srcv = src[VERSION] - srcsec = src[SECTION] or 'unknown' - f.write('%s %s source %s\n' % (src_name, srcv, srcsec)) - - f.close() def write_controlfiles(self, basedir, suite): """Write the control files @@ -2295,7 +2262,9 @@ class Britney(object): self.write_dates(self.options.testing, self.dates) # write HeidiResult - self.write_heidi(self.options.heidi_output) + self.__log("Writing Heidi results to %s" % self.options.heidi_output) + write_heidi(self.options.heidi_output, self.sources["testing"], + self.binaries["testing"]) self.printuninstchange() self.__log("Test completed!", type="I") diff --git a/britney_util.py b/britney_util.py index d4739e1..86d3f68 100644 --- a/britney_util.py +++ b/britney_util.py @@ -25,8 +25,8 @@ import re import time -from consts import (BINARIES, PROVIDES, DEPENDS, CONFLICTS, - RDEPENDS, RCONFLICTS) +from consts import (VERSION, BINARIES, PROVIDES, DEPENDS, CONFLICTS, + RDEPENDS, RCONFLICTS, ARCHITECTURE, SECTION) binnmu_re = re.compile(r'^(.*)\+b\d+$') @@ -292,3 +292,40 @@ def read_nuninst(filename, architectures): nuninst[arch] = set(packages.split()) return nuninst + +def write_heidi(filename, sources_t, packages_t, + VERSION=VERSION, SECTION=SECTION, + ARCHITECTURE=ARCHITECTURE, sorted=sorted): + """Write the output HeidiResult + + This method write the output for Heidi, which contains all the + binary packages and the source packages in the form: + + + source + + The file is written as "filename", it assumes all sources and + packages in "sources_t" and "packages_t" to be the packages in + "testing". + + The "X=X" parameters are optimizations to avoid "load global" in + the loops. + """ + with open(filename, 'w') as f: + + # write binary packages + for arch in sorted(packages_t): + binaries = packages_t[arch][0] + for pkg_name in sorted(binaries): + pkg = binaries[pkg_name] + pkgv = pkg[VERSION] + pkgarch = pkg[ARCHITECTURE] or 'all' + pkgsec = pkg[SECTION] or 'faux' + f.write('%s %s %s %s\n' % (pkg_name, pkgv, pkgarch, pkgsec)) + + # write sources + for src_name in sorted(sources_t): + src = "" + srcv = src[VERSION] + srcsec = src[SECTION] or 'unknown' + f.write('%s %s source %s\n' % (src_name, srcv, srcsec)) -- 1.7.10.4 From 4dc69e39d33f63125b1b475d6472ecf057e1b918 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Sat, 12 Jan 2013 19:58:15 +0100 Subject: [PATCH 14/18] Move eval_uninst to britney_util Signed-off-by: Niels Thykier --- britney.py | 31 +++++++++++-------------------- britney_util.py | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 20 deletions(-) diff --git a/britney.py b/britney.py index 58c354f..0b9b588 100755 --- a/britney.py +++ b/britney.py @@ -213,7 +213,8 @@ from hints import HintCollection from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, register_reverses, compute_reverse_tree, - read_nuninst, write_nuninst, write_heidi) + read_nuninst, write_nuninst, write_heidi, + eval_uninst) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -1672,20 +1673,6 @@ class Britney(object): res.append("%s-%d" % (arch[0], n)) return "%d+%d: %s" % (total, totalbreak, ":".join(res)) - def eval_uninst(self, nuninst): - """Return a string which represents the uninstallable packages - - This method returns a string which represents the uninstallable - packages reading the uninstallability statistics `nuninst`. - - An example of the output string is: - * i386: broken-pkg1, broken-pkg2 - """ - parts = [] - for arch in self.options.architectures: - if arch in nuninst and len(nuninst[arch]) > 0: - parts.append(" * %s: %s\n" % (arch,", ".join(sorted(nuninst[arch])))) - return "".join(parts) def is_nuninst_asgood_generous(self, old, new): diff = 0 @@ -2058,7 +2045,8 @@ class Britney(object): self.output_write(" finish: [%s]\n" % ",".join([ x.uvname for x in selected ])) self.output_write("endloop: %s\n" % (self.eval_nuninst(self.nuninst_orig))) self.output_write(" now: %s\n" % (self.eval_nuninst(nuninst_comp))) - self.output_write(self.eval_uninst(self.newlyuninst(self.nuninst_orig, nuninst_comp))) + self.output_write(eval_uninst(self.options.architectures, + self.newlyuninst(self.nuninst_orig, nuninst_comp))) self.output_write("\n") return (nuninst_comp, extra) @@ -2121,7 +2109,8 @@ class Britney(object): self.output_write("easy: %s\n" % nuninst_end_str) if not force: - self.output_write(self.eval_uninst(self.newlyuninst(nuninst_start, nuninst_end)) + "\n") + self.output_write(eval_uninst(self.options.architectures, + self.newlyuninst(nuninst_start, nuninst_end)) + "\n") if force or self.is_nuninst_asgood_generous(self.nuninst_orig, nuninst_end): # Result accepted either by force or by being better than the original result. @@ -2136,7 +2125,8 @@ class Britney(object): self.output_write(" end: %s\n" % nuninst_end_str) if force: self.output_write("force breaks:\n") - self.output_write(self.eval_uninst(self.newlyuninst(nuninst_start, nuninst_end)) + "\n") + self.output_write(eval_uninst(self.options.architectures, + self.newlyuninst(nuninst_start, nuninst_end)) + "\n") self.output_write("SUCCESS (%d/%d)\n" % (len(actions or self.upgrade_me), len(extra))) self.nuninst_orig = nuninst_end if not actions: @@ -2271,8 +2261,9 @@ class Britney(object): def printuninstchange(self): self.__log("Checking for newly uninstallable packages", type="I") - text = self.eval_uninst(self.newlyuninst( - self.nuninst_orig_save, self.nuninst_orig)) + text = eval_uninst(self.options.architectures, self.newlyuninst( + self.nuninst_orig_save, self.nuninst_orig)) + if text != '': self.output_write("\nNewly uninstallable packages in testing:\n%s" % \ (text)) diff --git a/britney_util.py b/britney_util.py index 86d3f68..c2c9cb0 100644 --- a/britney_util.py +++ b/britney_util.py @@ -293,6 +293,22 @@ def read_nuninst(filename, architectures): return nuninst +def eval_uninst(architectures, nuninst): + """Return a string which represents the uninstallable packages + + This method returns a string which represents the uninstallable + packages reading the uninstallability statistics "nuninst". + + An example of the output string is: + * i386: broken-pkg1, broken-pkg2 + """ + parts = [] + for arch in architectures: + if arch in nuninst and nuninst[arch]: + parts.append(" * %s: %s\n" % (arch,", ".join(sorted(nuninst[arch])))) + return "".join(parts) + + def write_heidi(filename, sources_t, packages_t, VERSION=VERSION, SECTION=SECTION, ARCHITECTURE=ARCHITECTURE, sorted=sorted): -- 1.7.10.4 From fb66622c4c489efb81235f24b8c6e385660f1df2 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Sat, 12 Jan 2013 20:13:43 +0100 Subject: [PATCH 15/18] Move newlyuninst to britney_util Also, renamed it for consistency with the other X_uninst functions. Signed-off-by: Niels Thykier --- britney.py | 24 +++++------------------- britney_util.py | 15 +++++++++++++++ 2 files changed, 20 insertions(+), 19 deletions(-) diff --git a/britney.py b/britney.py index 0b9b588..7d683c3 100755 --- a/britney.py +++ b/britney.py @@ -214,7 +214,7 @@ from britney import buildSystem from britney_util import (old_libraries_format, same_source, undo_changes, register_reverses, compute_reverse_tree, read_nuninst, write_nuninst, write_heidi, - eval_uninst) + eval_uninst, newly_uninst) from consts import (VERSION, SECTION, BINARIES, MAINTAINER, FAKESRC, SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, PROVIDES, RDEPENDS, RCONFLICTS) @@ -1576,20 +1576,6 @@ class Britney(object): # Upgrade run # ----------- - def newlyuninst(self, nuold, nunew): - """Return a nuninst statstic with only new uninstallable packages - - This method subtracts the uninstallable packages of the statistic - `nunew` from the statistic `nuold`. - - It returns a dictionary with the architectures as keys and the list - of uninstallable packages as values. - """ - res = {} - for arch in nuold: - if arch not in nunew: continue - res[arch] = [x for x in nunew[arch] if x not in nuold[arch]] - return res def get_nuninst(self, requested_arch=None, build=False): """Return the uninstallability statistic for all the architectures @@ -2046,7 +2032,7 @@ class Britney(object): self.output_write("endloop: %s\n" % (self.eval_nuninst(self.nuninst_orig))) self.output_write(" now: %s\n" % (self.eval_nuninst(nuninst_comp))) self.output_write(eval_uninst(self.options.architectures, - self.newlyuninst(self.nuninst_orig, nuninst_comp))) + newly_uninst(self.nuninst_orig, nuninst_comp))) self.output_write("\n") return (nuninst_comp, extra) @@ -2110,7 +2096,7 @@ class Britney(object): if not force: self.output_write(eval_uninst(self.options.architectures, - self.newlyuninst(nuninst_start, nuninst_end)) + "\n") + newly_uninst(nuninst_start, nuninst_end)) + "\n") if force or self.is_nuninst_asgood_generous(self.nuninst_orig, nuninst_end): # Result accepted either by force or by being better than the original result. @@ -2126,7 +2112,7 @@ class Britney(object): if force: self.output_write("force breaks:\n") self.output_write(eval_uninst(self.options.architectures, - self.newlyuninst(nuninst_start, nuninst_end)) + "\n") + newly_uninst(nuninst_start, nuninst_end)) + "\n") self.output_write("SUCCESS (%d/%d)\n" % (len(actions or self.upgrade_me), len(extra))) self.nuninst_orig = nuninst_end if not actions: @@ -2261,7 +2247,7 @@ class Britney(object): def printuninstchange(self): self.__log("Checking for newly uninstallable packages", type="I") - text = eval_uninst(self.options.architectures, self.newlyuninst( + text = eval_uninst(self.options.architectures, newly_uninst( self.nuninst_orig_save, self.nuninst_orig)) if text != '': diff --git a/britney_util.py b/britney_util.py index c2c9cb0..66d949b 100644 --- a/britney_util.py +++ b/britney_util.py @@ -293,6 +293,21 @@ def read_nuninst(filename, architectures): return nuninst +def newly_uninst(nuold, nunew): + """Return a nuninst statstic with only new uninstallable packages + + This method subtracts the uninstallable packages of the statistic + "nunew" from the statistic "nuold". + + It returns a dictionary with the architectures as keys and the list + of uninstallable packages as values. + """ + res = {} + for arch in ifilter_only(nunew, nuold): + res[arch] = [x for x in nunew[arch] if x not in nuold[arch]] + return res + + def eval_uninst(architectures, nuninst): """Return a string which represents the uninstallable packages -- 1.7.10.4

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -27,7 +27,7 @@
     version numbers specified as parameters are built from the same
     source. The main use of this code is to detect binary-NMU.
 
-    binnmu_re is an opt to avoid "load global".
+    binnmu_re is an optimization to avoid "load global".
     """
     if sv1 == sv2:
         return 1

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -54,7 +54,7 @@
     If given an iterable it returns a filtered iterator, otherwise it
     returns a function to generate filtered iterators.  The latter is
     useful if the same filter has to be (re-)used on multiple
-    iterators that are not known on before hand.
+    iterators that are not known on beforehand.
     """
     if iterable is not None:
         return ifilterfalse(container.__contains__, iterable)
@@ -67,7 +67,7 @@
     If given an iterable it returns a filtered iterator, otherwise it
     returns a function to generate filtered iterators.  The latter is
     useful if the same filter has to be (re-)used on multiple
-    iterators that are not known on before hand.
+    iterators that are not known on beforehand.
     """
     if iterable is not None:
         return ifilter(container.__contains__, iterable)

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -171,8 +171,8 @@
     sequence of packages
 
     This method registers the reverse dependencies and conflicts for a
-    given sequence of packages.  `packages` is a table of real
-    packages and `provides` is a table of virtual packages.
+    given sequence of packages.  "packages" is a table of real
+    packages and "provides" is a table of virtual packages.
 
     iterator is the sequence of packages for which the reverse
     relations should be updated.

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -226,15 +226,15 @@
     """Calculate the full dependency tree for the given package
 
     This method returns the full dependency tree for the package
-    `pkg`, inside the `arch` architecture for a given suite flattened
-    as an iterable.  The first argument `packages_s` is the binary
+    "pkg", inside the "arch" architecture for a given suite flattened
+    as an iterable.  The first argument "packages_s" is the binary
     package table for that given suite (e.g. Britney().binaries["testing"]).
 
-    The tree (or grapH) is returned as an iterable of (package, arch)
-    tuples and the iterable will contain (`pkg`, `arch`) if it is
+    The tree (or graph) is returned as an iterable of (package, arch)
+    tuples and the iterable will contain ("pkg", "arch") if it is
     available on that architecture.
 
-    If `pkg` is not available on that architecture in that suite,
+    If "pkg" is not available on that architecture in that suite,
     this returns an empty iterable.
 
     The method does not promise any ordering of the returned

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -265,8 +265,8 @@
 def write_nuninst(filename, nuninst):
     """Write the non-installable report
 
-    Write the non-installable report derived from `nuninst` to the
-    file denoted by `filename`.
+    Write the non-installable report derived from "nuninst" to the
+    file denoted by "filename".
     """
     with open(filename, 'w') as f:
         # Having two fields with (almost) identical dates seems a bit
@@ -280,7 +280,7 @@
     """Read the non-installable report
 
     Read the non-installable report from the file denoted by
-    `filename` and return it.  Only architectures in `architectures`
+    "filename" and return it.  Only architectures in "architectures"
     will be included in the report.
     """
     nuninst = {}

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -304,8 +304,8 @@
     <pkg-name> <pkg-version> <pkg-architecture> <pkg-section>
     <src-name> <src-version> source <src-section>
 
-    The file is written as `filename`, it assumes all sources and
-    packages in `sources_t` and `packages_t` to be the packages in
+    The file is written as "filename", it assumes all sources and
+    packages in "sources_t" and "packages_t" to be the packages in
     "testing".
 
     The "X=X" parameters are optimizations to avoid "load global" in

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -297,7 +297,7 @@
     """Return a string which represents the uninstallable packages
 
     This method returns a string which represents the uninstallable
-    packages reading the uninstallability statistics `nuninst`.
+    packages reading the uninstallability statistics "nuninst".
 
     An example of the output string is:
       * i386: broken-pkg1, broken-pkg2

diff -u b/britney_util.py b/britney_util.py
--- b/britney_util.py
+++ b/britney_util.py
@@ -297,7 +297,7 @@
     """Return a nuninst statstic with only new uninstallable packages
 
     This method subtracts the uninstallable packages of the statistic
-    `nunew` from the statistic `nuold`.
+    "nunew" from the statistic "nuold".
 
     It returns a dictionary with the architectures as keys and the list
     of uninstallable packages as values.