Re: How would you use britney for Kali and are generalization patches welcome?
On 2014-07-16 15:48, Raphael Hertzog wrote:
> Hello britney maintainers,
>
Hi, :)
Your problem in a nutshell: You triggered something that looks like
O(2^n) runtime in Britney. /o\
* It happens due to some aspell/ispell packages not being
co-installable with "affected packages" being tested
- "affected packages" being reverse dependencies (or reverse
conflicts) of libtext-iconv-perl
- I haven't figured out why those spell-packages are not
co-installable with anything (and yet not broken enough for
the installability-tester to optimise them out).
* I highly doubt that the problem is limited to libtext-iconv-perl
- That will migration will fail btw. because perl5.18 hasn't migrated
yet and that will require a hint.
Short-term options include:
* patch Britney reject on first new uninstallable
- something like the attached patch, which /vastly/ reduces the
runtime of testing libtext-iconv-perl
- downside: you have less to go on when she reports "migrating A
breaks B", because you only get one "broken" package (which
possibly is broken indirectly)
- *caveat 1*: It produces *different results* than the original
britney2
- especially in our live-data test suite. The results look a
bit weird (but at least the diffs appear to be relatively
small, so it might just be "uninstallability trades" that
no longer happen...)
- caveat 2: I did not have time to run it through your data
completely. It did get past libtext-iconv-perl though.
- you might have to disable the auto-hinters temporarily.
* figuring out a hint that solves the mess.
- I will try to hook up the auto-hinters on your dataset and see
what they suggest tomorrow.
* bootstrap from something closer to the "source" suite
Once your "target" suite gets more similar to your "source" suite, then
I suspect this problem will go away. If you run Britney with
--control-files you can repeat the run a couple of times and see if she
improves the result after the first run (assuming you can get the first
run to terminate).
- NB: she *will* overwrite the data files for the target suite with
that parameter.
> I would like your advice and comments about a possible usage of britney
> in the context of a Debian derivative.
>
Certainly :)
> [...]
>
> Now we want our underlying distribution to be "testing" and we obviously
> want to ensure installability of all packages in "kali". Thus I believe
> that we should use britney to build the (new) kali repository.
Seems like a reasonable use case.
> That said
> we are not interested in RC bug tracking and minimum delays (on this
> aspect we're probably closer to Ubuntu's usage of britney, thus ccing
> Colin Watson who might have some code to share and some advice as well).
>
> We are thus doing some tries with britney and I would like to have your
> opinion on how you would approach this problem.
>
> Our initial approach was to map repositories this way:
> britney's unstable => debian-testing
> britney's tpu => kali-only
> britney's testing => kali
>
> But this did not work at all because all the kali packages are missing
> from britney's unstable and thus britney is trying to remove the packages
> instead of updating them. So we changed the Sources/Packages files for
> britney's unstable to be a concatenation of debian-testing +
> kali-only (and I really mean a simple "cat" here, some
> source/binary packages might appear multiple times albeit with different
> versions) and britney's tpu became empty.
>
Indeed, Britney was written before "partial"/"overlay" suites or
whatever they are called. So currently you have to built a "complete"
suite yourself. In fact, she was written for the "old" mirror layout,
accordingly the "weird" layout for her data files.
Personally, I am interested in having her support a regular mirror
layout, so people can use Britney without having to mash their archives
into the old format. But so far it has had low priority.
> This no longer fails but it doesn't produce any satisfactorily result
> either because britney seems to hang in some sort of loop in the
> process. I attach the last lines of the log (up to a CTRL+C to interrupt
> it), the britney configuration and a script used to download the
> Packages/Sources files. The full log is at
> http://ouaza.com/~rhertzog/britney.log.xz if needed. Thus you can
> replicate the problem if you wish.
>
Thanks for the data and the setup script. It is much appreciated and
very helpful in figuring out what is wrong and for obtaining another
test case. :)
> If you have some advice on how to best debug that kind of problem,
> I'm all ears.
>
> Also I would like to know if you would be interested in patches
> that would make britney more easily reusable in other contexts
> than Debian itself. I am thinking of things like:
/Personally/, I am generally pro this (though with comments to some of
the points below). Though keep in mind that these changes are basically
0 benefit for Debian (short term at least).
> - allowing usage of multiple directories to define britney's unstable
I would be willing to accept/support it if it was combined with support
for processing the regular mirror layout. Mostly, I see no point in
having this, if you still have to mash your mirror into the layout we
used some 10 years ago.
Mind you, this will possible require that some of Britney's data files
being moved else where (e.g. BugsV, Dates)
> - renaming Debian jargon to generic concepts: unstable -> source, testing
> -> target, tpu/pu => updates, etc.
In general (and still personally) yes, but I am not ready to comment on
the concrete renaming of concepts.
> - make it easier to disable features that are not needed (right now I have
> to create quite a few empty files, and configure delays to 0 days, etc.)
>
Again, I could be interested here as well. Ubuntu have some patches for
this as well (though as I recall, we did not quite like the approach
they used, but it was quite a while since I last saw their patches).
Speaking of which, do you have a link to your Britney VCS tree, so I can
have a look at what patches you have (or haven't) applied. I noticed
that the line numbers in your stack trace was off compared to my version! :)
> Cheers,
>
> PS: Just for the avoidance of doubts, the way to reproduce is to put the
> attached files in a directory (it was next to britney.py for me) and to do
> this:
> # bash ./update-data.sh
> # ./britney.py -c britney.conf -v >britney.log 2>&1
>
Thanks again that script,
~Niels
diff --git a/britney.py b/britney.py
index d0fcdd0..f0c0c8d 100755
--- a/britney.py
+++ b/britney.py
@@ -2037,7 +2037,7 @@ class Britney(object):
return (item, affected, undo)
- def _check_packages(self, binaries, arch, affected, skip_archall, nuninst):
+ def _check_packages(self, binaries, arch, affected, skip_archall, nuninst, hint=False):
broken = nuninst[arch + "+all"]
to_check = []
@@ -2051,7 +2051,7 @@ class Britney(object):
nuninst_arch = None
if not (skip_archall and parch == 'all'):
nuninst_arch = nuninst[arch]
- self._installability_test(p, version, arch, broken, to_check, nuninst_arch)
+ self._installability_test(p, version, arch, broken, to_check, nuninst_arch, hint=hint)
# broken packages (second round, reverse dependencies of the first round)
while to_check:
@@ -2066,7 +2066,7 @@ class Britney(object):
nuninst_arch = None
if not (skip_archall and parch == 'all'):
nuninst_arch = nuninst[arch]
- self._installability_test(p, version, arch, broken, to_check, nuninst_arch)
+ self._installability_test(p, version, arch, broken, to_check, nuninst_arch, hint=hint)
def iter_packages(self, packages, selected, hint=False, nuninst=None, lundo=None):
@@ -2163,13 +2163,15 @@ class Britney(object):
nuninst[arch] = set(x for x in nuninst_comp[arch] if x in binaries[arch][0])
nuninst[arch + "+all"] = set(x for x in nuninst_comp[arch + "+all"] if x in binaries[arch][0])
- check_packages(arch, affected, skip_archall, nuninst)
-
- # if we are processing hints, go ahead
- if hint:
- nuninst_comp[arch] = nuninst[arch]
- nuninst_comp[arch + "+all"] = nuninst[arch + "+all"]
- continue
+ try:
+ check_packages(arch, affected, skip_archall, nuninst, hint=hint)
+ # if we are processing hints, go ahead
+ if hint:
+ nuninst_comp[arch] = nuninst[arch]
+ nuninst_comp[arch + "+all"] = nuninst[arch + "+all"]
+ continue
+ except Exception:
+ better = False
# if the uninstallability counter is worse than before, break the loop
if ((item.architecture != 'source' and arch not in new_arches) or \
@@ -2738,7 +2740,7 @@ class Britney(object):
self.__output.close()
- def _installability_test(self, pkg_name, pkg_version, pkg_arch, broken, to_check, nuninst_arch):
+ def _installability_test(self, pkg_name, pkg_version, pkg_arch, broken, to_check, nuninst_arch, hint=False):
"""Test for installability of a package on an architecture
(pkg_name, pkg_version, pkg_arch) is the package to check.
@@ -2754,11 +2756,13 @@ class Britney(object):
r = self._inst_tester.is_installable(pkg_name, pkg_version, pkg_arch)
if not r:
# not installable
+ if nuninst_arch is not None and pkg_name not in nuninst_arch:
+ nuninst_arch.add(pkg_name)
if pkg_name not in broken:
broken.add(pkg_name)
to_check.append(pkg_name)
- if nuninst_arch is not None and pkg_name not in nuninst_arch:
- nuninst_arch.add(pkg_name)
+ if not hint:
+ raise Exception()
else:
if pkg_name in broken:
to_check.append(pkg_name)
Reply to: