Hi, Quoting Paul Gevers (2018-06-12 14:56:44) > tl;dr: autopkgtest restriction "needs-recommends" and using aspcud > solver is horror. How to move forward? > > On 24-05-18 12:09, Paul Gevers wrote: > > tl;dr; I want to fix the "Test dependencies are unsatisfiable with using > > apt pinning. Retrying with using all packages from unstable" situation > > because it has unwanted side effects¹. But how to do it? > > > And yes, for both paths this means figuring out which packages are > > needed at all (maybe apt --simulate) and translating src:package to > > binary packages in more than the current place. > > Checking if a package came from the right release/pocket and (if > requested) has the right version has a possible implementation, ~100 > lines of code. > > > Maybe (I haven't checked) it may even fix bug 896698, where autopkgtest > > --needs-recommends is silently failing to install the recommends (@jak: > > do you know apt's behavior here?). > > And now this turns out to be causing me headaches. apt doesn't fail when > recommends can't be installed (wishlist bug to be filed), but worse, > with the alternative solver it doesn't even try. Ideally apt gets fixed > for this latter issue (bug #900989), but until then, and after several > discussion on IRC (#debian-devel, with DonKult and josch), I think the > best solution is to let autopkgtest determine the recommends of the test > dependencies itself or deprecate the need-recommends restriction. The > biggest issue with the former solution is that it can't know which > version of a package to look at, i.e. the recommends from the package in > unstable or the recommends from the package in testing. So I wonder what > you all think autopkgtest should do. > > These are most of the possibilities I see: > > 1) Don't do anything and forget about the alternative solver (bug 896698 > will still not be solved). Wait until apt is fixed, I don't expect that > to happen soon as it seems the spec is missing this part and is not core > apt, so a NO in my opinion. > > 2) Stop supporting needs-recommends and use the aspcud solver in a way > similar to what I implemented on salsa. Clearly this restriction has > been out there for a while. Will be painful for some (how many?) > packages as their autopkgtest will suddenly break. > > 3) Fix bug 896698 by using apt's internal solver and check afterwards if > the recommends of installed packages in the test dependencies are > installed and create a clear error message if they aren't (something > like my bts-896698-fix-needs-recommends branch). I don't like this > solution because it leaves the issue of too much packages from unstable > unsolved. > > 4) Use the aspcud solver but add the recommends from the test > dependencies to the list of test dependencies before asking apt to > install them. This scheme could detect if the list of recommends of a > package has changed between releases/pockets and error out, join them or > pick one set (which one?). > > 5) Use the aspcud solver to figure out which versions of test > dependencies to install and after they are installed, install their > recommends. It may be possible that installing the recommends of the > installed set is not possible, while there is a solution, what to do in > that case? I think this is my preferred solution. > > 6) More combinations, own solver (NO!!!) or what have you. > > Looking forward to responses. I don't think it was a wise choice to add the needs-recommends restriction in the first place. Recommends are not Depends for a reason: it is okay not to install them without the package in question becoming unusable. They are *optional*. If a test *requires* them, then they should be a Depends of the test. The docs say: > needs-recommends > Enable installation of recommended packages in apt for the test > dependencies. This does not affect build dependencies. Does this mean to only install the direct Recommends? Lets for a moment assume the answer is "yes": If yes, then you would miss the Recommends of a package on which you only depend through a meta package (an empty package with the right Depends). So will we have to implement a special rule for packages that are depended upon through meta packages? This also raises the question how you even want to list the packages that you want to gather the Recommends for. Suppose your test includes a virtual package that is provided by multiple real packages. Which package do you pick? If you pick the wrong package, then your Recommends might conflict with other packages in your installation set and you have to backtrack and choose a different one. Basically at this point you are writing a solver. The situation does become even more tricky if you can pick versions from different suites. Lets assume the answer is "no" and it also means the indirect Recommends of reverse dependencies. This would be relatively straight forward to implement in apt-cudf/aspcud. It would be an option that lets Recommends from the EDSP be treated like Depends. Unfortunately, we never test for this kind of setup. Since Recommends originally are optional, many packages suddenly become uninstallable as a result even when they were installable before. I wrote a small Perl script (attached) that modifies a Packages file such that Recommends are merged into Depends. If I run the script on a Packages file from Debian unstable and then check installability using dose-distcheck, then we see that suddenly 1102 more Packages become uninstallable. In my opinion, the needs-recommends restriction is a misguided attempt to have a lazy solution for not having to specifically list hard test *dependencies*. If a package really requires a Recommends, then make it a Depends. If a test really requires a package, list it as a dependency. If it's optional, then put it into Recommends but then not installing it should not make a difference for the test. According to codesearch there are 154 source packages that use the needs-recommends restriction. 84 of them are php-horde-* packages. Probably not all of them fail when removing the restriction. A mass bug filing might not only be *much* less work than implementing the requested feature in apt, apt-cudf or aspcud "somehow", but would also do away with an option that I don't think makes much sense for the reasons I listed above. Paul, I cannot tell you what to do with your free time and I see that you invested already a lot of work on this issue. But maybe you would spend less time by supplying patches to the 154 source packages that currently use the needs-recommends restriction and then deprecate this field for the future. Thanks! cheers, josch
Attachment:
signature.asc
Description: signature