Re: next generation apt/dpkg-cross
Neil Williams <codehelp@debian.org> writes:
> On Tue, 19 Jan 2010 23:21:17 +0100
> Goswin von Brederlow <goswin-v-b@web.de> wrote:
>
>> >> I've been working on a next generation apt/dpkg-cross that will
>> >> support apt, aptitude, synaptic (anything libapt based) and dpkg
>> >> directly.
>> >
>> > Why? There is no future for apt-cross or dpkg-cross - there should
>> > be no next generation. We get through the multiarch transition and
>> > both dpkg-cross and apt-cross disappear for GOOD.
>>
>> Why? Because dpkg-cross didn't do enough for running 32bit binaries on
>> amd64 and apt-cross was, well, apt-cross. :) The multiarch transition
>> will be a lengthy process so there is still a few years use left in
>> this.
>
> As long as we're clear that we're both talking about interim measures
> and that the goal is that all this stuff goes away.
Absolutely. I have 2 reasons for why I want the multiarch parts added
to apt/dpkg-cross:
1) ia32-libs has no security support and security bugs and missed some
libs needed for some apps.
2) I want to use multiarch packages as soon as they are available and
not be stuck till all of ia32-libs has been changed.
So you see this is also motivated by getting rid of it faster.
> If you're talking about your apt-cross and I'm talking about the
> apt-cross in Debian, things are going to get missed. Can we define the
> terms? Are you talking about fixes in apt that replace apt-cross (or
> allow apt to work with full multiarch and therefore make apt-cross
> redundant) or are you talking about yet another script that tries to
> work around the brokenness in apt?
I'm talking about a completly new apt-cross-ng, nothing to do with the
apt-cross mess in Debian currently. And the way it works is that it
uses the first of the multiarch support in apt to have apt itself
download the index files for multiple architectures (That removes a
big mess apt-cross in Debian does) and then adding a little
post-processing of the index files via the Apt::Update::Post-Invoke
hook. So I don't work around apt brokenness but I still need a little
helper to make apt do the right thing on its own. The idea is to be
minimally invasive.
As multiarch progresses the Apt::Update::Post-Invoke hook will do less
and less until finaly it can be removed completly. The Hook does to
the binary-arm/Packages file what dpkg-cross does to DEBIAN/control in
each package. No more and no less. It just makes apt see the same
informations that dpkg-cross would produce when a package gets
installed.
>> >> - Wrappers in /usr/lib/apt-cross/. Add that dir to your PATH or
>> >> link them from /usr/local/bin and you can install foreign packages
>> >> transparently with just dpkg. This has some sideeffects so maybe
>> >> don't use this.
>> >
>> > I can't see that as a good idea but I also don't see why it is
>> > considered necessary. /usr/lib/apt-cross/ is the wrong name. It has
>> > to be a name based on an arch triplet, but we're already using
>> > those.
>>
>> Sorry, I ment /usr/share/apt-cross/. Nothing arch specific in there.
>
> So these would be shell or perl scripts?
--- dpkg wrapper --------------------------------------------------
#! /bin/sh
# Wrapper for dpkg
PATH="/usr/share/apt-cross-ng:$PATH" exec /usr/bin/dpkg "$@"
-------------------------------------------------------------------
And 100 lines shell script for dpkg-deb.
>> >> - apt-get, aptitude, synaptic (anything libapt based) are
>> >> automatically setup to support -cross packages (just needs
>> >> configuration to know which archs).
>> >
>> > This is the main area of work, but not for -cross packages, for true
>> > multiarch packages.
>>
>> I find those verry important to upgrade. Having to download all debs
>> and dpkg-cross -i them would be horrible.
>
> This is precisely the problem that the new updates for apt-cross and
> dpkg-cross try to solve - by populating that suite-specific list of
> packages that are multiarch-compliant and not creating dependencies on
> native packages from -cross packages unless those native packages are
> multiarch-compliant.
>
> We must avoid having -cross packages depending on non-multiarched
> native packages - even the dummy -cross ones. Arch:all is fine.
I don't see why. Since I'm populating the apt database with the
native, the foreign multiarch and the -cross packages the update
process can use either one or any combination of them. So when a new
version of a package comes out the -cross package will be upgraded
just like a native or foreign multiarch package. When the -cross
package is no longer needed (no reverse depends on it) the package
will show up in the list of automatic removals (unless set to manual)
in apt or aptitude.
Since you do not have suite informations on the dpkg-cross level I
would stick with the -cross name for a dependency as long as any used
suite has that package non-multiarch. Only when all used sites have a
package as multiarch should the dependency use the multiarch package
directly. For any chroot with just one suite that would mean it would
always use the multiarch packages as soon as available, which is what
you (and me) want. On the other hand for setups with multiple suites
it would remain on the safe side and rather use -cross when in doubt.
Would that be a acceptable middle ground?
>> > dpkg has to do that, not dpkg-cross.
>> >
>> > Why are we still looking at -cross packages?
>>
>> Because it will be years before no more -cross packages are needed.
>
> Increasingly, -cross packages will become empty dummy packages though.
> I know it's going to take longer than I'd hoped.
You and me both.
>> alternative. I would expect "apt-get -o Apt::Architecture=armel ..."
>> to cause apt-get to consider armel the native architecture.
>
> You'd think that but from what I've done with apt-cross, apt does seem
> to leave that job to dpkg. i.e. if a binary-dependent armel package
> depends on a binary-independent all package, apt handles it correctly
> whether the machine is native armel or native amd64.
>
> What's been missing is how to get apt to do that *after* we've mangled
> the package names by adding '$arch-cross'. Once we stop doing that,
> apt is just fine with the dependency resolution in download mode. (See
> multistrap.) Once we fix that and get dpkg to be able to install the
> downloaded .debs, apt can take over from apt-cross.
I never had a problem with that in my implementation. As said I
process the Packages file so they exactly reflect the meta
informations dpkg-cross will create during installation. That way the
dependency resolution and downloading in apt matches exactly what dpkg
expects and needs. Apt knows about the libfoo-arm-cross package and
that it depends on libbar-arm-cross and that it needs to download
libfoo_1.2-3_arm.deb and libbarm_1.2-3_arm.deb and feet it to
dpkg-cross. By feeding apt the same meta information as dpkg ends up
with that proble solves itself.
>> That would
>> likely try to install make:arm instead of make:<native> then for
>> example.
>
> That's what would be needed - once make is multiarch-compliant. As long
> as the armel make executable gets put into /usr/bin/$triplet (or
> $triplet/usr/bin or /usr/$triplet/bin) or thrown away, that's fine.
>
>> >> Last there is also a tool (convert-cross) to convert deb packages
>> >> instead of installing them. The same mechanisms used during install
>> >> (see implementation below) are used to convert libfoo_1.2-3_arm.deb
>> >> into libfoo-arm-cross_1.2-3_all.deb (or a dummy
>> >> libfoo-arm-cross_1.2-3_arm.deb if multiarch). Those packages can
>> >> then be put into a repository and installed with plain apt and
>> >> dpkg. Those that don't like on-the-fly conversion can use that to
>> >> get identical results.
>> >
>> > Please can we get rid of all -cross ideas? The whole point of
>> > multiarch is that all -cross stuff goes away.
>>
>> I would love nothing more but I don't see it done yet.
>
> OK then, I'm not sure what convert-cross needs to do that dpkg-cross
> and apt-cross do not do now. With the current changes, libfoo would be
> handled as you describe above - except that these are local packages.
It does what apt-cross/dpkg-cross do now but not install a package. It
would be to just convert a package so it can be put into a repository
for later use. On the other hand my dpkg wrapper or dpkg-cross
converts and installs directly.
> That's the whole problem with having -cross anything in the package
> name, it breaks all this stuff. We need to stop renaming packages ASAP.
Look at it this way: What I add to the existing apt-cross/dpkg-cross
is that renaming packages to -cross does no longer break things. So
while packages should still become multiarch ASAP one isn't hampered
by having -cross packages till that is done.
Does that explain what I aim to improve better?
>> > apt needs to be taught how to keep multiple lists safely - changing
>> > arch causes apt to delete all lists for the other arch, changing
>> > back removes them again. Some of the problems with apt-cross are
>> > because apt-cross tries to stop apt doing this and then the apt
>> > bindings get confused.
>>
>> That is included in the patch already. It adds the arch to the
>> filename of index files.
>
> Good! (except that is going to break apt-cross really, really badly.)
> :-|
*BOOOM* There really is no way around that with multiarch. The current
names plainly don't allow for multiple architectures.
>> >> /etc/apt-cross/black.list:
>> >>
>> >> Filter blacklisting packages from conversion. By default this
>> >> prevents any binary available natively to be converted.
>> >
>> > ? There is support in the MultiArch Spec for such limitations, why a
>> > separate one?
>>
>> Is there? https://wiki.ubuntu.com/MultiarchSpec doesn't list any that
>> I saw.
>
> Why doesn't Multi-Arch: foreign cover those?
>
> (It shouldn't be /etc/apt-cross - possibly /etc/multiarch/
> or /etc/apt/multiarch.d/)
Because "foreign" means that the package may be installed from exactly
one architecture at a time and can be used to satisfy a depends from
any architecture package. That still allows the package to removed and
installed from a different architecture.
For example make will be Multi-Arch: foreign. But on amd64 you can
(eventually) install make:amd64 or make:i386. For make that is
perfectly acceptable and even desirable (for e.g. rar or flash). But
for dpkg, apt or type-handling that wold seriously mess up things.
Note that even without any Multi-Arch setting you can remove a package
and reinstall it from a different arch. The only difference is that
dependencies on that package will only be satisfied for arch:all and
the same arch. So if I install the existing gzip:i386 package under
multiarch the reverse depends (battery-stats, file-roller, rsyncrypto)
would also have to be arch i386 or all.
In conclusion the black.list solves a problem the multiarch apt/dpkg
will face too that isn't covered in the specs. I don't know how they
will solve that so for now i have my own little file.
>> The black.list is a security precaution preventing the user to shoot
>> himself in the foot in obvious ways. While it should work to install a
>> 32bit bash on amd64 that is normaly not desired so the default
>> black.list prevents experiments like that. It also prevents
>> apt/aptitude/synaptic from going crazy with e.g. type-handling. Having
>> type-handling from multiple architectures confuses the hell out of
>> apt.
>
> The blacklist might be a safeguard for executables 32/64bit but I'm not
> convinced it's needed for cross-building.
>
> If the package is fully Multi-arch compliant, shouldn't the foreign
> executable be put somewhere other that /usr/bin ? If it isn't multiarch
> compliant, what's it doing declaring Multi-Arch: foreign?
No, all binaries remain in /bin, /sbin/, /usr/sbin or /usr/bin. The
reason for this is that the path for binaries is hardcoded at many
places, most noticeable the shebang line of any script (#!/bin/bash,
...). If we suddenly had /bin/amd64-linux-gnu/bash nothing would work
anymore. The "Multi-Arch: foreign" ensures that at most one bash
package is installed at any time and that one package provides
/bin/bash.
I think for cross compilation purposes multiarch will need some option
that limits installable packages of an architecture to Multi-Arch:
same only. Installing any Multi-Arch: foreign or non-multiarch arm
package on amd64 would make that package unusable.
>> >> It further explicitly
>> >> excludes dpkg, apt, aptitude, type-handling and a few more from
>> >> ever being converted.
>> >
>> > That won't work. apt provides a SONAME that needs to be available as
>> > armel when cross-building things like aptitude. apt does need to be
>> > installed as a multiarch package.
>>
>> In the long run apt needs to be split into apt and libapt for
>> that. You can't install an apt:arm on amd64 as that would give you a
>> non-functioning /usr/bin/apt. And even if it would work it would
>> suddenly install arm packages natively across the board and refuse to
>> install amd64.
>
> I'm getting confused - I thought the point of this was that a
> Multiarch: same puts it's files in places where this is no conflict
> with the native files. That conflict would include not being in PATH.
>
> i.e. a multiarched dpkg will do what dpkg-cross currently does and move
> things around to avoid conflicts and replacements.
>
> Prior to that, dpkg-cross works as normal - with the changes I'm
> uploading tomorrow.
No. The design of multiarch expects that nothing is moved around at
all and nothing needs moving around. A Multi-Arch: same package is
expected to have its libraries moved around already into the multiarch
directories. That is something the debian/rules file has to do during
build and not something dpkg will do. Again the reason is that paths
are hardcoded in binaries and moving files around breaks some things.
A Multi-Arh: same package from two architectures MUST not have any
files in common unless they are identical. That exception is made so
packages can have /usr/share/doc/package/copyright for example. Or
more general architecture independent files that are too small to
warant a seperate arch:all libfoo-common package.
>> > Umm, it works OK if the installation locations are done properly.
>> > The current apt-armel-cross package IS a necessary one.
>>
>> But that is because you only handle libraries. My use case also
>> includes handling foreign binaries.
>
> OK - wasn't there the idea of /usr/$triplet/bin or similar?
Doesn't work with
#!/usr/bin/foo
>> If apt-armel-cross is really
>> needed before apt has a multiarchified libapt I have to change
>> something to support that. Do you remember why you needed it?
>
> It is because apt thinks it is a library. apt-armel-cross doesn't
> contain apt itself, just the libapt-pkg library.
Sure. I gathered that. But what depends on apt-armel-cross that you
need it? Is it just so you can cross compile aptitude and synaptic?
>> I can call it apt-cross-ng if you like to make clear it is a different
>> package than apt-cross.
>
> It's not a different package that causes me to ask, it's that it's a
> multiarch interim step, not a cross. I don't want things like this
> delaying the removal of apt-cross. That and I find it confusing about
> what you mean by apt-cross when apt-multiarch would be clearer or
> whether we're simply talking about fixes within apt itself.
It is an interim step befre true multiarch that allows for both
running different archs and cross compiling for different archs. Sorry
for the confusion.
>> > Why can this not be calculated from the Multi-Arch fields?
>>
>> Because when you install libfoo_1.2-3_arm.deb, which Depends: libbar,
>> you do not have access to the Multi-Arch field of libbar. And libbar
>> might not be multiarchified yet, which you also don't know.
>
> That is why we have the suite-specific list that is coming in the next
> releases of dpkg-cross and apt-cross.
>
> The list tells you if libbar is multiarched or not. Reliably.
> dpkg-cross will then use that information when considering the Depends:
> line of libfoo.
You say suite-specific but when installing libfoo_1.2-3_arm.deb what
suite does it belong too? But lets ignore this for now. If dpkg-cross
can reliable say when -cross is needed and when not then I'm happy to
use that same mechanism. Our only disagreement is wether dpkg-cross
can reliable say so and that we will see when it is used more.
>> > You cannot unpack into /usr/ anything - it has to /tmp or /var/
>>
>> Who said anything about unpacking to /usr?
>
>> >> /usr/lib/apt-cross/dpkg-deb is used to unpack debs.
>
> Is that a script or an unpacking location?
script location.
> What's a dpkg-deb version doing in /path/apt-cross/ and can't we think
> of a better place for it? /usr/lib/dpkg/ ? /usr/lib/dpkg-multiarch/ ?
The place doesn't matter as long as it is in PATH when dpkg gets
called. But I don't think the dpkg maintainer would like me polluting
their namespace. /usr/share/<package>/ really is the best place with
them being a small shell script.
> I'm trying to see a clear dividing line between -cross - OLD, BAD and
> multiarch - NEW, GOOD! A dividing line that is consistent across package
> names, script names, paths, behaviour, the lot.
>
> It's going to confuse the hell out of me if people keep talking about
> -cross when talking about the transition away from -cross. -cross must
> die! (eventually)
This is still all about the OLD, BAD -cross. It will go away with
multiarch.
> dpkg-cross is to be absorbed into a multiarch-competent dpkg.
I think that is your main misconception. Dpkg will not absorb
dpkg-cross.
Dpkg will become multiarch aware and -cross packages will be replaced
one by one with multiarchified packages. The multiarchified packages
will be instaled identical on arm native or as foreign arch. There
won't be any ugly moving or deleting of files like dpkg-cross needs to
do.
So the multiarch dpkg will never take over the handling of the -cross
packages but the -cross packages will become obsolete. But as long as
even one -cross package is still needed dpkg-cross will be needed. At
least I have not seen any plans to pull the -cross mess into dpkg
while multiarch tries to avoid the mess alltogether.
> apt-cross is to be replaced by a multiarch-aware apt/aptitude
>
> Each is to have an ever-reducing role, migrating from useful packages
> to empty dummy packages and thence to no packages to process at all.
>
> Any new scripts that will outlast either dpkg-cross or apt-cross should
> not use the -cross name, whether in packages, scripts, conversions or
> file paths.
They will not outlast it.
>> > Why? If this is during the transition, we just use dpkg-cross as
>> > now. If it's after the transition, there should be no need for any
>> > -cross package.
>>
>> Why do this on the fly as opposed to the existing way of creating a
>> temporary deb and then install that? I don't want to gunzip
>> data.tar.gz, mangle it, gzip and then have dpkg gunzip again. Doing
>> this on the fly saves a gzip and gunzip run.
>
> Why add another script when what we really need is for dpkg to do the
> entire job internally? i.e. dpkg will do without the temporary .deb
> stage. If that is what you proposed, then I misunderstood that section.
As said above I think you have a misconception about multiarch. Dpkg
won't do the job of dpkg-cross internally but only make it
unneccessary in the end (or rather the multiarchified packages will
do).
>> >> With --fsys-tarfile first the control.tar.gz file is unpacked and
>> >> checked if this is a native package, foreign package and if it is
>> >> multiarch or not. Native packages are left unchanged. Multiarch
>> >> foreign packages without -cross in the name are also left
>> >> unchanged.
>> >
>> > Why is "foreign" insufficient? Why are we expecting to have -cross
>> > versions at all?
>>
>> Because there will be libfoo-arm-cross dummy packages for as long as
>> some reverse depends needs it.
>
>> Again, just for the years of transition to full multiarch.
>
> OK.
>
>> >> Multiarch foreign packages with -cross in the name are changed to
>> >> dummy packages just containing a changelog.
>
> Actually they'll contain no files at all, not even a changelog.
>
>> And I'm not changing anything there. Just describing it.
>
> OK.
>
>> >> Non multiarch foreign
>> >> packages are unpacked to a temporary directory and then files are
>> >> moved around. Include files to /usr/include/arm-linux-gnu,
>> >> libraries to /usr/lib/arm-linux-gnu, binaries thrown away and so
>> >> on the same way dpkg-cross moves files around now (except with
>> >> multiarch dirs now). If it
>> >> exists /usr/share/apt-cross/<package>.data is executed to do extra
>> >> modifications needed.
>> >
>> > I don't see how this fits with the changes I've already got pending.
>>
>> It should be exactly the changes you have pending.
>
> What is /usr/share/apt-cross/package.data about?
It is an executable that can alter the package in any way neccessary
to make it functional as -cross package.
For example the current (non-multiarch) libc6-dev package contains an
GNU ld script (/usr/lib/libc.so) containing
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )
The libc6-dev-cross package now needs that changed to
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )
That change would be made in libc6-dev.data and become obsolete when
eglibc is multiarchified.
>> > I think you're putting far too much weight onto apt-cross - it
>> > really, really cannot take it and fixing it is simply not
>> > reasonable. Forget apt-cross, for multiarch purposes, it simply
>> > isn't usable, it doesn't exist. dpkg-cross will merely be a set of
>> > config files in dpkg and -cross packages should not exist, apart
>> > from those few empty dummy ones from the current pending changes in
>> > dpkg-cross and apt-cross.
>>
>> Sorry, way too late. I've been using this with apt for nearly 2 years
>> now to run 32bit binaries on amd64 and I don't expect it to be fully
>> obsolete before squeeze+2 if then. The original amd64+i386 specific
>> implementation has over 50 users and worked just fine for multiarch
>> purposes. And I'm 95% done adapting it to work with any arch and for
>> cross-compile purposes and it would be a shame not to finish it now.
>>
>> Note that the only common thing with apt-cross is the name, as it
>> fullfills its purpose as well as do much more. I agree that the
>> current apt-cross is unusable and I didn't try to fix it. I just want
>> to replace it with something that actually works well.
>
> It is really confusing to talk of apt-cross when you're thinking of a
> new script and I'm thinking of the current one. apt-multiarch ?
>
> apt-cross can only survive as long as the last reverse dependency is
> not multiarched. It does need to be replaced but I'm not sure we need
> another version, rather I think we need to fix apt/aptitude. I'm not
> clear where your replacement fits in.
It just replaces apt-cross until the multiarch transition is
finished. I expect that to still be a few years so we need a working
version.
>> I don't want to create work for you. I will maintain and use this
>> myself and if you don't like it you can keep the old dpkg-cross and
>> ignore it. And while any -cross package is inferior and "deprecated"
>> it still is the only usable way for now and the near future.
>
> Yes, -cross packages will hang around but I just want to be sure that
> they aren't going to hang around forever. :-)
>
>> The workload of apt-cross is also as reduced as I think it can be as
>> long as there is dpkg-cross. All it does is run the index files
>> through the renaming process dpkg-cross does to create the
>> libfoo-arm-cross packages. As long as there is a dpkg-cross being used
>> that does that renaming of package names apt-cross has to do the same
>> to be able to upgrade cross packages.
>
> ... if only the life of apt-cross was that simple . . .
s/of apt-cross//
MfG
Goswin
Reply to: