Re: next generation apt/dpkg-cross
Neil Williams <codehelp@debian.org> writes:
> On Tue, 19 Jan 2010 20:57:19 +0100
> Goswin von Brederlow <goswin-v-b@web.de> wrote:
>
>> I've been working on a next generation apt/dpkg-cross that will
>> support apt, aptitude, synaptic (anything libapt based) and dpkg
>> directly.
>
> Why? There is no future for apt-cross or dpkg-cross - there should be
> no next generation. We get through the multiarch transition and both
> dpkg-cross and apt-cross disappear for GOOD.
Why? Because dpkg-cross didn't do enough for running 32bit binaries on
amd64 and apt-cross was, well, apt-cross. :) The multiarch transition
will be a lengthy process so there is still a few years use left in
this.
> Has this been done alongside the existing changes already described for
> the multiarch transition?
As far as I saw them mentioned but I will check with the latest
dpkg-cross upload.
> Can you make the patches available somewhere?
Will take me a while because I need to build apt from the multiarch
branch and recompile aptitude and synaptic against that and I'm
probably bussy till Saturday at least.
>> Interface:
>> ==========
>>
>> - dpkg-cross just like you are used to.
>
> Only until enough packages have been made multiarch-compliant. In the
> meantime, dpkg-cross merely needs to make the empty, dummy, packages
> already described.
>
>> - Wrappers in /usr/lib/apt-cross/. Add that dir to your PATH or link
>> them from /usr/local/bin and you can install foreign packages
>> transparently with just dpkg. This has some sideeffects so maybe don't
>> use this.
>
> I can't see that as a good idea but I also don't see why it is
> considered necessary. /usr/lib/apt-cross/ is the wrong name. It has to
> be a name based on an arch triplet, but we're already using those.
Sorry, I ment /usr/share/apt-cross/. Nothing arch specific in there.
This is just so you can simply say "dpkg -i foo.deb" and it will
install even if the package has a foreign architecture. Saves you from
typing dpkg-cross every time. It used to also contain apt-get,
apt-cache, aptitude, synaptic wrappers but they've become
obsolete. With them gone this is far less important. But if someone
can't get used to having to type "dpkg-cross -i ..." then they can
still use it.
>> - apt-get, aptitude, synaptic (anything libapt based) are
>> automatically setup to support -cross packages (just needs
>> configuration to know which archs).
>
> This is the main area of work, but not for -cross packages, for true
> multiarch packages.
I find those verry important to upgrade. Having to download all debs
and dpkg-cross -i them would be horrible.
>> Using those you can manage your system just like you are use to. The
>> following will all work fine including for -cross packages:
>>
>> apt-get update
>> apt-get upgrade
>> apt-get dist-upgrade
>> apt-get install foo libbar-arm-cross
>> dpkg[-cross] -i libbaz_1.2-3.arm.deb
>
> dpkg has to do that, not dpkg-cross.
>
> Why are we still looking at -cross packages?
Because it will be years before no more -cross packages are needed.
>> apt-get build-dep bash-arm-cross
>>
>> This will install all the build-depends of bash but install the
>> libraries as arm-cross instead/on top of the native libs so you can
>> cross compile bash. This is still experimental but it looks like it
>> works well so far.
>
> If multiarch works properly, this isn't necessary. What we'd need is
BIG IF. Or rather WHEN.
> just:
>
> $ sudo apt-get -o Apt::Architecture=armel build-dep foo
>
> (if someone can make that into '-a armel' it would be a lot easier.)
I considered that too. It is on my ToDo to support that but the name
mangling was far simpler to do and without changes to apt. I expect
that the multiarch support in apt will come to implement that in some
way on its own. Maybe as "apt-get build-dep foo:armel" to give better
alternative. I would expect "apt-get -o Apt::Architecture=armel ..."
to cause apt-get to consider armel the native architecture. That would
likely try to install make:arm instead of make:<native> then for
example.
>> This would install a 32bit bzip2 on amd64 including any needed
>> libraries. It requires a non-default configuration saying that it is
>> OK to have foreign binaries though and is only usefull for e.g. i386
>> binaries on amd64 where they can be executed. But you could use the
>> same with qemu to run them. Usefull to test a bug that only happens on
>> one architecture. But, let me repeat it, this will require extra
>> configuration to prevent accidentally replacing native binaries with
>> foreign ones.
>
> Why? If apt or aptitude understands multiarch properly, the dependency
> resolution will be able to work out which are same and foreign etc.
> and proceed as appropriate.
Because they do not yet and packages are not yet converted. When they
do I imagine
apt-get install bzip2:i386
will work. But so far that isn't possible with plain apt.
My use case for this is
apt-get install rar=2:3.9.1-1~i386.cross.1
while amd64 only has rar 1:3.8.0-2 (actualy I have rar pinned but the
effect is the same).
>> Last there is also a tool (convert-cross) to convert deb packages
>> instead of installing them. The same mechanisms used during install
>> (see implementation below) are used to convert libfoo_1.2-3_arm.deb
>> into libfoo-arm-cross_1.2-3_all.deb (or a dummy
>> libfoo-arm-cross_1.2-3_arm.deb if multiarch). Those packages can then
>> be put into a repository and installed with plain apt and dpkg. Those
>> that don't like on-the-fly conversion can use that to get identical
>> results.
>
> Please can we get rid of all -cross ideas? The whole point of multiarch
> is that all -cross stuff goes away.
I would love nothing more but I don't see it done yet.
> Simon did have a -cross repo - I'm not sure if that is still
> operational or whether the problems were fixed.
>
>> That is usualy it. That automatically enables all apt sources for the
>> archs listed. But you can do more in /etc/apt/sources.list[.d]:
>
> apt needs to be taught how to keep multiple lists safely - changing
> arch causes apt to delete all lists for the other arch, changing back
> removes them again. Some of the problems with apt-cross are because
> apt-cross tries to stop apt doing this and then the apt bindings get
> confused.
That is included in the patch already. It adds the arch to the
filename of index files.
>> /etc/apt-cross/black.list:
>>
>> Filter blacklisting packages from conversion. By default this prevents
>> any binary available natively to be converted.
>
> ? There is support in the MultiArch Spec for such limitations, why a
> separate one?
Is there? https://wiki.ubuntu.com/MultiarchSpec doesn't list any that
I saw.
The black.list is a security precaution preventing the user to shoot
himself in the foot in obvious ways. While it should work to install a
32bit bash on amd64 that is normaly not desired so the default
black.list prevents experiments like that. It also prevents
apt/aptitude/synaptic from going crazy with e.g. type-handling. Having
type-handling from multiple architectures confuses the hell out of apt.
>> It further explicitly
>> excludes dpkg, apt, aptitude, type-handling and a few more from ever
>> being converted.
>
> That won't work. apt provides a SONAME that needs to be available as
> armel when cross-building things like aptitude. apt does need to be
> installed as a multiarch package.
In the long run apt needs to be split into apt and libapt for
that. You can't install an apt:arm on amd64 as that would give you a
non-functioning /usr/bin/apt. And even if it would work it would
suddenly install arm packages natively across the board and refuse to
install amd64.
>> They would really mess up the system if you
>> tried.
>
> Umm, it works OK if the installation locations are done properly. The
> current apt-armel-cross package IS a necessary one.
But that is because you only handle libraries. My use case also
includes handling foreign binaries. If apt-armel-cross is really
needed before apt has a multiarchified libapt I have to change
something to support that. Do you remember why you needed it?
>> This is used only by "apt-get update".
>
> Please drop any mention of apt-cross from such things. I have no
> intention of maintaining apt-cross after multiarch - it IS going away
> and dpkg-cross will migrate into dpkg (what is left of dpkg-cross
> after multiarch).
>
>> /etc/apt-cross/rename.list:
>
> Change the path name, please.
I can call it apt-cross-ng if you like to make clear it is a different
package than apt-cross.
>> List of patterns that decide what packages are libraries and what are
>> binaries. Also handles packages available natively under lib32* or
>> lib64* name (or other) where the native one will be prefered. E.g. on
>> amd64 libc6-i386 is to be used instead of libc6-i386-cross. This is
>> the only external and global information used.
>
> Why can this not be calculated from the Multi-Arch fields?
Because when you install libfoo_1.2-3_arm.deb, which Depends: libbar,
you do not have access to the Multi-Arch field of libbar. And libbar
might not be multiarchified yet, which you also don't know.
>> The implementation of all this is a bit tricky and scary at first but
>> keep an open mind. The main idea is to change as little as possible
>> while getting all the features needed.
>
> So is this still part of the transition or are you expecting any of
> this to be necessary AFTER multiarch?
Still part of the transition. As you see below multiarch packages are
passed through unaltered. Only -cross packages are afected.
>> Dpkg:
>>
>> The wrapper around dpkg just adds /usr/lib/apt-cross/ to PATH so
>> /usr/lib/apt-cross/dpkg-deb is used to unpack debs.
>
> Please drop that mention of apt-cross - it is misleading.
>
> You cannot unpack into /usr/ anything - it has to /tmp or /var/
Who said anything about unpacking to /usr?
>> With -e, --control the control.tar.gz is first unpacked and then the
>> control file is processed. The package name and package relationships
>> are changed just like for apt. So "dpkg -e libfoo_1.2-3_arm.deb" will
>> result in a DEBIAN/control file for libfoo-arm-cross that depends on
>> libbar-arm-cross and so on. The renaming happens like the old
>> dpkg-cross did with the changes recently mentioned about multiarch
>> packages. The big change is that this happens when the control.tar.gz
>> is unpacked by dpkg instead as seperate process between downloading
>> and installing. If it exists /usr/share/apt-cross/<package>.control is
>> executed to do extra modifications needed.
>
> Why? If this is during the transition, we just use dpkg-cross as now.
> If it's after the transition, there should be no need for any -cross
> package.
Why do this on the fly as opposed to the existing way of creating a
temporary deb and then install that? I don't want to gunzip
data.tar.gz, mangle it, gzip and then have dpkg gunzip again. Doing
this on the fly saves a gzip and gunzip run.
>> With --fsys-tarfile first the control.tar.gz file is unpacked and
>> checked if this is a native package, foreign package and if it is
>> multiarch or not. Native packages are left unchanged. Multiarch
>> foreign packages without -cross in the name are also left unchanged.
>
> Why is "foreign" insufficient? Why are we expecting to have -cross
> versions at all?
Because there will be libfoo-arm-cross dummy packages for as long as
some reverse depends needs it. When you run
apt-get install libfoo-arm-cross
it will potentially install
- libfoo-arm-cross_1.2-3_arm.deb (originally libfoo_1.2-3_arm.deb)
- libbar-arm-cross_1.2-3_arm.deb (originally libbar_1.2-3_arm.deb) dummy
- libbar_1.2-3_arm.deb (originally libbar_1.2-3_arm.deb) multiarch
Again, just for the years of transition to full multiarch.
>> Multiarch foreign packages with -cross in the name are changed to
>> dummy packages just containing a changelog.
>
> That's already being done in the changes already described on this list
> about the Multiarch transition. However, *all* multiarch packages are
> handled that way.
And I'm not changing anything there. Just describing it.
>> Non multiarch foreign
>> packages are unpacked to a temporary directory and then files are
>> moved around. Include files to /usr/include/arm-linux-gnu, libraries
>> to /usr/lib/arm-linux-gnu, binaries thrown away and so on the same way
>> dpkg-cross moves files around now (except with multiarch dirs now).
>> If it exists /usr/share/apt-cross/<package>.data is executed to do
>> extra modifications needed.
>
> I don't see how this fits with the changes I've already got pending.
It should be exactly the changes you have pending.
>> Not all packages can be transformed fully automatically. Some need
>> some extra handholding. For those cases the <package>.control and
>> <package>.data hooks are available to do whatever custom
>> transformation they need. For example libgtk2.0-0 needs this for i386
>> debs on amd64 (old hook from ia32-libs):
>>
>> # Fix path for plugins
>> sed -i 's,/usr/lib/,/usr/lib32/,'
>> usr/lib32/gtk-2.0/2.10.0/immodule-files.d/libgtk2.0-0.immodules sed
>> -i 's,/usr/lib/,/usr/lib32/,'
>> usr/lib32/gtk-2.0/2.10.0/loader-files.d/libgtk2.0-0.loaders
>
> For gtk to be multiarch-compliant, that needs to be done in the package.
But hasn't yet. Otherwise it would be
/usr/lib/i486-linux-gnu/gtk-2.0/2.10.0/immodule-files.d/libgtk2.0-0.immodules
and the multiarch dir used in that file. And multiarch packages don't
get converted anyway, that is the point of multiarch. The hook is just
till then. NOW gtk supports only /usr/lib32/... so that is what must
be used.
>> So that's it. Let the flame fest begin.
>
> I think you're putting far too much weight onto apt-cross - it really,
> really cannot take it and fixing it is simply not reasonable. Forget
> apt-cross, for multiarch purposes, it simply isn't usable, it doesn't
> exist. dpkg-cross will merely be a set of config files in dpkg and
> -cross packages should not exist, apart from those few empty dummy ones
> from the current pending changes in dpkg-cross and apt-cross.
Sorry, way too late. I've been using this with apt for nearly 2 years
now to run 32bit binaries on amd64 and I don't expect it to be fully
obsolete before squeeze+2 if then. The original amd64+i386 specific
implementation has over 50 users and worked just fine for multiarch
purposes. And I'm 95% done adapting it to work with any arch and for
cross-compile purposes and it would be a shame not to finish it now.
Note that the only common thing with apt-cross is the name, as it
fullfills its purpose as well as do much more. I agree that the
current apt-cross is unusable and I didn't try to fix it. I just want
to replace it with something that actually works well.
> Please reduce the expected workload of dpkg-cross and apt-cross and
> think of any -cross package as automatically inferior and "deprecated".
I don't want to create work for you. I will maintain and use this
myself and if you don't like it you can keep the old dpkg-cross and
ignore it. And while any -cross package is inferior and "deprecated"
it still is the only usable way for now and the near future.
The workload of apt-cross is also as reduced as I think it can be as
long as there is dpkg-cross. All it does is run the index files
through the renaming process dpkg-cross does to create the
libfoo-arm-cross packages. As long as there is a dpkg-cross being used
that does that renaming of package names apt-cross has to do the same
to be able to upgrade cross packages.
MfG
Goswin
Reply to: