Re: next generation apt/dpkg-cross
Neil Williams <codehelp@debian.org> writes:
> On Thu, 21 Jan 2010 13:00:37 +0100
> Goswin von Brederlow <goswin-v-b@web.de> wrote:
>
>> >> > How are you populating the apt cache? Trying to fool apt into
>> >> > thinking that -cross packages exist in a repository is going to
>> >> > break when dependency chains get long.
>> >>
>> > If the package name has changed in that process, the new package name
>> > is unknown to apt.
>>
>> I think now we are getting somewhere. That is exactly where my code
>> differs.
>
> I was coming to that realisation too.
>
>> The Apt::Update::Post hook changes the binary-arm/Packages
>> file apt downloaded so that it does list the -cross packages with the
>> altered package relationship but keeps the Filename pointed to the
>> native arm deb. That way apt does know about the -cross packages, can
>> resolve the dependency chains and downloads the native arm package and
>> renames it to -cross locally (just renames the file). It then calls
>> Dir::Bin::dpkg (which points to the dpkg wrapper) to install
>> libfoo-arm-cross_1.2-3_all.deb (or _arm.deb). The dpkg wrapper handles
>> the actual conversion from arm to -arm-cross.
>
> Quite a different process from dpkg-cross/apt-cross. You're doing the
> renaming *ahead* of the conversion instead of after it because that
> allows you to insert the renamed listing into the apt cache.
>
> Presumably your new apt-hookma is in /usr/sbin/ rather than /usr/bin
> like apt-cross?
No, it just adds /etc/apt/apt.conf.d/00apt-hookma to insert itself
into libapt. The user then uses the normal apt-get, aptitude or
synaptic and -cross support is totaly transparent.
>> I don't provide a option identical apt-cross binary (or any apt-cross
>> binary) so it isn't a drop in replacement. So scripts that call
>> apt-cross will have to be changed. But for users it should be just a
>> matter of forgetting about apt-cross and use only apt-get.
>
> Which means that the new script should have a different name, in a
> package other than apt-cross or dpkg-cross. Then, once emdebian-tools
> has migrated, emdebian-tools can Conflict: with apt-cross even if it
> retains libcache-apt-perl.
Makes sense.
> The new package can conflict with apt-cross but it can't Provide: it -
> although during testing we don't want a Conflict: either.
I have to test how they behave if you have both installed. It might be
that my /etc/apt/apt.conf.d/00apt-hookma sniplet has an adverse affect
on apt-cross. I don't know if apt-cross overrides the apt.conf.d dir.
> Scripts need the following functionality (not necessarily using this
> syntax or these options):
>
> 1. $ sudo apt-foo -a armel build-deps PACKAGE
Could that be 'apt-get build-deps PACKAGE-armel-cross'? That I can
feed into the apt database now. Anything else will need the proper
syntax implemented in apt itself and that can take a while.
> 2. $ apt-cache search [a given once we can use the main system apt]
Yep. Works flawlessly. Too well actualy as apt-cache search for any
library will give you every entry once per configured architecture.
Something that will go away when the -cross names are dropped though.
> 3. $ sudo apt-get -a armel install PACKAGE
Like 1 that is 'apt-get install PACKAGE-arm-cross' for now. When
multiarch support is there and multiarchified packages then the
proposed syntax is 'apt-get install PACKAGE:armel'.
> 3. $ apt-get -o Apt::DownloadOnly=true install PACKAGE
Now that is a tricky one (You know about -d, --download-only, right?).
While this will download packages and rename the files it will not
convert them. So 'apt-get -d install libfoo-armel-cross' will download
libfoo_1.2-3_armel.deb and rename it to
libfoo-armel-cross_1.2-3_all.deb. But it will still be the orginial
arm deb. The conversion only happens when the package is actually
installed.
But 'dpkg-hookma -i' will install them correctly from that or from
their original name so that should be OK.
> 4. ONLY require sudo for installation or update processes, just like
> existing apt - all cache queries must be userland. It would be good if
> [3] could be done without sudo (as it is now with apt-cross) but it's
> not as big an issue as having apt-cache accessible without sudo.
All of the apt-get/apt-cache operations work just like now.
I think to download a bunch of deb packages you might want to use
aptitude though with 'aptitude download libfoo-armel-cross' and then
convert the downloaded package with the convert-hookma tool (or
dpkg-cross, should give the same result). With apt-get it will
download into its cache directory (needs root or overriding the dir)
including any dependencies that aren't yet installed, which gives a
somewhat random selection of debs downloaded.
Later we should probably have a session to talk about usage other than
installing/upgrading like this and see wether there already is a good
way to do it or if some small tool should be added to automate the
job. But let me finish the current "install & upgrade" work first
before you sidetrack me.
> However, porting the scripts to use any version of a replacement for
> those commands will involve large amounts of work - and testing.
>
> The work will be considerably eased when we have a way of installing
> the toolchains without needing to calculate the current version of gcc
> and libc6 etc., i.e. once toolchains are available within Debian
> (zumbi). :-)
>
> What I'd like - for now - is to know if your scripts can do a full
> build-deps install working in a clean chroot (with only a
> cross-toolchain added to a normal debootstrap+build-essential) at least
> as high as the complete build-dependencies for packages which
> build-depend on libgpewidget-dev - say gpe-calendar or gpe-bluetooth
> (to pick a hard one for the dependency list but one that will at least
> build with just dpkg-buildpackage -a without needing emdebuild itself).
>
> How do your scripts handle build-dependency lists where some (like
> automake) need to be native and some (libfoo-dev) need to be cross but
> Build-Depends contains all in one line? (apt-cross/emdebian-tools use a
> debian/xcontrol file in each package with Build-Depends and
> Build-Depends-Tools.)
The same way it handles depends. The rename.list says what is a binary
and what is a library package. Binary packages are not renamed so
automake remains automake. Library packages get renamed so libfoo-dev
becomes libfoo-dev-armel-cross. With the exception of libraries that
also contains tools (as in
/usr/lib/libfoo/stupid-tool-the-build-needs) that should work.
For Build-Depends it might be good idea to install both the native and
cross package though. So
Build-Depends: automake, libfoo-dev (>= 1.2-3)
becomes
Build-Depends: automake, libfoo-dev (>= 1.2-3), libfoo-dev-armel-cross (>= 1.2-3)
That would solve the problem of tools in library packages.
The only critical thing is that any renaming must be able to work
on the Packages.gz file and the DEBIAN/control file inside the
debs in exactly the same way and on Sources.gz on their own. You
can not use information from the debian/xcontrol file unless those are
put into the Sources.gz file in some way. So "apt-get build-dep" might
not give you 100% of what debian/xcontrol later says it needs.
> That's the biggest issue for me - I might have the wrong impression
> here (not for the first time) but your work appears to concentrate on
> runtime requirements and preparing a buildtime environment is a subtly
> different problem.
Yes. I'm coming from the runtime side of things. The last 2 years I've
used this to run 32bit apps on amd64. I'm just now adopting it to
cross compiling because it already did like 95% of the job for it. The
support for apt-get build-dep is completly new and verry much
experimental. If it doesn't work right from the start then I have to
see if it can be fixed or if it is a lost cause. Worst case you do
start a cross compile, take the list of missing Build-Depends and
apt-get install them. I do that often enough with native packages that
I wouldn't notice a difference.
> It *might* be possible to convert the libcache-apt-perl module to get
> data from the main system apt cache - at least as an interim measure.
> (It would have to abandon all attempts to update the main apt cache
> with apt-get update so that scripts can continue to operate without
> sudo but I think I can add a new function that disables the relevant
> components.)
The way apt itself does this is that it checks if the cache is current
and if not it creates a new one in memory. Shouldn't the perl module
be able to do the same?
> I did anticipate that libcache-apt-perl might survive longer than
> apt-cross itself - turns out that work on Grip has provided other ways
> to read data from Packages files which are far, far, easier to handle
> in perl than the actual apt cache that apt builds from the same Packages
> files.
>
> i.e. The problems underlying libcache-apt-perl are simply down to the
> fact that we are trying to understand what C++ does to a collection of
> Packages files using perl bindings on that C++ when it may turn out to
> be easier to read the Packages files into perl directly and skip the
> pkgcache.bin files and the whole C++ mess entirely.
I can imagine that. Perl is really great at parsing Packages files and
not having to mangle data between c++ and perl for every request might
be faster overall too. I leave the perl side to you, I hate it. :)
MfG
Goswin
Reply to: