Re: RFC: Patches for supporting cross-building
Hello Helmut,
Sorry for my delayed response.
On 2018/07/03 6:30, Helmut Grohne wrote:
Hi Kazuhiro,
On Mon, Jul 02, 2018 at 07:35:47PM +0900, Kazuhiro Hayashi wrote:
Thanks, helpful report for us to know what's going on.
Regarding the build environment, as the first step,
I guess it's better to have the similar build environment as yours locally,
properly recognize the issues you are facing in your reply to Baurzhan,
then consider the next action.
I think there are two standard build environments. One is sbuild and the
other is pbuilder. Once either is set up for native building, cross
building is (in theory) a single flag away. For sbuild, it is --host and
for pbuilder --host-arch.
Thanks, we also met the two candidates then selected sbuilder.
The reason is that sbuild is official build infrastructure for Debian packages
(of course, even though it's for native building); we should use the same tools
and environment as Debian official package building to share the issues and
to avoid getting troubles unrelated to the official package building.
After that, the devil is in the detail. While pbuilder works around
#815172, sbuild does not, so most packages will need
--add-depends='libc-dev, libstdc++-dev' for sbuild. Then sbuild sets up
a CONFIG_SITE with cross-config (formerly dpkg-cross), but pbuilder
doesn't. It's not exactly clear whether that's good or bad. cross-config
lacks many assignments and at the same time it ships a fair number of
wrong ones. Both add "nocheck" to DEB_BUILD_OPTIONS, but only pbuilder
adds "nocheck" to DEB_BUILD_PROFILES. Neither adds the "cross" profile,
so you likely want --profiles=nocheck,cross for both. Both default to a
full build, but building Architecture: all packages doesn't make much
sense. You likely want --no-arch-all or --binary-arch respectively.
Personally, I use sbuild (because it supported cross building earlier).
However I explicitly disable the CONFIG_SITE stuff, because I consider
it too broken.
Thanks much for telling the detailed states of the two builders.
Regarding the current way, we also set CONFIG_SITE for dpkg-buildpackage.
https://github.com/meta-debian/debian-cross-patches/blob/master/HOWTO.md
At this time, I'm planning to keep using sbuild for cross-building packages,
so we need to consider which is the better approach
by trying cross-building multiple packages.
(snipped)
Actually, other several developers are mainly created the patches.
When they post the patches to debian bug report,
"Usertags=rebootstrap" is still enough to easily share our efforts.
Also, I think "User=debian-cross@lists.debian.org" would make sense
because it makes easier for other developer who are interested in cross-building
to find the activities in this mailing list.
If we switch the user to debian-cross@lists.debian.org, we should also
switch tags to gain more precision. The usertag user however does not
connect the mailing list in any way. The list will not receive any
traffic due to such tagging. The user is more like an arbitrary
identifier.
If switching to "user debian-cross@lists.debian.org", I propose using
the following tags:
* "ftcbfs" for bugs about cross building a package where Build-Depends
are cross-installable, but the build fails.
* "cross-satisfiability" for bugs about packages that fail to satisfy
their cross Build-Depends or that cause other packages to fail
dependency satisfaction. For instance, when a foo needs to be marked
Multi-Arch: foreign to be able to satisfy src:bar's cross
Build-Depends, foo should receive a bug report with this tag.
I got it, these tags are helpful to categorize issues.
In my opinion, Ccing should be avoided because it might become harder
for some newcomers like me to find key discussions about cross-building
which are not for specific packages.
Yeah, that's a trade-off. I kinda concur as I've rarely Cced the list
for my own submissions.
OK. By default, I don't Cc to this list except cases in which
the issues include some difficulties or important things and
should be shared with this list.
Instead, it would be great if there is a small summary somewhere
about such preferred labels (like User and Usertags) in patch submissions.
So please use one of:
* user debian-cross@lists.debian.org usertags ftcbfs
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftcbfs;users=debian-cross@lists.debian.org
0 bugs
* user debian-cross@lists.debian.org usertags cross-satisfiability
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=cross-satisfiability;users=debian-cross@lists.debian.org
79 bugs
* user helmutg@debian.org usertags rebootstrap (for general
cross/bootstrap bugs)
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=rebootstrap;users=helmutg@debian.org
1654 bugs
But please do not use any other tagging schemes involving these users
without prior discussion.
Thank you for the summarizing, we'd like to follow the schemes from the next submission.
Some topics seem to be challenging but also important for my cases.
Many packages in http://subdivi.de/~helmut/cross_report_20180614.html
are unsatisfiable because of perl-base, is it related to the first topic?
That's a good question and the answer is "no". perl-base happens to be
"Essential: yes", so it always is installed for the build architecture.
When some package (transitively) Build-Depends: perl-base, it asks for
the host architecture perl. A conflict ensues. The presentation in my
report says "missing", because I discarded host architecture essential
packages from the package lists. It turns out that rather a lot of
things include transitive dependencies on perl (which happens to depend
on perl-base). In most cases, where you see such a conflict, we've got a
problem with dependencies. Requesting the host architecture perl is
usually not what you want, because you cannot run host architecture
programs, so fulfilling the request would be fairly useless: Running
perl would just result in an "Exec format error".
I know that Build-Depends actually has at least two kinds of dependency;
1. host architecture packages (e.g. development libraries to be linked) and
2. build architecture packages (e.g. commands or tools for support building).
"Build-Depends on perl-base" in this case is 2, I guess.
My question here is why the host architecture essential packages are discarded.
What kind of package lists are you meaning the above?
It might seem like
perl should be marked "Multi-Arch: foreign", but that happens to be
wrong as you can integrate with perl in architecture-dependent ways via
perl extensions. Thus perl is marked "Multi-Arch: allowed" and shifts
the responsibility to decide which perl you need to the consumer. It
turns out that many consumers just say "perl" even when they need
"perl:any".
In essence, when you see "missing perl-base", some dependency is wrong.
The difficult question tends to be "which one?".
So, the thing we need to do for fixing "missing perl-base" issues is
to find out perl packages, which need to have ":any" but actually not,
from Build-Depends of each package one by one?
I'd like to know an example of this kind of fixes if already exists.
(snipped)
3. Changing the base system; e.g. init
busybox is still usable for many targets which provide
basic hardware controls and quite simple application only.
e.g. meta-debian ([4] in my last post) provides busybox based tiny RFS as well as systemd.
I know that such environment is no longer Debian, but
it still keep a lot of benefits from Debian 'source code' (e.g. security fixes).
This one will be quite hard with Debian. The major reason is not that
replacing init is hard. The reason is that packages need not declare
dependencies on essential packages. So whenever you remove any kind of
functionality from an essential package, you might break some other
package.
Yes, actually some programs (commands, daemons, etc.) were broken
when we simply replaced essential packages by busybox,
because of missing packages, differences in usage (e.g. available options), etc.
Going this route means trading a lot of advantages you'd get from
Debian. Note that init is no longer essential, but init-system-helpers
now is.
My another interest related to this topic is that Debian installer (udeb) is also
using busybox based system and still keep maintained as 'Debian'.
I think there still be several possibilities to make use of such
existing efforts in embedded world, but I don't have concrete idea
to make good relation between the two different worlds.
I don't have any experience with reusing the installer like that. I do
caution though that adding udebs is a pretty manual process. The udeb
coverage is quite limited and adding your own udebs again means trading
a lot of advantages by diverging from Debian. Nonetheless I welcome you
reporting on this idea.
udeb should not be mixed with deb packages, so
I don't have any expectation in mixing them.
Also I know that udebs' coverage is limited, but
it's sometime enough to implement the tiny system.
The problem is resources for developing and testing udeb 'binary' packages
are completely splitted from efforts for deb packages.
('source' packages are still shared)
If Debian installer could consist of essential deb packages instead of udeb
and their footprint become less than now, there might be benefits for
the both of Debian community and product developers.
Something about this topic would be discussed in DebConf18.
4. Compiling packages with CPU specific optimizations to improve performance
That's something https://wiki.debian.org/HelmutGrohne/rebootstrap is
supposed to enable. The idea behind rebootstrap is that it enables you
to build a base system (and potentially more) from source. In doing so,
you can add your own architecture to the dpkg database or tweak the
compiler defaults to arrive add a customized architecture.
I caution though that the shell script approach is quite fragile and
difficult to maintain. It becomes easier as more packages adopt pending
patches, but for creating OS images, a different approach (still using
the same principles and tools) likely is necessary. Until we get there,
forking rebootstrap for your own purposes (e.g. adding your extra
packages) likely is feasible.
Yeah, minimizing the essential set is the most important and difficult issue.
Also, other non essential packages (service daemon, utils, etc.) are usually appended
into RFS then their big dependencies sometimes affect to the final footprint.
Well, there is one other approach to reducing installation sets. I've
been pondering this for a while, but I couldn't figure out good and
clear semantics yet.
A number of packages depend on other packages solely for the purpose for
configuring, upgrading or removing. For instance, a dependency on
adduser is rarely used outside maintainer scripts. Once you have your
final OS image, you likely don't need adduser anymore. It seems that
most dependencies can be classified into "real runtime dependencies" or
"configuration or installation dependencies". Similar to how we
augmented Build-Depends to Build-Depends-Indep and Build-Depends-Arch,
it might be possible to split Depends into e.g. "Runtime-Depends" and
"Maintscript-Depends". For OS images, we may be able to use DPKG_ROOT
and --force-script-chrootless and thus install those
"Maintscript-Depends" outside the OS tree being installed effectively
avoiding them in our OS image.
Thanks for your idea.
There might be other packages which can be categorized into
"configuration or installation dependencies".
Removing such dependent packages from the final image makes sense
if the system doesn't need to install/update any packages in run-time.
Kind regards,
Kazu
Reply to: