Re: ARMv4-support in armel/squeeze?
On Tue, Dec 21, 2010 at 12:36 AM, Wookey <wookey@wookware.org> wrote:
hi wookey, thanks for responding here. just a note [at the end of
writing what's below] - it's a bit long, goes over some of the things
i wrote previously, but clarifies in the context of what _you_ wrote,
which is always a good thing i've found. patience to all readers
greatly appreciated.
>> so i'm curious: is there any particular reason why, instead of
>> turning an existing tool which does 95% of the work with 10% extra
>> effort to create the tool, a complete from-scratch
>> debootstrap-cross-compiler was started instead?
>
> Essentially because I think it'll work better if it's done with Debian
> infrastructure and tools ...
... on the basis that this is what's best known, and it's often
quicker to utilise that and work from there than it is to incorporate
/ learn someone else's tool, regardless of how good it is.
... yep, i know this one veeery well :)
> ... than if it's done with OE tools. The OE tools
> depend on the OE packagaing ('recipies') and that's not the same as
> the Debian packaging.
NO - they don't! [depend on the OE packaging]. please do re-read
what i wrote. ok, stopstopstop, me, let's try again :)
how do i put this another way... i know, how about this: "ok, to give
you an analogy, imagine if openembedded were thinking of using buildd
instead of bitbake. what effectively you've said is that if you use
buildd and dpkg-buildpackage the OE team *must* utilise the exact same
debian/rules files that are specific for debian packaging...
because... welll... because!" and it's the same thing here: just
because you utilise bitbake DOES NOT mean that you need to utilise the
same recipes that the OE team have spent years accumulating in the
same way that the debian team have spent years accumulating
debian/rules+debian/control files.
what i recommended was to *throw away* the recipes, utilise "bitbake
the tool" and to write virtually from-scratch an entire new set of
classes, tasks and recipes [openembedded's infrastructure is divided
up along those lines - it should be evident what they do, from the
names].
"bitbake the tool" has nothing per-se to do with the OE recipes.
it's a bit like buildd (without the last d :) in that regard
now, it just so happens that *in amongst* some of the recipes, there
are also things that the OE team call "classes" and these contain some
absolutely incredibly valuable stuff (such as dual-libtool installers
and a pkg-config replacement which end up munging the stuff that goes
through them) BUT, it may be the case that yes, in the emdebian
packages and the dpkg-cross-tools much of what's needed has *already*
been taken care of, so it might turn out that there's absolutely
nothing in the existing OE recipes that's needed _at all_! [in which
case, that still leaves bitbake itself as an absolutely stonking basis
for any automated multi-level task dependency situation, which this
most definitely is]
so, the general premise of the debian-OE hybrid approach i'm
advocating is to create from-scratch entirely new entirely independent
recipes that have, at their core, a shell-script call to "cd
${PACKAGENAME}; dpkg-buildpackage -T${TASKNAME}", and to *learn* from
the remaining openembedded recipes. NOT "take them wholesale and
throw away all of debian expertise". that would just be f*****g
stupid [hint: much appreciated if people didn't assume i'm stupid, and
to please take into account that i don't know "the rules" regarding
normal social interaction].
> You could use OE receipies to patch Debian packages in order to make
> them cross-build and generate staging packages and generally be
> buildable using bitbake.
ok - what i recommended was to auto-generate OE recipies *from* the
debian packages (bypassing bitbake's "read-the-OE-recipes-folder"
system and to actually just create the python BB cache datastructures
DIRECTLY in memory), and then, only in very exceptional cases to
create a "real" OE recipe file which overrides that default behaviour.
this couuuld, for example, be utilised to apply
"patches-to-debian-package-patching-which-the-debian-package-maintainer-doesn't-want-to-accept-right-now-but-are-bloody-essential-to-get-onto-the-other-side-of-the-build-deps-barrier"
sort of thing
and also to apply patches for example - again which the debian
maintainer might not want to accept - which make tiiiny modifications
to the "default" behaviour, for example to replace "Build-Depends:
perl" with the very cross-compiler-specific "Build-Depends:
perl-native" err "Thing" *hand-waving* god that's dreadful english -
apologies.
> And that would also provide a sequencing
> tool. This would probably work reasonably well, and I've even
> considered trying it in the past. It remains an interesting idea and
> if you want to try it for real I'd be interested to know how it goes.
yes. at the sprint. and/or if i get some time and/or money [you have
to bear in mind i haven't been paid _any_ money for approximately 2
months and am now 5 months behind on rent. and have enough cash to
pay for about 2 weeks of food. sometimes i wonder why i stick to free
software principles, i really do].
> However the recipies produced would sit outside the packages
> and thus would immediately start to bitrot,
ahhh, ok: 1) but this is what happens to debian packages _anyway_ -
they bitrot. it's just that we're now envisaging a
maintainer-on-top-of-maintainer (ooer) situation [regardless of what
tool is utilised to do the building, it's worth mentioning] which just
makes my head spin somewhat :)
2) as patches or "overrides" (e.g. the Build-Depends: perl-native to
replace "Build-depends: perl idea) the general lesson learned from
OpenEmbedded Recipes is that it actually takes quiiite a long time for
bitrot to set in: years in some cases.
but yes you're right: the more complex you get, the quicker that
bitrot sets in. this is why it would be good practice to let the
"overrides" be an "interim" step to test out stuff that's awkward to
get to the maintainer, see how it goes, and _then_ on the next
version, well, here's the funny thing: the next version from the
maintainer will *not* be the same as the override version for that
package, and, because it has the patches incorporated already, it will
"just work!"
so this demonstrates that the path should be:
* start with an
auto-generator-which-turns-debian/control-into-internal-bitbake-recipe-datastructures
* provide an "override" mechanism to workaround cases which don't fit
the above (native host dependent packages for cross-compile instead of
native-to-the-target for example)
* utilise the existing "patch" system of bitbake to add extra patches
to the debian/rules (etc.) as required to get cross-compiling working
(which may involve having a completely new target in the debian/rules
which is conditional on 'cross-compile' - but whatever it is, it
DOESN'T go into the actual debian/rules etc. it DOES go into bitbake
as a PATCH to debian/rules etc.)
* submit those exact same patches upstream to the debian package maintainer
* bitch like mad at the maintainer if they give useless reasons to
reject the patches out-of-hand.
* do nothing further, unless the package changes and a cross-compile
fails, in which case you go back to step 2) or 3).
> so next time someone wants
> to do a bootstrap they'd proabbly need to do a lot of updating of
> recipies to current packages. And some of the patches in there would
> be pretty hacky because the packages don't contain the right metadata
> and don't have good mechanisms for making the necessary cross-build
> and bootastrapping changes.
exactly. yes.
....well, i think it's worth mentioning that regardless of whatever
mechanism is chosen (bitbake, home-grown-specially-for-debian,
portage) this issue doesn't *actually* have anything to do with the
mechanism per se.
i.e. _whatever_ the mechanism utilised to ultimately carry out the
"calling-of-dpkg-buildpackage", you _are_ going to have to make
necessary changes [which the package maintainer _is_ going to end up
having to accept, in the end]. the sooner that's generally accepted
[and debian policy mandatorily updated to reflect that] the easier
this will all be.
> I think it's better to work out proper Debian-packaging-friendly
> mechanism (or use existing ones where there is one) to do what is
> needed, and (having shown that this stuff works and is useful)
> enshrine it in policy. This is much more robust in the long term.
hurrah :)
> This is almost all about metadata and packaging, not about special
> build tools.
yyyep, absolutely.
it just so happens that i believe you could get away with about....
*guessing* 400 lines of python code [a hybrid which uses python-apt to
then create bitbake internal data structures] and *guessing* ooo...
100(?) lines of replacement-bitbake-tasks-which-call-dpkg-buildpackage
(as a bitbake class, which all of the tasks inherit. hurrah!) to get
this all kick-started, *instead* of having to do a complete
"from-scratch" total new wheel-shaped special build-tool. and [appx]
600 lines of code is exactly the kind of high bang-per-buck ratio to
get a big job done that i absolutely frickin love.
> Now - I could be wrong, and perhaps this can't be made to work well
> without all the magic 'pre-configure' and 'post-staging' sort of
> tweaking that goes on in OE. But I haven't yet seen reasons to conclude
> that.
well, as a "first cut" i would suggest *not* trying to do
cross-compile at _all_ - and to in fact throw away alll the OE
specialised tasks *entirely* - and instead to try doing a "native"
build (effectively an alternative to buildd) on amd64 or x86.
i believe that it should be pretty damn trivial to do that. you
don't have the issues of circular build dependencies, for a start,
because these are instantly solved by having one of the tasks be "sudo
apt-get install ${DEPENDENCY}" ok it's not quite that simple but it
almost is!
once that's been proven, you then have a good base on which to try to
do a cross-compile, starting _not_ with adding in an equivalent to
"pre-configure" and "post-staging" but instead to add in, to the
"build" task, an environment variable which tests and utilises
"dpkg-cross-XXXX" instead of "dpkg-XXXXX".
and then see how far _that_ gets you.
theeeen, if that works for the majority of cases, the first one that
this whole exercise falls flat on its face, we go "ah. rrrright."
and get out the drawing board :)
> In practice this question will only be determined by trying it.
yah that's a fact :)
>> so - the system you're creating: will it be possible, right at the
>> beginning, to specify the compiler flags and options required?
>
> It doesn't address that issue directly. The way you set the default
> compiler options is to build a new 'flavoured' toolchain with the
> correct defaults. Personally I think that's really ugly, but it seems
> to be the only way that works reliably, and due to the way gcc builds
> itself to generate libgcc, there are real problems with building a
> toolchain that can be given loads of different build options.
mmm yehhhhh openembedded solves this by not only having those
defaults initially (set into the toolchain) but also by then setting
environment variables CFLAGS, CXXFLAGS etc. etc. before running the
configure (or scons, or whatever) and it's "taken care of".
i take it dpkg-buildpackage _can_ have its environment variables
overridden like this?? somewhere?? (ok, dpkg-cross-xxxx _definitely_
you should have this in, somewhere!)
some upstream packages are just f*****d and don't listen to CFLAGS
etc. etc. so in these cases hurrah! bitbake's built-in patch
mechanism comes to the rescue! :) dreadful, i know, but hey.
btw - the whole reaason whyyyy i mention the use of openembedded (to
"learn" from them) is becauuuuse many of the flags/options which are
absolute buggers to get right on a per-machine basis aaare in faaaaact
already enshrined in OE's machine/includes files! so if someone's
gone to all the trouble of doing that, already, then it makes a hell
of a lot of sense to pick up OE's pre-existing recipes for
gcc+binutils (if possible / practical) and to investigate munging them
into debian packaging. as a first "attempt" i'd look, very simply, at
how to drop debian/rules+debian/control "on top of" the existing OE
gcc+binutils recipes...
... BUT, butbutbut it's worthwhile mentioning that THIS IS NOT AN
ESSENTIAL STEP. just as with OE it is perfectly possible to use
preexisting toolchains, so it is also possible to build some
replacement-recipes which, in the first instance, do the same.
and, LATER, add in an
auto-generate-me-a-cross-compile-toolchain-in-a-deb-package step.
> Wrappers can also work (as used in pentium-builder and apt-build).
> But, as Joey says here:
> http://joey.kitenet.net/blog/entry/ten_years_of_free_software_--_part_8_pentium-builder/
> it would be better to fix the underlying packaging issues which
> prevent default options from propogating properly. This would require
> a load of packages fixing and probably some policy adressing it.
... well, the package fixing can be maintained as
patches-on-top-of-patches until such time as the maintainer accepts
them.
_regardless_ of what "special tool" is utilised, to do anything else
- for example to expect the maintainer to just blindly accept
cross-compile-specific patches to his "lovingly maintained debian
package" - is pissing/whistling in a gale.
policy - dunno :)
whew. long. apologies.
l.
Reply to: