Re: Best practice for cleaning autotools-generated files?
* Neil Williams <codehelp@debian.org> schrieb:
> > Which ones exactly (besides the usual autotools) ?
>
> bison, flex and a few others.
Standard tools available on virtually any system that likes to
build anything ...
To make sure they're installed, use build-time dependencies. Done.
> Also that certain stages of the transition from VCS checkout to
> tarball are not actually scripted anywhere, just described in a file.
Bad practise, unnecessarily burns manpower and leaves more chances
for errors.
Actually, not quite that unsual out there, but I'm fixing this for
packages I need in OSS-QM.
> > I don't want to repeat the last decades if SCM theory and common
> > practies here, but one of the major goals of an SCM infrastructure
> > is that you always get can grab the source of some product in some
> > specific version and run a fully rebuild, anytime. _Reliable_
> > reproducability. Tarballs that are not fully-automatically produced
> > from the appropriate SCM tag (and can be reproduced anytime) simply
> > dont meet that requirement.
>
> Disagree. A tarball is easier to get hold of than some weird tag in an
> unfamiliar VCS.
But tarballs dont exactly scale well when you need to maintain many
versions and downstream branches.
> Common format, common methods to download and obtain it, simple URL's
Yes, simple as an doSomething() function ;-o
Practical for the end-user, but lacks a lot of useful information
(eg. history).
> > Maybe that all worked for you over many years. Fine.
>
> It has indeed - about 10yrs at last count.
Well, for me, in the last 10yrs that way exactly did not work.
My current models and workflows are product of an 10yrs evolutions,
lots of trials and errors.
> Personally, I dislike sysroot because it doesn't actually make my
> cross-building work any easier.
Actually, I've switched to sysroot several years ago, because it
made it a lot easier and more reliable. Of course, many packages
dont work with it yet (mostly due stupid buildfiles). So I fix it,
and I fix it once and for all. Takes some bit more time in the
first place, but gets you rid of a whole lot of trouble later.
> > Guess what, I wouldn't work with upstreams that don't provide
> > reliable release tags in the VCS (an VCS that I can import to git).
> > Actually, I sometimes do, but it adds a lot of extra burden.
>
> Fine. That means that any difference on that issue is personal
> preference and has no place in any "best practice" guide or
> specification.
Heavily depends on the scope of that guide.
If, for example, Debian would add that point to its packaging
guidelines, somebody would have to fill the gap. hmm. oss-qm ? ;-)
> > My buildsystem (called Briegel), doesn't even bother with things
> > like tarballs or patches (still supported, but unused for years now).
> > It simply grabs the source tree from git by a normalized ref (eg. tag).
> > Normalized means: there's a simple rule to transform a vector of
> > package name and (also normalized) version number into a ref name.
> > Period.
>
> Then your build system cannot cope with any of my packages.
It doesn't need to. If I ever needed one of your packages, I'd
cleanup and maintain it the OSS-QM project. Briegel then would
just pull from there. A clear separation of roles.
> > Any changes needed in the source tree are done through SCM.
> > (the processes behind may vary between individual packages).
>
> No, those should be done via patches, managed by dpkg-source format 3
> or directly incorporated into the upstream tarball (which is how I
> prefer to do it).
Maybe this approach works for Debian, but for my projects it doesn't
scale well. It essentially means manually maintaining text-based
patches (especially when it comes to rebasing, moving to other
branches, etc, etc). I dont wanna do such manual things anymore
since the SCM can do that automatically.
> Depending on the purpose of the patches, the patch can go upstream or
> into the Debian package.
It's trivial to export patches from git automatically.
I also thought about adding additional changeset headers, so an
robot could export and mail the patches automatically.
> > True. It's a distro-agnostic issue. It's about how to be a good
> > upstream to make distro's lifes easier and extends into the whole
> > QM topics.
>
> And that includes the ability to compare the checksums of tarballs
> included in the distributed sources of an embedded cross-built
> distribution to check that you really do have the same version as other
> distributions.
Why care about tarball checksums if you have cryptographically
secured tags and histories ?
> I never have seen any point in that. When preparing patches for an
> embedded distribution, it is pointless to modify the sources
> themselves. Provide patches in your own build system (i.e. debian/) and
> let people have the assurance that your sources match everyone else's
> by keeping the original tarball unchanged and putting your changes into
> either a .diff.gz or a .debian.tar.gz.
You'll have to patch anyways, no matter if you manually create an diff
and let the build system apply it or let it fetch the finished source
tree directly from some VCS. As modern VCS'es like git already provide
the necessary operations for easily doing the whole SCM there, I don't
see any point in having the whole patch management complexity in the
build system and package maintenance side.
In the end, both approaches want the same: a clean source tree which
can be built automatically.
> > Obviously, you almost never have to change the input files.
> > Or need to fix/tailor autotools. For mainline distros this might
> > work well, but in the embedded world you're easily out of luck
> > with this approach.
>
> Hah! Nonsense. If so, it's a problem with your embedded build system.
> I've done several thousand embedded builds of packages across all of
> Debian and the result?
>
> 99.99% of those packages ONLY needed changes in the debian/ directory
> to be able to cross-build and mostly any changes we needed to make to
> select different ./configure options were all done via debian/rules.
Aha. How do you then, for example, repair broken autoconf inputs ?
(things like AC_TRY_RUN, AC_CHECK_FILE, ...)
And for not mixing up things: are you sure that you just didn't
encounter certain upstream bugs since they were already fixed by
standard Debian ?
> > And it *IS* necessary, as soon as you have to change some of the
> > input files. In my daily projects, this happens on the majority
> > of the packages.
>
> Nope. Not true. Not even slightly. I have several years of embedded
> experience with some 800 packages in Debian which prove the opposite.
Now you really make me curious: how do you cope with things like
AC_TRY_RUN or AC_CHECK_FILE without touching the source ?
> > a) everybody does the required fixups and workarounds all on its own
> > b) collaborate and provide a stabelized midstream together and so
> > save a lot of human workforce.
>
> ... or just use the tarballs which exist in Debian where others have
> already done all that work for us.
We're running in circles. I was talking about situations where upstream
(being tarball or vcs tag) is buggy and somebody has to fix these bugs.
Maybe in your case, sitting ontop of standard Debian, other Debian folks
already have fixed most of them, but in my case, having nothing to do with
Debian at this front, that doesnt help in any way (okay, I can manually
take patches from Debian, but that's a completely different issue)
> > Why dont you just tag your releases properly ?
>
> Why should I? Who defines "properly"?
A tag is just a pointer (or name) to a specific version of the tree.
(the whole tree, not just individual files)
> > And did you ever consider that fact that certain downstreams would
> > like to use their VCS to rebase their changes to newer releases ?
>
> I doubt you can actually assert that this is a fact vs mere opinion
> with respect to my actual upstream packages.
Well, as soon as you maintain patches/changesets ontop some upstream's
package version (so, being a downstream) and going to port them to
subsequent versions, you're essentially doing a rebase operation,
either manually or by help of a VCS. And modern VCS'es like git can
do that quite automatically (yes, you'll sometimes still have to resolve
some conflicts manually, but quite few compared to applying text-based
patches). And the number of people using their VCS for rebasing is growing.
> > You're aware of the fact that you're making life harder for downstreams ?
>
> I doubt it. If you want to make that assertion, prove it in relation to
> my upstream packages. You are making universal claims which are
> manifestly individual preferences.
I don't know your packages in particular, so I don't even know if I
would be required to some downstream works on them. But for the majority
of packages I have to deal with, I need to maintain my own downstream
branches (yes, for some I could work around by moving all the complexity
to the distro build system, but I dont wanna open that pandorra box
anymore, so I'll fix the source). And for those packages, having the
source in a modern VCS (eg. git) is a big big bonus.
> > > Indeed, in a few upstream projects, the only files which ever get
> > > tagged are the debian/ files. ;-)
> >
> > You tag individual files ?!
>
> Only occasionally. For experimental purposes only. Generally don't find
> tags useful in upstreams.
Ah, okay, now it starts clearing up to me ...
You're using a file-based VCS ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
Reply to: