On Sun, Feb 17, 2008 at 03:07:59PM +0000, Colin Watson wrote: > On Mon, Feb 11, 2008 at 10:53:48AM +0100, Bas Wijnen wrote: > > This is not true if you simply build the whole package from source. > > That is, run autotools during build, remove all generated files, > > including Makefile.in, configure, etc, in the clean target. > > > > For some reason many people seem to think that the whole package must be > > built from source, except for configure and Makefile.in. I still don't > > understand what the idea behind this exception is. Especially when it > > leads to unwanted concequences (unreadable diff.gzs, for example), I > > don't think it is a good idea to hold on to the idea that running > > autotools is not part of building the package. > > It's a well-established principle of the GNU build system that there are > targets that are run by maintainers and there are targets to be run by > people building the package. This division is there to make our lives > easier, and ignoring it, IMHO, is cutting off our nose to spite our > face. (Even the section in the GNU Coding Standards regarding the > maintainer-clean target doesn't go so far as deleting configure.) So according to the GCS, even the maintainer shouldn't need to regenerate configure, or at least not by default. This doesn't quite prove your point, that this is a "maintainter-only" thing. ;-) Anyway, AFAIK the reason that autoconf exists is to allow building programs reliably, even on exotic architectures. They may have really weird shells, or a weird make, or libraries in weird places needing special flags, etc. That is a good thing. I am happy that it is possible to build the programs on those architectures with relatively little trouble. One of the essential things of autoconf there, is that it doesn't require itself to be installed. The generated configure script should be very platform-independent. It is pre-built exactly for that reason: to make sure that the build does not require autoconf to be installed. So far so good. But Debian isn't an exotic system. It's GNU with a Linux kernel. You can't get more standard than that, from autoconf's (thus GNU's) point of view. We don't need this super-platform-independence on our system. We have a proper shell, GNU make, and autoconf and automake can be installed without any trouble. In other words, there is no reason to use the pre-built files. Their reason for existing isn't applicable to Debian. > The underlying problem here is that users who change configure.ac etc. > need to do some additional manual work to update the build system, and > that this should be done automatically. I agree. However, this is not > the same as regenerating the full build system from scratch on every > build. Let's compare it with a Java program. Would you consider it acceptable for the packager to build it, uuencode it, put it in the diff.gz and from debian/rules just uudecode it, instead of regenerating it? I fail to see the difference between this and using a pre-built configure and Makefile.in. Compiled Java is also platform independent (AFAIK), so there isn't even a problem with architectures. > You should be aware, if you aren't already, that the process of > generating a package's build system from scratch is often really quite > complicated Of course it is. That's exactly the reason that I care so much about it. If it would be trivial, everyone could just do it, even without using debian/rules, or having it as an example. > and may well involve network access, which build daemons do > not have. Building the package should not require network access. The package is built using the version which was "the newest" (I hope) when it was created. While it could be nice to be able to use newer versions of things, I agree that we do not want that. It makes builds less reproducible, for example. > For example, the usual update process for a package that uses > Gnulib involves rsyncing translations for some messages from > translationproject.org. And so this step is not needed for the Debian package (but it would be nice if debian/rules did actually contain a target for it). > If you mandate that Debian maintainers must support full build system > regeneration on every build, then you will force some of them to > maintain a separate bootstrapping process which is guaranteed not to > access the network, which will involve duplicating work already done > by upstream and will undoubtedly introduce errors at some point. While I think Gnulib is not a good example (AFAIK it contains GNU libc functions for platforms without GNU libc, which are unused if they are available natively), I see your point. I think this should be uncommon, though, and worth the trouble. Also, if it's really too hard, it can be decided on a per-package basis to ignore policy and do things in a way that works. That means there is a bug in the package, which is hard to fix, but it doesn't mean the package can't be released. I'm not suggesting that breaking this rule should be a release-critical bug. (At least not yet. If it turns out that there are no packages (anymore) which can't follow the rule, then we may change that.) > Mere incremental regeneration on demand is much simpler and is often > already supported by upstream build systems without requiring any > change (see below). Of course it's simpler to skip a part of the compilation process. It's also simpler for Java packages to use pre-built versions instead of recompiling, but that doesn't make it right to do it. > > I suggest to mandate "remove all generated files in the clean target" > > (formulated in a way which includes "generated by upstream", not only > > "generated by the build target), which implies "rebuild everything in > > the build target". > > I object to this proposal. As an example, I am upstream for man-db and > know that its build system is properly generated. Requiring that I > regenerate it every time I build the package would introduce nothing but > hassle for me, and would not in practice provide significant benefits > for users since the build system already caters for them out of the box > (see below). The fact that there exist packages which work properly without recompiling from source doesn't mean it's a good default. IMO the default should be to always compile from source. Yes, that means hassle for the packager; it's pretty much the whole task of packaging. The benefits I see are that we know things can be generated, that users don't have to find out how to regenerate things, and what to install for it (your package may be fine, but that doesn't mean they all are). And also important, that we are sure that the shipped source files are really the source for what is generated. > > With the current wording it is allowed to use shipped built files from > > the upstream tarball, as long as the source is present. It is even > > allowed to ship the build results (uuencoded, for example) in the > > diff.gz and use those. I suppose we all agree that this is unacceptable > > for normal build results. > > configure is readable, if you have to. It's not the preferred form for > modification, certainly, but it's not an opaque object file either, and > there's no need to treat it exactly as we would treat opaque object > files. True, but also not much different. It is a generated file. > In other words, this proposal will have the effect of requiring that > Debian maintainers become experts in the Autotools before being able > to upload policy-compliant packages again, Not at all. Autoconf is pretty stable, so I suppose you're talking about automake. Because of its interface instability, you should never depend on "automake", but always on "automake1.10" (or whatever version you tested with). It's also a good idea to force the build process to use that version. That requires setting two environment variables (AUTOMAKE and ACLOCAL). I suppose you're not suggesting that knowing to set those makes you an autotools expert (especially when it will be documented, for example in the developers' reference). > whereas that burden could formerly be passed to upstream to be dealt > with on their schedule. When you're explicitly using a certain version of automake, this is still the case. There may be a bit of pushing when the automake maintainers want to remove the version you're using, but they will wait until you finished converting to the newer version (or help you with that). In fact, this makes things easier for the maintainer: if you don't have the build-dependency recorded, your automake version may disappear, leaving you unable to build the new version of the package (considering worst case, where it doesn't work with the new version and it's not easy to fix). > I wouldn't expect that figuring out enough about the Autotools to be > able to do this should be beyond most Debian maintainers, but neither > would I expect that they could all deal with some of the weird stuff > I've seen already. If really weird stuff is needed to compile the package (from source), then IMO it's totally reasonable that that weirdness is in debian/rules, so users can see what's happening, if they want to. > I agree that behavioural changes due to regenerating the build system > are bugs, but what you're suggesting is that all such bugs be elevated > to the status of showstoppers, as it will be impossible to upload > packages without it. No no, you misunderstood me. I'm saying you should get a lintian error (and a bug) without it, not that you can't upload, and not even that the bug should be RC. > I'm sure that that would make the security team's life a lot harder, > for instance Huh? I would think life only gets easier for them... Packages which do follow the rule simply don't get uploaded if they do things wrong. For packages which ignore the rule, there is no difference. Perhaps you mean that lots of bugs pop up when "automake" increases its version? In that case I apologize for being unclear; I thought it was commonly known and agreed upon that you should always build-depend on the versioned package. > Furthermore, if as a user you're desperate to make a small tweak to > configure.ac, it *is* often quite possible - if ugly! - to just make the > obvious corresponding change to configure and then get on with your > life. You can't realistically do that with e.g. compiled C object files, > which is one reason we treat them differently. I agree that this means it is less unacceptable to not regenerate them, but it doesn't make it right. Also, it is quite improper to expect this from someone who wants to NMU the package. > > Now reread the previous paragraph while thinking of Makefile.in instead > > of "normal build results". It is common practice to do exactly that: > > ship and use pre-built versions, or even ship them in the diff.gz (which > > then gets huge). Makefile.am is only present as source-file, but it > > isn't used at all during the build. Any changes the user makes will not > > be noticed by the build system. > > But this simply isn't true across the board! Makefile.am changes are > only ignored if you use AM_MAINTAINER_MODE. If you don't use it, you need to do the "touch all targets in the right order" game if you want to patch anything. Very ugly as well, if you ask me. > If you don't use AM_MAINTAINER_MODE, then Makefile.am changes are > automatically noticed and taken into account. I know. But then the system tries to run executables which aren't in build-depends. It works, but it isn't nice. Also, as I wrote above, you need to do the touching game if there are any patches. If you do this in debian/rules, the user's changes are ignored again. If you don't, it's hard to patch these parts in an NMU (since the touching must be added as well). Altogether, the behaviour is unpredictable for someone who doesn't know what's going on. I think it's reasonable for a user to edit any source file and use debian/rules to build a new package with it. > Rather than incurring the pain of gratuitous full regeneration every > time, we just regenerate it when the user has changed something. Yes, > the user now gets to resolve any problems that might have been > pre-existing, but realistically either the Debian maintainer or the > upstream maintainer is probably going to run into those at some point > anyway. I don't think that's very realistic at all for all cases. For example, if the Debian maintainer is also upstream, there's only one system. If on that system some weird things are installed which make things work even though they shouldn't, there is no reason to believe he will find out. When building the package in pbuilder, though, it will immediately show. But only if the files are indeed regenerated during that build. > I think we should recommend (but not require) that AM_MAINTAINER_MODE > not be used, Why? If you consider it acceptable for a user to edit configure, it should certainly be no problem to add --enable-maintainer-mode to the configure invocation. > and perhaps work to specify an optional debian/rules target that > regenerates the build system in an appropriate way. That seems to > provide the necessary benefits for users who need to change these > files Not at all. If it's optional, it's likely that many packages will not have it. Also, if the build system doesn't use it by default, it is likely that many of those targets are never tested and don't actually work. > without imposing an unacceptable burden on developers. I hope you do agree that maintainers should anyway be able to build their package from source? The only difference is that it should be done in debian/rules. I don't see how this is an "unacceptable burden". > I don't think there's a good cause to go much further than that at > this point. We're currently trying to allow packages to be built twice in a row as a release goal. If the clean target would be as I propose, this would automatically be fulfilled as well. Also, considering the bug we're writing to, what would you propose that the clean target should do? "Remove all generated files, except the ones that are generated by autotools"? I can't properly express how completely wrong I find such an exception when written in Policy... Thank you for responding. It would be a pretty big change in Policy, if adopted, and so it's good when there's some discussion about it, to detect any problems it might cause. Thanks, Bas -- I encourage people to send encrypted e-mail (see http://www.gnupg.org). If you have problems reading my e-mail, use a better reader. Please send the central message of e-mails as plain text in the message body, not as HTML and definitely not as MS Word. Please do not use the MS Word format for attachments either. For more information, see http://pcbcn10.phys.rug.nl/e-mail.html
Attachment:
signature.asc
Description: Digital signature