Re: Building Debian Completely From Source
On Friday 05 December 2003 08:03 pm, Goswin von Brederlow wrote:
> John Goerzen <jgoerzen@complete.org> writes:
[snip]
> > In other words, at no time would a .deb be downloaded. All .debs would
> > be built locally and installed locally.
I did on-target-system builds of personal backports for quite a while, but
have been much happier with the reproducibility of my work since I switched
to using pbuilder and duploading to a personal repository running
debarchiver. I haven't fully automated the build-deps process, partly
because I'm modifying sources one package at a time and partly because I am
learning how to construct a reasonably secure environment for building and
signing packages for public release. At this stage I'm using my builds on
test systems but suggesting that others fetch my diffs, look them over, and
build their own packages if they want to test my work.
Depending on your paranoia level about package tampering, you may also find
that you don't want to build your packages on general-purpose systems. One
approach is to harden one machine to serve as a local repository and firewall
behind which to put your build machines. Install the build systems from
reliable media (official Debian CDs), pre-seed the repository with
build-essential packages built however you like, and then pbuilder/sbuild
your way to nirvana.
This way you will have a central location with a reasonable audit trail of how
you built the packages you are running. Your degree of trust in the package
contents rests on your evaluation of security measures within your build
farm, your confidence in the binaries used to bootstrap your process, and
your ways of obtaining upstream source code and auditing diffs. Sounds a lot
like the Debian buildds, actually. It's a big investment, and while it may
be worth it to you, you might consider holding off until the process of
building a Debian autobuilder is better documented (or, if you are able,
contribute to that effort).
> There are several packages that have depends/build-depends
> loops. Without having a previous version you can't build them. That
> would make quite a bit of debian uninstalable for you.
These are presumably the packages with which you will need to pre-seed your
repository. Obviously the kernel and build-essential packages are in this
category. If the other build-depends loops in Debian have not at least been
enumerated and evaluated to establish whether they are real or packaging
bugs, the person who tackles this will be doing Debian a service.
> The method I use here (still experimental stage) is to have the
> original debian debs available for apt and a local repository of
> compiled sources. A script in /etc/apt/apt.conf.d monitors any deb
> that gets installed and adds original debian packages to the
> autobuilder queue. The autobuilder fetches the source and adds a 0.0.1
> to the version, a version higher than debians but lower than the next
> version, build the package and uploads it to the local repository. The
> next day an apt-get upgrade will upgrade to the locally build package.
Goswin, I'd be interested in reproducing your setup here, if you wouldn't mind
sharing it. Sounds like a good system for keeping a "guinea pig" machine
current and catching FTBFS issues in the packages you actively track.
> The benefit is that you don't have to wait for days to install
> something. You don't end up in deadlocks with dependency cycles
> either.
The former is true if the original packages work on your system. You might be
running, say, a PaX kernel -- I understand that some binaries don't work
until they are recompiled to be PaX-friendly. Or you might be sticking to a
"known good" glibc release, as I did with 2.2.5-final for quite a while, and
so need to rebuild many binaries with versioned depends. Or you might just
not want ever to run a binary someone else compiled (barring, of course, your
bootstrap build-essential packages).
The latter is not true if you are unwilling to install the debian binary
package long enough to build your own. Depends again on your reason for
wanting to build your own copy of everything. Personally, I have never yet
had a business reason for this level of caution, but I can certainly imagine
circumstances under which I would.
[snip]
> > We seem to already have the metadata necessary to do this, but as far as
> > I can tell, our existing tools don't support it. For instance:
> >
> > * apt-build has an option to build a package and all packages it
> > depends upon, but always downloads binary .debs of -dev packages.
> > (Or maybe I got this backwards; I'm not sitting at my apt-build
> > machine right now.)
> >
> > * pbuilder and sbuild don't bother with actual installation,
> > so they don't resolve dependencies at all. Also, they only download
> > .debs of build-deps.
> >
> > The nearest I have seen is fink, but I know little about it.
> >
> > Am I missing something? Is anyone working on this?
>
> buildd, wanna-build?
buildd's don't install build results on the live system either, and AFAIK they
fetch binary debs for build-deps. If someone writes (or has written) the
abovementioned recursive build-depends analysis tool (presumably ~20 lines of
your choice of scripting language plus <80 lines of module imports and
syntactic sugar), it would be a convenience if it could consume a list of
currently installed and/or available (binary) packages and produce an ordered
list of packages to be built and installed to reach a target package and all
its dependencies. Is this what wanna-build does?
I think it's unrealistic at this point to be able to automate building and
installing a complex set of build-depends on your actual target system with
zero human intervention; perhaps libfoo-dev rarely asks a debconf question,
but installing, say, tetex or imagemagick probably does. Besides, you
probably don't want to burden all your target systems with all of the build
machinery; how many DNS servers would otherwise install, say, jadetex? And
remember that the build-depends for, say, vim includes everything needed to
build all the binary packages derived from the same source package, which
will pull in all the build-depends for ruby, tcl, ...
In my experience, pbuilder works well, so accepting the defaults isn't a bad
strategy for constructing a chroot build environment. (Come to think of it,
that's an additional testament to the high quality of current Debian
packaging for development tools -- not just compilers, but document
generation systems and so forth.)
Binary packages of open source code are a good thing, as long as they are
trusted and reproducible. By all means build your own if it increases your
confidence in them, but IMHO not on the target system itself.
- Michael
Reply to: