[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Building Debian Completely From Source

On Friday 05 December 2003 08:03 pm, Goswin von Brederlow wrote:
> John Goerzen <jgoerzen@complete.org> writes:
> > In other words, at no time would a .deb be downloaded.  All .debs would
> > be built locally and installed locally.

I did on-target-system builds of personal backports for quite a while, but 
have been much happier with the reproducibility of my work since I switched 
to using pbuilder and duploading to a personal repository running 
debarchiver.  I haven't fully automated the build-deps process, partly 
because I'm modifying sources one package at a time and partly because I am 
learning how to construct a reasonably secure environment for building and 
signing packages for public release.  At this stage I'm using my builds on 
test systems but suggesting that others fetch my diffs, look them over, and 
build their own packages if they want to test my work.

Depending on your paranoia level about package tampering, you may also find 
that you don't want to build your packages on general-purpose systems.  One 
approach is to harden one machine to serve as a local repository and firewall 
behind which to put your build machines.  Install the build systems from 
reliable media (official Debian CDs), pre-seed the repository with 
build-essential packages built however you like, and then pbuilder/sbuild 
your way to nirvana.

This way you will have a central location with a reasonable audit trail of how 
you built the packages you are running.  Your degree of trust in the package 
contents rests on your evaluation of security measures within your build 
farm, your confidence in the binaries used to bootstrap your process, and 
your ways of obtaining upstream source code and auditing diffs.  Sounds a lot 
like the Debian buildds, actually.  It's a big investment, and while it may 
be worth it to you, you might consider holding off until the process of 
building a Debian autobuilder is better documented (or, if you are able, 
contribute to that effort).

> There are several packages that have depends/build-depends
> loops. Without having a previous version you can't build them. That
> would make quite a bit of debian uninstalable for you.

These are presumably the packages with which you will need to pre-seed your 
repository.  Obviously the kernel and build-essential packages are in this 
category.  If the other build-depends loops in Debian have not at least been 
enumerated and evaluated to establish whether they are real or packaging 
bugs, the person who tackles this will be doing Debian a service.

> The method I use here (still experimental stage) is to have the
> original debian debs available for apt and a local repository of
> compiled sources. A script in /etc/apt/apt.conf.d monitors any deb
> that gets installed and adds original debian packages to the
> autobuilder queue. The autobuilder fetches the source and adds a 0.0.1
> to the version, a version higher than debians but lower than the next
> version, build the package and uploads it to the local repository. The
> next day an apt-get upgrade will upgrade to the locally build package.

Goswin, I'd be interested in reproducing your setup here, if you wouldn't mind 
sharing it.  Sounds like a good system for keeping a "guinea pig" machine 
current and catching FTBFS issues in the packages you actively track.

> The benefit is that you don't have to wait for days to install
> something. You don't end up in deadlocks with dependency cycles
> either.

The former is true if the original packages work on your system.  You might be 
running, say, a PaX kernel -- I understand that some binaries don't work 
until they are recompiled to be PaX-friendly.  Or you might be sticking to a 
"known good" glibc release, as I did with 2.2.5-final for quite a while, and 
so need to rebuild many binaries with versioned depends.  Or you might just 
not want ever to run a binary someone else compiled (barring, of course, your 
bootstrap build-essential packages).

The latter is not true if you are unwilling to install the debian binary 
package long enough to build your own.  Depends again on your reason for 
wanting to build your own copy of everything.  Personally, I have never yet 
had a business reason for this level of caution, but I can certainly imagine 
circumstances under which I would.

> > We seem to already have the metadata necessary to do this, but as far as
> > I can tell, our existing tools don't support it.  For instance:
> >
> >  * apt-build has an option to build a package and all packages it
> >    depends upon, but always downloads binary .debs of -dev packages.
> >    (Or maybe I got this backwards; I'm not sitting at my apt-build
> >    machine right now.)
> >
> >  * pbuilder and sbuild don't bother with actual installation,
> >    so they don't resolve dependencies at all.  Also, they only download
> >    .debs of build-deps.
> >
> > The nearest I have seen is fink, but I know little about it.
> >
> > Am I missing something?  Is anyone working on this?
> buildd, wanna-build?

buildd's don't install build results on the live system either, and AFAIK they 
fetch binary debs for build-deps.  If someone writes (or has written) the 
abovementioned recursive build-depends analysis tool (presumably ~20 lines of 
your choice of scripting language plus <80 lines of module imports and 
syntactic sugar), it would be a convenience if it could consume a list of 
currently installed and/or available (binary) packages and produce an ordered 
list of packages to be built and installed to reach a target package and all 
its dependencies.  Is this what wanna-build does?

I think it's unrealistic at this point to be able to automate building and 
installing a complex set of build-depends on your actual target system with 
zero human intervention; perhaps libfoo-dev rarely asks a debconf question, 
but installing, say, tetex or imagemagick probably does.  Besides, you 
probably don't want to burden all your target systems with all of the build 
machinery; how many DNS servers would otherwise install, say, jadetex?  And 
remember that the build-depends for, say, vim includes everything needed to 
build all the binary packages derived from the same source package, which 
will pull in all the build-depends for ruby, tcl, ...

In my experience, pbuilder works well, so accepting the defaults isn't a bad 
strategy for constructing a chroot build environment.  (Come to think of it, 
that's an additional testament to the high quality of current Debian 
packaging for development tools -- not just compilers, but document 
generation systems and so forth.)

Binary packages of open source code are a good thing, as long as they are 
trusted and reproducible.  By all means build your own if it increases your 
confidence in them, but IMHO not on the target system itself.

- Michael

Reply to: