Autocompilation and source dependencies
We (Falk hueffner and me) have started to do some real autocompilation
programm/script. Source dependencies should be determined by the prog
itself and then stuffed somewhere into the package.
Heres how it should work out to work:
----------------------------------------------------------------------
- A temporary directory will be created somewhere with a lot of space
(sorry, but less than 500 MB probably won`t do at all).
- "base2_0.tgz" will be extracted there.
- Now a changeroot will be started and everything else will run in the
temporary directory.
- The debian mirror will be mounted via nfs as read-only filesystem.
- "libstdc++" and "apt" and some neccessary tools for building will be
installed.
- The package to be compiled will be installed in "/usr/local/src" via
dpkg-source -x
- All packages that are known to be necessary to build the package are
installed via apt-get install. Questions asked and answerers given
should be recorded and reused for further tries. At the first try,
no dependencies will be known, so nothing more will be installed.
- libtricks is initiated to report any read or execute command to any
file (within the changeroot and only for subprocesses of cause).
libtricks then reports any use of a file/programm on stderr.
- The build process will be started via "dpkg-buildpackage" inside the
libtricks enviroment.
- The output of libtricks is cleaned (libtricks output and errors from
compilation must be seperated), is sorted and uniquified and, if
possible, made canonical.
- Now we have a list of all files the package depends on during
build. This is compared to the Contents-<arch> file and thus
Packages are determined. In case no match is found for a file a near
match will be searched (filenames are not generally canonical) and
the near match(es) marked as possible dependency.
Now the procedure depends on the success of the buildprocess:
- If the package was build, the possible dependencies are kept in a
logfile and the real dependncies are put into the package.
- If the build fails, the dependencies and possible dependencies are
recorded in the package and the package is flaged for recompilation
and inserted at the back of the list. Esspecial care is taken with
the files that file right at the end of the build process, since that
is usually the missing dependencie that made the build fail.
If a package fails to build, but no new dependencies could be found,
the package is considered to be broken and is reported. No further
build tries will be done then until manual interaction reinsert the
package into the build process.
- The changeroot is finished. The package (if existent) are stored
together with the dependencies for that package. The tempdirectory
is deleted and the next package in the compile list is done.
- if all packages are compiled (or rejected), we eigther quit or wait
(depending how the "to be compiled" list works.
----------------------------------------------------------------------
We tried this on an Alpha 533MHz with 128 MB with xfig (also the
script is not complete yet) and several points are to be mentioned:
- Buy a fast disk, better 2 to do raid. (Its the main
limiting factor on the alpha at the moment).
- Buy a lot of mem. Parsing the libtricks output and correlating it
with the Contents file takes 50 MB ram already.
- Build a timemaschine. The buildprocess takes ages. We faked the
build on a P90, because libtricks (or is it libc`s fault?) and alpha
don`t yet like each other. Alone the amount of output generated by
libtricks slows down the build process by several factors. The
output will repeat itself many times (e.g. /usr/bin/gcc will be
called all the time, and stdio.h is read in by nearly any .c
file). We are thinking to let libtricks cache the last N lines and
do a uniquify on those N lines. That should prevent most of the
output.
- The build will usually fail, because the basic tools are not
installed. Its probably a good idea to install a set of tools by
default and have a look at the dependencie output later to see if they
are used at all. Once a package was build correctly, only the needed
packages will be installed.
- The build will find all and any package that is used during
build. This includes required packages and packages from base. If
/bin/sh is used, that will be reported. We consider that as a benefit,
since with this information a source only distribution could be build.
Any comments or suggestions are welcome.
May the Source be with you.
Mrvn
Reply to: