[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Autocompilation and source dependencies

We (Falk hueffner and me) have started to do some real autocompilation
programm/script. Source dependencies should be determined by the prog
itself and then stuffed somewhere into the package.

Heres how it should work out to work:


- A temporary directory will be created somewhere with a lot of space
  (sorry, but less than 500 MB probably won`t do at all).

- "base2_0.tgz" will be extracted there.

- Now a changeroot will be started and everything else will run in the 
  temporary directory.

- The debian mirror will be mounted via nfs as read-only filesystem.

- "libstdc++" and "apt" and some neccessary tools for building will be

- The package to be compiled will be installed in "/usr/local/src" via 
  dpkg-source -x

- All packages that are known to be necessary to build the package are 
  installed via apt-get install. Questions asked and answerers given
  should be recorded and reused for further tries. At the first try,
  no dependencies will be known, so nothing more will be installed.

- libtricks is initiated to report any read or execute command to any
  file (within the changeroot and only for subprocesses of cause).
  libtricks then reports any use of a file/programm on stderr.

- The build process will be started via "dpkg-buildpackage" inside the 
  libtricks enviroment.

- The output of libtricks is cleaned (libtricks output and errors from 
  compilation must be seperated), is sorted and uniquified and, if
  possible, made canonical.

- Now we have a list of all files the package depends on during
  build. This is compared to the Contents-<arch> file and thus
  Packages are determined. In case no match is found for a file a near
  match will be searched (filenames are not generally canonical) and
  the near match(es) marked as possible dependency.

Now the procedure depends on the success of the buildprocess:

- If the package was build, the possible dependencies are kept in a
  logfile and the real dependncies are put into the package.

- If the build fails, the dependencies and possible dependencies are
  recorded in the package and the package is flaged for recompilation
  and inserted at the back of the list. Esspecial care is taken with
  the files that file right at the end of the build process, since that
  is usually the missing dependencie that made the build fail.
  If a package fails to build, but no new dependencies could be found, 
  the package is considered to be broken and is reported. No further
  build tries will be done then until manual interaction reinsert the
  package into the build process.

- The changeroot is finished. The package (if existent) are stored
  together with the dependencies for that package. The tempdirectory
  is deleted and the next package in the compile list is done.

- if all packages are compiled (or rejected), we eigther quit or wait
  (depending how the "to be compiled" list works.


We tried this on an Alpha 533MHz with 128 MB with xfig (also the
script is not complete yet) and several points are to be mentioned:

- Buy a fast disk, better 2 to do raid. (Its the main
  limiting factor on the alpha at the moment).

- Buy a lot of mem. Parsing the libtricks output and correlating it
  with the Contents file takes 50 MB ram already.

- Build a timemaschine. The buildprocess takes ages. We faked the
  build on a P90, because libtricks (or is it libc`s fault?) and alpha
  don`t yet like each other. Alone the amount of output generated by
  libtricks slows down the build process by several factors. The
  output will repeat itself many times (e.g. /usr/bin/gcc will be
  called all the time, and stdio.h is read in by nearly any .c
  file). We are thinking to let libtricks cache the last N lines and
  do a uniquify on those N lines. That should prevent most of the

- The build will usually fail, because the basic tools are not
  installed. Its probably a good idea to install a set of tools by
  default and have a look at the dependencie output later to see if they 
  are used at all. Once a package was build correctly, only the needed
  packages will be installed.

- The build will find all and any package that is used during
  build. This includes required packages and packages from base. If
  /bin/sh is used, that will be reported. We consider that as a benefit, 
  since with this information a source only distribution could be build.

Any comments or suggestions are welcome.

May the Source be with you.

Reply to: