(CCing the list as I think it is useful to have this archived and I guess
it could be interesting to others as well)
Hi Felipe,
I hope you do not yet regret your impulsive reaction of agreeing to take
over management of the installation guide ;-)
Anyway, thanks a lot!
I suggest we take this step-by-step. First I think would be to get you
started taking care of the "daily" builds; we can discuss other areas
like later; these include:
- troubleshooting build problems
- official builds for uploads
- official builds for the website
- inclusion of the manual on CD images
- release management
- acting as editor (optionally)
If you'd like to use my mails and things you learn along the way to create
some sort of more structured and permanent documentation, that would be
fine by me. I think it would be very useful.
I put "daily" in quotes because I have never actually done them daily.
There really is no need for that as the mutation frequency is not all
that high. When I had the builds cronned, they were done 4 times per
week; currently I run them manually [1] and basically as needed.
IMO it would be good to have the builds running more regular again. It is
also good to run more frequent builds, for example when a release is
scheduled and loads of translation updates are being committed.
Types of builds
---------------
There are basically four types of builds:
- single, manual builds for a specific architecture/language
uses build/buildone.sh from the commandline
- daily builds for alioth using the infrastructure described below
- official builds for uploads
done from debian/rules which calls build/build.sh
- official builds for the website
done on www-master using build/buildweb.sh; triggered by uploads
Setting up the build environment
--------------------------------
The following commands should be enough to set up a build environment:
$ cd <wherever you want to put this>
This directory will be $ROOT. I'd suggest /srv/d-i_manual.daily/.
Make sure you are allowed to write to this directory and that correct
permissions are set for files and subdirs that are created in it.
$ mkdir bin
$ mkdir manual
$ cd bin
$ svn co svn+ssh://svn.debian.org/svn/d-i/trunk/scripts/manual-daily
This first checkout gets you the daily build scripts. It is fine to make
changes here and commit these.
I'll introduce the files in this directory later.
$ cd ../manual
$ svn co -r 47661 svn+ssh://svn.debian.org/svn/d-i/trunk/manual
$ dpkg-checkbuilddeps
$ sudo aptitude install <whatever you're missing>
$ sudo aptitude install rsync openssh-client # if not already there
I have included '-r 47661' because that is the revision for which I have
run my last daily build, so that is where you should start. If you don't
have time to set up things quickly enough, I can continue to run builds,
but we should then agree on a new revision for you to start at.
You need a separate checkout of the manual the daily builds because builds
are triggered by updates (an 'svn up' is done at the beginning of a
build). This also means that as a general rule you should never trace or
fix build problems in this checkout, but rather use a "normal checkout"
to trace, fix and test the problem, then commit. After that just run a
daily build again (possibly a retry) and let that pull in the changes.
Next, you'll need to create a config file in the $HOME of the user you
will use to run the builds; this should probably just be your own account
as you need commit and upload access to alioth/svn.
This config file is sourced by several of the build scripts. I created it
for ensure consistency and to make sure I only needed to make changes in
one place instead of having to grep endlessly and still forget changes in
certain places.
For me, this file contains:
<snip>
$ cat ~/.d-i_manual_daily
# Configuration settings for "daily" builds of the Installation Guide
ROOT=/srv/d-i_manual.daily
source=$ROOT/manual
destination=$ROOT/build
logdir=$ROOT/log
NAME_FROM="Frans Pop"
MAIL_FROM="elendil@planet.nl"
</snip>
The first set of values tells the scripts where to find/write things.
The second set is used to send log mails for the builds (please change the
mail address and name!). So you'll also need to make sure that the system
is correctly set up to send mail.
If you intend to run the builds cronned, make sure that you also receive
any mails cron will send if there is output from cronjobs.
Note that the scripts currently assume that you can just 'svn ci', 'scp'
and 'rsync' to alioth using an ssh key; if you need anything special,
you'll probably need to modify things.
Contents of the build environment
---------------------------------
Directories:
./bin - scripts and support files; from D_I SVN ./scripts
./manual - local checkout of the manual for daily builds
./build - place were built files are moved; rsynced to alioth from there
./log - log files are created here; some are copied to alioth
Files in ./bin:
README - ...
archlist - architectures to be built + descriptions
langlist - languages to be built + various info needed
build-main - build wrapper script: this is what you execute
(indentation in following lines indicates calls)
.build-manual - main build script: does the actual builds and
copying of results to alioth
..create-index - creates index page for dev. version of manual [2]
...translation-stats - calculates translation stats for XML based transl.
...po_stats/ - script/files to calculate translation stats for
PO based transl.
.mail-logs - sends log mails after build completes/fails
index.head - template file for index page used by create-index
index.single - same (table of langs for which only i386 is built)
index.stats - same ("statistics" section)
index.pohead - same ("PO statistics" section)
index.tail - same (page footer)
index.changes - items for "Recent changes" section
translators.html - web page linked from index page
msgtxt.common - template for build log mails to translators
msgtxt.ok - same (for successful builds)
msgtxt.fail - same (for failed builds)
nopdf.pdf - placeholder file uploaded if build of PDF fails
notxt.txt - placeholder file uploaded if build of text fails
Running the builds
------------------
You will need to study the scripts themselves for the gory details about
how things are done. Most scripts do have some comments to explain what's
happening and why. Running with 'set -x' will probably also help a lot to
understand the details :-)
Normally you should just execute the 'build-main' script. If I run a build
manually I will mostly do:
$ ./build-main &
$ tail -f ../log/main.log
The tail of the overview log allows me to follow the build.
You can also execute 'build-manual' separately if you want to avoid
sending log mails for some reason. It is possible to manually run
'mail-logs' afterwards; that can also be run separately to resend the log
mails from the last build run.
The steps for a build run are follows:
- check for lock files (see next section)
- create lock file to prevent two builds running at the same time
- do 'svn up'
- if there are updates for en:
- build en version
- if successful, move to output directory
- update POT files
- for all PO-based translations
- update PO files (if either the language itself or en has updates)
- generate XML files (if the language has updates)
If there are is any build failure in the en version, the daily build will
be aborted. This is mainly to avoid such breakage to leak into the PO
files and thus affect any PO-based translations.
The same goes when there are errors generating PO or XML files (can happen
if an invalid PO file is committed by a translator).
See the next section on how to deal with this and other breakage.
- 'svn ci' any changes (in POT and PO files) from the previous steps
- for all other languages that have updates
- build the language
- if successful, move to output directory
- upload its log file to alioth
For translations, build failures in the PDF or text versions will not
prevent the HTML version from being uploaded.
- generate statistics and index page
- upload anything that was built to alioth
- remove lockfile
- mail language log files to translators (unless build was aborted)
if log upload was successful only a link will be included, if not the
log will be attached
- mail overview log file to faw :-)
The log directory will contain:
- main.log - overview log file
- <lang>.log - detailed build log for a specific language
- svn_up.log - log for svn update; parsed to check for updates
- svn_ci.log - log for svn commit
- po-stats.log - log file from po_stats/gen
- tr-stats - output file from XML-based translation stats
Build failures
--------------
Build failures (of output files) in individual languages will not abort
the build as a whole and fixing them is in principle the responsibility
of translators. However, in some cases (especially failures in PDF
builds), they may need some help. In my experience managing the daily
builds will give you plenty of experience in debugging build errors :-)
If the build is really aborted for any reason, a file do_not_build will be
created in the $ROOT directory. This will prevent any other builds from
starting until you have deleted it manually (after fixing the cause of
the build failure).
After a build failure, you should in principle always restart the build
manually with the --retry (or --restart, or -r) option. If a build is
restarted, the SVN logs are not deleted. Instead the new 'svn up' or 'svn
ci' are appended to whatever was left after the previous build, which
ensures that all translations that had updates will be built again.
Basically, the whole build will be done again and any commits to D-I SVN
that were done in the mean time (hopefully including the fix for the
build failure) will be included.
Managing the builds
-------------------
This interacts somewhat with release management, so I'll keep it limited
for now.
One task - that should probably be shared among editors of the original
(English) version, but may fall to you by default - is adding notices
about important changes affecting translators in 'index.changes'.
I'm not quite sure if translators actually check these, but well...
Changes worth being listed include:
- addition of new sections
- major changes in existing text (especially if text is moved unchanged)
- changes in the use of optional entities (build/entities/l10n/) or in
translatable entities (build/lang-options)
- ...
(It would probably make sense to remove most items that predate the Etch
release.)
The other task is (de)activating languages for the daily builds, which is
done in bin/languagelist.
Note that for official builds there are separate archlist and langlist
files in the debian/ dir of the manual. This is mostly because other
criteria are used to activate a language for the development version than
for the official version.
There are three possible states for a language:
- full builds - "Y" in column 'Build'
For languages that are maintained and fairly complete.
- single builds (only i386) - "S" in column 'Build'
For languages that are very outdated (more that a release behind)
and languages that are very incomplete (new translations).
These languages are listed in a separate section on the index page.
- disabled - line commented out
For completely abandoned/aborted translations. These languages are not
listed on the index page at all.
Otherwise, just be careful when making changes in the build system. In my
experience seemingly innocent changes often have unexpected consequences.
That should not prevent you from making improvements if you see them
though.
Hope this is enough to get you started.
In practice all this is not all that much work. Builds aborting happens
only rarely (but always when preparing a release...).
Most work is in release management, not in the daily builds. Still, if you
feel this is too much, now is your last chance to back out :-)
Cheers,
FJP
[1] My ancient PIII server was unreliable for a while and then I got
another box (not a server) that could run the builds a lot faster, so I
kept them manual.
[2] http://d-i.alioth.debian.org/manual/
Attachment:
pgp8Ni89UKrkH.pgp
Description: PGP signature