[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[long] Management of the installation guide - part I

(CCing the list as I think it is useful to have this archived and I guess 
it could be interesting to others as well)

Hi Felipe,

I hope you do not yet regret your impulsive reaction of agreeing to take 
over management of the installation guide ;-)
Anyway, thanks a lot!

I suggest we take this step-by-step. First I think would be to get you 
started taking care of the "daily" builds; we can discuss other areas 
like later; these include:
- troubleshooting build problems
- official builds for uploads
- official builds for the website
- inclusion of the manual on CD images
- release management
- acting as editor (optionally)

If you'd like to use my mails and things you learn along the way to create 
some sort of more structured and permanent documentation, that would be 
fine by me. I think it would be very useful.

I put "daily" in quotes because I have never actually done them daily. 
There really is no need for that as the mutation frequency is not all 
that high. When I had the builds cronned, they were done 4 times per 
week; currently I run them manually [1] and basically as needed.
IMO it would be good to have the builds running more regular again. It is 
also good to run more frequent builds, for example when a release is 
scheduled and loads of translation updates are being committed.

Types of builds

There are basically four types of builds:
- single, manual builds for a specific architecture/language
  uses build/buildone.sh from the commandline
- daily builds for alioth using the infrastructure described below
- official builds for uploads
  done from debian/rules which calls build/build.sh
- official builds for the website
  done on www-master using build/buildweb.sh; triggered by uploads

Setting up the build environment

The following commands should be enough to set up a build environment:
$ cd <wherever you want to put this>

This directory will be $ROOT. I'd suggest /srv/d-i_manual.daily/.
Make sure you are allowed to write to this directory and that correct 
permissions are set for files and subdirs that are created in it.

$ mkdir bin
$ mkdir manual
$ cd bin
$ svn co svn+ssh://svn.debian.org/svn/d-i/trunk/scripts/manual-daily

This first checkout gets you the daily build scripts. It is fine to make 
changes here and commit these.
I'll introduce the files in this directory later.

$ cd ../manual
$ svn co -r 47661 svn+ssh://svn.debian.org/svn/d-i/trunk/manual
$ dpkg-checkbuilddeps
$ sudo aptitude install <whatever you're missing>
$ sudo aptitude install rsync openssh-client # if not already there

I have included '-r 47661' because that is the revision for which I have 
run my last daily build, so that is where you should start. If you don't 
have time to set up things quickly enough, I can continue to run builds, 
but we should then agree on a new revision for you to start at.

You need a separate checkout of the manual the daily builds because builds 
are triggered by updates (an 'svn up' is done at the beginning of a 
build). This also means that as a general rule you should never trace or 
fix build problems in this checkout, but rather use a "normal checkout" 
to trace, fix and test the problem, then commit. After that just run a 
daily build again (possibly a retry) and let that pull in the changes.

Next, you'll need to create a config file in the $HOME of the user you 
will use to run the builds; this should probably just be your own account 
as you need commit and upload access to alioth/svn.
This config file is sourced by several of the build scripts. I created it 
for ensure consistency and to make sure I only needed to make changes in 
one place instead of having to grep endlessly and still forget changes in 
certain places.

For me, this file contains:
$ cat ~/.d-i_manual_daily
# Configuration settings for "daily" builds of the Installation Guide


NAME_FROM="Frans Pop"

The first set of values tells the scripts where to find/write things.
The second set is used to send log mails for the builds (please change the 
mail address and name!). So you'll also need to make sure that the system 
is correctly set up to send mail.
If you intend to run the builds cronned, make sure that you also receive 
any mails cron will send if there is output from cronjobs.

Note that the scripts currently assume that you can just 'svn ci', 'scp' 
and 'rsync' to alioth using an ssh key; if you need anything special, 
you'll probably need to modify things.

Contents of the build environment

./bin    - scripts and support files; from D_I SVN ./scripts
./manual - local checkout of the manual for daily builds
./build  - place were built files are moved; rsynced to alioth from there
./log    - log files are created here; some are copied to alioth

Files in ./bin:
README                - ...
archlist              - architectures to be built + descriptions
langlist              - languages to be built + various info needed
build-main            - build wrapper script: this is what you execute
                        (indentation in following lines indicates calls)
.build-manual         - main build script: does the actual builds and
                        copying of results to alioth
..create-index        - creates index page for dev. version of manual [2]
...translation-stats  - calculates translation stats for XML based transl.
...po_stats/          - script/files to calculate translation stats for
                        PO based transl.
.mail-logs            - sends log mails after build completes/fails
index.head            - template file for index page used by create-index
index.single          - same (table of langs for which only i386 is built)
index.stats           - same ("statistics" section)
index.pohead          - same ("PO statistics" section)
index.tail            - same (page footer)
index.changes         - items for "Recent changes" section
translators.html      - web page linked from index page
msgtxt.common         - template for build log mails to translators
msgtxt.ok             - same (for successful builds)
msgtxt.fail           - same (for failed builds)
nopdf.pdf             - placeholder file uploaded if build of PDF fails
notxt.txt             - placeholder file uploaded if build of text fails

Running the builds

You will need to study the scripts themselves for the gory details about 
how things are done. Most scripts do have some comments to explain what's 
happening and why. Running with 'set -x' will probably also help a lot to 
understand the details :-)

Normally you should just execute the 'build-main' script. If I run a build 
manually I will mostly do:
$ ./build-main &
$ tail -f ../log/main.log
The tail of the overview log allows me to follow the build.

You can also execute 'build-manual' separately if you want to avoid 
sending log mails for some reason. It is possible to manually run
'mail-logs' afterwards; that can also be run separately to resend the log 
mails from the last build run.

The steps for a build run are follows:
- check for lock files (see next section)
- create lock file to prevent two builds running at the same time
- do 'svn up'
- if there are updates for en:
  - build en version
  - if successful, move to output directory
  - update POT files
- for all PO-based translations
  - update PO files (if either the language itself or en has updates)
  - generate XML files (if the language has updates)

If there are is any build failure in the en version, the daily build will 
be aborted. This is mainly to avoid such breakage to leak into the PO 
files and thus affect any PO-based translations.
The same goes when there are errors generating PO or XML files (can happen 
if an invalid PO file is committed by a translator).
See the next section on how to deal with this and other breakage.

- 'svn ci' any changes (in POT and PO files) from the previous steps
- for all other languages that have updates
  - build the language
  - if successful, move to output directory
  - upload its log file to alioth

For translations, build failures in the PDF or text versions will not 
prevent the HTML version from being uploaded.

- generate statistics and index page
- upload anything that was built to alioth
- remove lockfile
- mail language log files to translators (unless build was aborted)
  if log upload was successful only a link will be included, if not the
  log will be attached
- mail overview log file to faw :-)

The log directory will contain:
- main.log     - overview log file
- <lang>.log   - detailed build log for a specific language
- svn_up.log   - log for svn update; parsed to check for updates
- svn_ci.log   - log for svn commit
- po-stats.log - log file from po_stats/gen
- tr-stats     - output file from XML-based translation stats

Build failures

Build failures (of output files) in individual languages will not abort 
the build as a whole and fixing them is in principle the responsibility 
of translators. However, in some cases (especially failures in PDF 
builds), they may need some help. In my experience managing the daily 
builds will give you plenty of experience in debugging build errors :-)

If the build is really aborted for any reason, a file do_not_build will be 
created in the $ROOT directory. This will prevent any other builds from 
starting until you have deleted it manually (after fixing the cause of 
the build failure).

After a build failure, you should in principle always restart the build 
manually with the --retry (or --restart, or -r) option. If a build is 
restarted, the SVN logs are not deleted. Instead the new 'svn up' or 'svn 
ci' are appended to whatever was left after the previous build, which 
ensures that all translations that had updates will be built again.

Basically, the whole build will be done again and any commits to D-I SVN 
that were done in the mean time (hopefully including the fix for the 
build failure) will be included.

Managing the builds

This interacts somewhat with release management, so I'll keep it limited 
for now.

One task - that should probably be shared among editors of the original 
(English) version, but may fall to you by default - is adding notices 
about important changes affecting translators in 'index.changes'.
I'm not quite sure if translators actually check these, but well...
Changes worth being listed include:
- addition of new sections
- major changes in existing text (especially if text is moved unchanged)
- changes in the use of optional entities (build/entities/l10n/) or in
  translatable entities (build/lang-options)
- ...
(It would probably make sense to remove most items that predate the Etch 

The other task is (de)activating languages for the daily builds, which is 
done in bin/languagelist.
Note that for official builds there are separate archlist and langlist 
files in the debian/ dir of the manual. This is mostly because other 
criteria are used to activate a language for the development version than 
for the official version.

There are three possible states for a language:
- full builds - "Y" in column 'Build'
  For languages that are maintained and fairly complete.
- single builds (only i386) - "S" in column 'Build'
  For languages that are very outdated (more that a release behind)
  and languages that are very incomplete (new translations).
  These languages are listed in a separate section on the index page.
- disabled - line commented out
  For completely abandoned/aborted translations. These languages are not
  listed on the index page at all.

Otherwise, just be careful when making changes in the build system. In my 
experience seemingly innocent changes often have unexpected consequences. 
That should not prevent you from making improvements if you see them 

Hope this is enough to get you started.

In practice all this is not all that much work. Builds aborting happens 
only rarely (but always when preparing a release...).
Most work is in release management, not in the daily builds. Still, if you 
feel this is too much, now is your last chance to back out :-)


[1] My ancient PIII server was unreliable for a while and then I got 
another box (not a server) that could run the builds a lot faster, so I 
kept them manual.
[2] http://d-i.alioth.debian.org/manual/

Attachment: pgpc756aQlirS.pgp
Description: PGP signature

Reply to: