PROPOSAL: automatic installation and configuration
Hi,
I have done a few experiments about automatic configuration of packages
and have now some working code. I want to describe my ideas and how to
integrate in the distribution. You can download my experimental but
almost working code at:
http://www.cs.unitn.it/~dz/debian/dpkggetconfig_1.0.deb
My idea of automatic installation is that if I need to install many similar
machines I could install manually the first, remember all the answer to
each configuration question, then go to the other machines and just type
install without any more prompting. At the moment this is only a dream
given the terrible way the install scripts are designed and written.
So I've written a little package which could be used by developers in
their postinst scripts to allow the automatic installation and
configuration of a debian system with a very minimal changes to the
existing code.
The overall goal of my project are:
* make possible a totally automatic installation without prompting
* save automatically all answers during a manual installation for later
reuse in subsequent automatic, semi-automatic or manual installations
* be able select different levels of interaction with the user, which
could correspond to user expertise levels, but not necessarily
* have the least impact on existing install scripts. Obvioulsy they
have to be modified but the changes are usually very small
* provide a uniform and consistent interface for user interaction
* use a graphical interface when possible or the old plain tty, at
user choice
* save important installation messages, which currently are only shown
on the screen with a 'press enter' prompt, for later inspection
* use a flexible database structure with more levels of configuration
files (script, debian, site, host, package, updates, etc.) and
inheritance of configuration values
* use only ascii databases and base programs for ease of maintenance
and independence from installed packages or esotheric software
My proposal doesn't attempt to address the aspects of package configuration,
like the recent proposal of Goswin Brederlow did. I'm only interested in
replicating automatically the sequence of answers that users must currently
do by hand. A structured configuration system could provide these features
and many more but is is a much more ambitious project and would require much
more effort and developement time. My proposed API could however be reused
in a subsequent configuration project.
THE CODE.
Of the about 700 installed packages in my system I found that only 70
scripts need to interact with the user. Most of the interaction of those
scripts falls in three cathegories:
1) echo "message"
echo -n "enter value:"; read answer
2) echo "message"
echo -n "do you want... ? (y/n)"; read answer
if [ $answer = y ]; then ...
This is the most frequent case. I believe that 90% of the prompts
are for yes/no questions
3) echo "read this message"
echo -n "press enter"; read enter
This is also a very common case
Obviously each scripts implements the above code in a completely different
and clever way. My idea is that if we replace all the above fragments in
every script with calls to a standardized interface we could intercept all
the user interaction and implement whatever control we want. So I wrote two
commands which could be used to easily replace the above code providing
an easier uniform method of getting answers from the user and a transparent
interface with a simple flat ascii configuration database. The commands are:
dpkg-getconfig <varname> [options...]
options:
-\?|-h|--help print this help
-a|--auto|--from-db don't ask, get value from db
-b|--bool|--boolean accept only true/false values
-c|--choice <values...> "." accept choice between values
-C|--check <check-command> input check command
-d|--default <default> set default value
-D|--DEFAULT <default> force default value
-e|--expert set query level = expert
-f|--file|--database <db> set database file
-i|--important set query level = important
-l|--level <query-level> set the query level
-L|--lines|--text accept multi-line text
-m|--msg|--message <message> message text (- = stdin)
-o|--once|--missing set query level = once
-O|--output|--outfile <filename> write answer to filename
-p|--prompt <prompt> prompt message
-P|--package <package> querying package name
-s|--set|--set-value <value> don't ask, write value to db
-t|--title|--subject <title> set window title
-v|--var|--variable <varname> set variable name
-V|--verbose set query level = verbose
-y|--yesno|--yes-no accept only yes/no values
dpkg-message [options...] <message>
options:
-\?|-h|--help print this help
-a|--auto|--to-mailbox don't show, save in mailbox
-e|--expert set notify level = expert
-f|--file|--database <db> set database file
-i|--important set notify level = important
-l|--level <query-level> set notify level
-m|--msg|--message <message> messsage text (- = stdin)
-n|--nowait don't wait for Enter
-o|--output|--mailbox <mailbox> default mailbox
-O|--OUTPUT|--MAILBOX <mailbox> force mailbox
-t|--title|--subject <title> window title or mail subject
-V|--verbose set notify level = verbose
-w|--wait|--sleep <n> wait n seconds after message
Using only these two commands the above script fragments could be easily
rewritten as:
1a) answer=$(dpkg-getconfig PACKAGE_VARIABLE \
--default default --message "message" --prompt "prompt)
1b) answer=$(dpkg-getconfig PACKAGE_VARIABLE \
--choices "aaa" "bbb" "ccc" . \
--default "ccc" --message "message" --prompt "prompt)
1c) answer=$(dpkg-getconfig PACKAGE_VARIABLE \
--lines --message "message" --prompt "prompt)
2) answer=$(dpkg-getconfig PACKAGE_VARIABLE \
--yesno --default y --message "message" --prompt "prompt)
3) dpkg-message --title "title" --message "message"
The benefits, besides the simplification of the code and the uniform user
interface, are that all values entered by the user can be automatically
saved into a database, and that by setting a global configuration variable,
transparent to the scripts, the values can be later retrieved automatically
from the database instead of prompting the user.
The messages printed with dpkg-message are shown and automatically saved
in a mailbox, and can be later be examined with elm or any mail program.
The messages are appended directly to the mailbox and don't require an
installed and working MTA. So after the installation is finished one can
find many nice mail messages telling him exactly what the developer wanted
to notify. In this way nothing gets lost and you don't need to pay
attention to the terminal during the installation.
The level of interaction with the user can be controlled by the combination
of two values, an installation mode, or prompt level, or user level, which
can be one of the following:
auto don't ask nothing, use defaults or fail
important ask only very important questions
once ask 'once' questions only the first time
normal ask all normal questions every time (default)
expert ask also questions for expert users
verbose ask everything, for masochists only
and a corresponding importance value for each question which can be:
auto don't ask, read from db or script defaults
important important question
once normal question, but ask only once
normal normal question (default)
expert question for expert users
verbose noise question
The user is prompted only if the install mode is equal or higher than the
query mode. For example a normal question(3) is asked only it the install
mode id normal(3) or higher. If one chooses to see only the important(1)
questions a normal or expert question will never asked but only retrieved
from the database or from the script defaults. The script can specify two
defaults for a query, with different priority respect the database value.
For example:
dpkg-getconfig VAR_NAME --DEFAULT $x --default $y
will return, or propose to the user, $x if not empty, or the database
default if not empty, or $y. This is rarely used but sometimes handy.
If the user is prompted he will have anyway the last choice.
If the user selects the "auto" install mode and has stored answers or
scripts defaults for every question the system can be installed in a
totally automatic way while he is playing tetris on another console.
The anwers can be easily and automatically collected by running a manual
installation the first time and are stored in an ascii file which can be
edited with vi or any standard text editor.
The format of the database is very simple, it is actually one or more
bash scripts which are sourced in sequence. The updates db collects all
the user answers and is loaded last so that its values override any setting
from other files.
The fact that the database is a script makes possible to implement easily
whatever inheritance and priority scheme is needed. My suggestion is that
we use the following files:
config.db the main config file which sources all the others
debian.db debian official defaults (read-only)
site.db site defaults
host.db host specific settings
package.db an optional specific db for each package
updates.db collects all user answers
The database and its format could be easily changed in the future if all
the scripts use only the dpkg-* API, so it is really not very important.
I choose bash scripts becaues they are handy for experimenting with new
ideas but we could in future use a more efficient and fast format.
For consistency the names of the configuration variables should be in
uppercase and should follow the following conventions:
<PACKAGE>_<VARIABLE>
for example:
NET_ETH0_NETMASK for the variable ETH0_NETMASK owned by package net
or
CONFIG_DEFAULT_PAPERFORMAT for a common variable shared between
many packages and not owned by any of them
All the common configuration variables should begin with the prefix CONFIG,
assuming that there will never be a package with that name.
PRELIMINARY TESTS.
I have tested the scriprs on my slink. I haven't done a complete install
because it would have meant modifying and repackaging a lot of things, but
I tested it with dotfile-bash and I seems to work fine. I will show this
example to illustrate how to change postinst scripts. This is the original
dotfile-bash postinst:
------------------------------------------------------------------------
#!/bin/sh
set -e
if [ ! -f /usr/X11R6/lib/X11/dotfile/bash/bytecompile ]
then
echo " Byte compiling modules makes them run significantly faster"
echo -n "Do you want to bytecompile this module now? (Y/n) ? "; read answer
case $answer
in
N*|n*)
echo "you can bytecompile this module later by typing:"
echo "dotfile bash bytecompile";;
*)
/usr/bin/tclsh /usr/X11R6/lib/X11/dotfile/Generator/dotfile.tcl /usr/X11R6/lib/X11/dotfile/Generator /usr/X11R6/lib/X11/dotfile/bash bytecompile ;;
esac
fi
if [ -x /usr/bin/update-menus ] ; then update-menus ; fi
------------------------------------------------------------------------
This is a simple yes/no question which could have been rplaced by the
following code:
answer=$(dpkg-getconfig DOTFILE_BASH_BYTECOMPILE \
--yesno --default y --package dotfile-bash \
-m "Byte compiling modules makes them run significantly faster"\
-p "Do you want to bytecompile this module now?")
I choose however to make the code a little more complicated to show also
how to simplify the installation from the user point of view. This is the
new script modified for use of dpkg-getconfig:
------------------------------------------------------------------------
#!/bin/sh
set -e
if [ ! -f /usr/X11R6/lib/X11/dotfile/bash/bytecompile ]; then
bytecompile=$(dpkg-getconfig CONFIG_DOTFILE_BYTECOMPILE \
--message "\
Byte compiling modules makes them run significantly faster.
Do you want to bytecompile every module?" \
--choices "yes - byte-compile every package" \
"no - don't compile anything" \
"ask - prompt for each package" \
"auto - use package default" . \
--default yes --once --package dotfile-bash)
case "$bytecompile" in
y*)
bytecompile=y
;;
n*)
bytecompile=n
;;
auto*)
bytecompile=$(dpkg-getconfig DOTFILE_BASH_BYTECOMPILE \
--yesno --default y --auto --package dotfile-bash)
;;
*)
bytecompile=$(dpkg-getconfig DOTFILE_BASH_BYTECOMPILE \
--yesno --default y --package dotfile-bash \
-m "Byte compiling modules makes them run significantly faster"\
-p "Do you want to bytecompile this module now?")
;;
esac
case "$bytecompile" in
y*)
/usr/bin/tclsh /usr/X11R6/lib/X11/dotfile/Generator/dotfile.tcl \
/usr/X11R6/lib/X11/dotfile/Generator \
/usr/X11R6/lib/X11/dotfile/bash bytecompile
;;
*)
dpkg-message --title "${0##*/}" --message - <<-EOF
The dotfile-bash module has not been byte-compiled.
You can bytecompile this module later by typing:
dotfile bash bytecompile
EOF
;;
esac
fi
if [ -x /usr/bin/update-menus ] ; then update-menus ; fi
------------------------------------------------------------------------
It is a little more complicated than the original because it now offers some
extra choices about byte-compiling, but it is a good example of what could
be done with dpkg-getconfig and dpkg-message.
The multiple prompting for byte-compilations, which makes so many people
angry, can be easily solved by defining a common configuration variable
CONFIG_DOTFILE_BYTECOMPILE which controls the overall compilation policy and
a variable specific for each package, for example DOTFILE_BASH_BYTECOMPILE,
which controls the byte-compilation of the specific package.
In the above example CONFIG_DOTFILE_BYTECOMPILE is asked with the "--once"
option, that is only if it is not aready defined, and the compilation of
dotfile-bash is asked only if CONFIG_DOTFILE_BYTECOMPILE is "yes" or "ask".
Note also that the script provides its own defaults in case the user selects
a totally automatic installations or just chooses to press enter.
In this way if one doen't want to be bothered by dotfile compilation he can
set CONFIG_DOTFILE_BYTECOMPILE=n once in the config db and he will never be
prompted again. The same thing could be done for emacs add-ons.
In the examples directory of the debian package you can find some other
examples of postinst scripts and configurations files.
The dpkg-getconfig provides also a nice graphical interface if whiptail or
dialog is installed. Unfortunately gdialog can't be used because of its bugs.
Also whiptail and dialog have their problems but they can be worked around.
The graphical interface is choosen automatically by dpkg-getconfig but can
also be controlled by the user with the DIALOG environment variable.
Selecting DIALOG=none will revert to the old plain tty mode.
INSTALLATION POLICY.
My proposal is that the above mechanism is adopted in the official debian
distribution, once it is finished and tested. This has the following
implications:
1) changing the policy about the installation scripts. From a certain
debian version no package should be allowed to interact directly
with the user and all prompting should be done trough an API.
I suggest that by policy all scripts are executed with </dev/null.
2) modifying a lot of install scripts. This seems an almost impossible
task but it could actually be done in a short time and with very
little effort because each developer will have to change only his
few scrips and the changes are usually very simple. The orphaned
packages could be distributed between active developers. Since the
changes are very easy anyone should be able to modify also scripts
written by others.
3) modifying the packages so that they can always provide some
reasonable default for variables which can't be asked ot which
the user don't want or don't know how to ask. Think of an
inexperienced user installing debian. How could he answer to
complex questions about sendmail, ip-masquerading, etc.
It should be possible to installa a standars machine knowing
only few basic things (hostname, ip, dns and mailserver).
This is a long term goal but in my opinion it is important
that a package could be installed totally automatically,
although not configured optimally. The expert user knows
anyway what to change.
4) split each postinst script into a the real postinst and a separate
config script which can be invoked at any time to reconfigure the
package. This is also what Goswin and other have proposed.
6) Assign different prompting levels to various packages and questions
7) Use the config db also for the first step of the installation from
the rescue floppies. It should be possible to load a config file
from a floppy or from the net and find all the values needed for
the base installation.
The first two steps are in my opinion essential. The other require more
work for developers and some discussiona. Thy are also important but not
so urgent.
Comments?
--
Massimo Dal Zotto
+----------------------------------------------------------------------+
| Massimo Dal Zotto email: dz@cs.unitn.it |
| Via Marconi, 141 phone: ++39-0461534251 |
| 38057 Pergine Valsugana (TN) www: http://www.cs.unitn.it/~dz/ |
| Italy pgp: finger dz@tango.cs.unitn.it |
+----------------------------------------------------------------------+
Reply to: