Use and abuse of debconf (revisited)
>From the list archive, at:
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200101/msg01834.html>
Luca De_Vitis <devitis@students.cs.unibo.it> wrote last January:
>
> But, what I mean with template system to configure, is for first
> configuration or inexpert user configuration. If you are talking
> about expert users that open configuration file and add or remove
> fields and play with configuration, you probably are talking about
> people who do not need for a tool that configure their application,
> and would hit "take may old configuration" when upgrading.
What about the case (eg) where you have a machine, at an ISP, that
performs virtual hosting ... You construct an interface (sort of a
webmin but hopefully a little better and debconf compatible) for your
customers and clients that allows them to have control only over their
rented peice of the system. That interface's internals needs to
modify daemon configuration files, BUT you want it to do so IN A WAY
THAT IS COMPATIBLE WITH HAND EDITTING of those configuration files
(including comment preservation please), in case you need to do some
custom setup, above and beyond the out-of-the-box (turn-key) default.
Ideally, (daemon) software upgrades will not only preserve the
configuration settings, but will perform the conffile upgrade as well,
and also ideally, in some "turn-key" commercial applications I can
envision[1], that upgrade can proceed without any operator
intervention. A great example of a program that does this well is
"fwbuilder"[2], which uses an XSLT sheet to bring all conffiles (which
are XML with accompanying strict DTD) from one version to the next.
(I believe Mac OS-X uses XML for all conffiles) Another interesting
"template with central meta-conffile" based approach may be found at:
<URL:http://www.e-smith.org/architecture/>
The diagram there gives me a few interesting ideas worth discussing.
(... and if you know of previous discussion we may not have read,
please reply here with a list archives or web site URL!)
- What if Debian packages, instead of installing the conffiles
themselves, installed instead conffile templates. (Perhaps those
templates can be XML documents? (much complexity)) "debconf" could
ask for the fill-in confdata, then the postinst can run the
template processor to generate the real conffile. Perhaps not all
subsystems will require this, since some of them can just use a
shell snippet in "/etc/default", for instance.
- Perhaps "dpkg" could handle conffile templates as a special case of
conffile? Perhaps this is part of the reconciliation of the
"package cannot self-modify it's conffiles" quandary?
o As an operator or consultant, I may want to modify the template,
but when that package is upgraded, I need to know if a new
version of that template has been installed. I don't want to
lose my changes, and I don't want the daemon to suddenly stop
working. In the case of a "turn-key" commercial product, I don't
want customers to have to hire an expensive consultant every time
they use an update service. And what if there's several thousand
machines out there in need of an update that requires operator
intervention? It would be really dreamland cool if a template
merge could be made reliably automatic!
- What ever happened to the idea of registering "extrafiles" or
"autofiles" with dpkg?
- This idea seems to imply a standardized templating system. eperl?
template toolkit? autogen? m4? cpp? Imakefile + xmkmf?
- Maybe this would implement a total Rube Goldberg contraption that
would prove to be quite fragile -- easy to break and difficult to
dpkg manage. Or perhaps it would be too complex for most people to
use?
- Not all system software will require a setup this complex.
- We already have similar systems in Debian, but from upstream
sources.
o Sendmail
o autoconf + automake
ConfModule.pm
=============
In the original debian-devel thread abour "Use and Abuse of debconf",
at one point, "libconfmodule-perl" was mentioned. ConfModule.pm
operates, presently, by working only on a chunk of the conffile that
is delimitted by special comment markers. It's quick and dirty and
works fine for some things, but is not quite what I want.
What if the ConfModule.pm was expanded upon, and turned into a more
complete framework? It could conceivably, in my mind, provide objects
for each major type of configuration file, with reader and writer
methods. That could eliminate the need for conffile templates, since
the native conffiles would be manipulated directly[3].
A set of shell script functions could be created also, to use from
maintainer scripts. They would be implemented by creating some
command line tool perl programs that use the ConfModule.pm framework,
similar to "debhelper".
This framework could build an abstract syntax tree of the conffile
it's reading; a DOM or DOM-like representation might be ideal. It
would then be capable of reading the conffile's native format
(httpd.conf, for instance) into memory, manipulating it, then writing
it back out, like the Lisp reader/writer.
- Values filled in from the "debconf" managed conffdata should be
clearly marked in the output, for the benefit of the advanced
system administrator or consultant who is performing customized
hand configuration.
- When the conffile is hand-editted and one of those values changed,
the next time that the debconfiscated configuration machinery reads
that conffile in, some conflict resolution must take place and the
debconf database reconciled with the actual conffile. Generally,
the value from the conffile should be put into the debconf
database, overwriting what's there, prior to presentation by the
management user interface. There may be cases where that is not
the right thing to do though. What if the configuration has been
hand-broken, and the operator wants to restore it via "dpkg
reconfigure"?
- What if an overseer daemon, perhaps one that utilizes "fam", kept
an eye on certain conffiles and listened for communications from
"debconf", and performed reconfiguration initiation at the
appropriate times?
- When a conffdata/conffile schema migration is necessary, there must
be a confmodule subclass for the old format and one for the new,
and a script that can read in the old format and write it back out
in the new format; and/or merge a new version with an old. What I
have in mind is something like what is done by the debianized
PostgreSQL upgrade setup.
- Something must be done in the case where a new confdata item has
been added to the daemon's schema, but the value to be filled in
cannot be defaulted or computed automaticly. Potentially there
will be situations where there's no way around a requirement for
operator interaction during an upgrade.
- There is also the case where one package must perform some
configuration of another package's software. For example,
installing a chroot BIND v9 "named" requires that the syslogd be
informed of a log socket inside the "named" chroot. Note that
"sysklogd" does not provide an API for this task... (but read on)
+ Debian Policy states that no package may modify another
package's conffile, and that any package that may need to be
configured by another package must provide an API for doing so.
But that requirement is not always forseen. (It occurs to me
that the designers of "debconf" must have already thought of
this...)
o The thing is, once the conffile has been modified, even
automaticly, it's md5sum changes, and during future upgrades,
this will cause "dpkg" to prompt whether it should replace or
keep that conffile.
o It would be better if that conffile is not changed, but
confdata is instead updated via "debconf", which provides a
general purpose and common API.
o In the case of "sysklogd" above, what is required is a
"debconf" slot to hold the value of an "OPTS_syslogd"
variable. That value then must be written, during "postinst
configure", to a "/etc/default/sysklogd" shell snippet, for
the init.d script to source. What remains is a way for the
other program to append to that value in a portable fashion.
- That is all very ad-hoc... it all suggests to me that a standard
intermediary form would simplify, especially, the schema migration
and fully automated upgrade. That implies XML and XSLT, where
those who have gone before us have developed and tested such
systems already (libfwbuilder).
The subthread beginning at:
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200101/msg01691.html>
... brings up an important point -- migration of a service from one
machine to another is more difficult if that service's configuration
data is kept in a "registry" database. "debconf" is not meant to be a
full on registry or directory service, but is apparently meant for use
by maintainer scripts only; that is, for initial and minimal
configuration.
Native XML conffiles
====================
An eventual migration to XML as the native conffile format for all
daemons would be awesome. (Isn't that what Apple has done with Mac
OS-X? They use LDAP too, right? I cannot afford to purchase it and
find out for myself -- that is why I must ask.) A co-ordinated team
of Good Programmers could patch the daemons and #ifdef around the new
code, perhaps, leaving the old confcode in place for users who want to
do it the old way. This is really the best long-term solution we have
available, yous must agree!
- The library that supports native XML conffiles (or directory
service based confdata) should also provide the mechanism for
confdata reload signalling. In the case of the directory service
based system, the DS ought to notify it's registered clients
(gconf, PostgreSQL "LISTEN" and "NOTIFY" triggers).
- There must be a way for it to check not only the syntax, but the
semantic consistency of the updated conffile, prior to actually
reloading it, to avoid crashing the software and interrupting
service. This point ties in with the one about "conflict
resolution", above.
- That implies transaction support or an "RMON polka", so that there
is never an incomplete and inconsistent complex-confdata set pushed
to a peice of software. (You can't fill out only part of a struct
and expect the software to reload, for example.)
Other Issues
============
We must also consider conffdata scoping -- I mean "site-wide",
"machine-wide" scoping here. It should be possible for the box to
look up site-wide configuration information from a central source when
one is available. (cf: Active Directory; LDAP; debconf; DHCP)
Another important issue is standardization and registration of
(deb)confdata keys (and menu headings, docu and wm). For instance, I
don't think I should have to tell "libnss-ldap" (let's pick on him
again) the base DN of my LDAP server -- it should have the ability to
look that up and offer a default (or list of them?) somehow. It's not
a simple problem, and designing a good solution will require much
study and discussion.
- When a meta-package that ties together sundry subsystems --
"sourceforge" is a prime example -- is installed, there ought to be
a meta-debconfiscation of some kind, to try and "druidize" the
installation and configuration of those interacting subsystems.
I believe that these issues have all been faced by other project
groups in the computer industry. Whenever possible, we should strive
to learn from their solutions and mistakes, and to try hard to be
compatible and interoperative with them. This probably means
utilizing standards such as "SLP", "LDAP", "SNMP", "WBEM", and "CIM",
for instance, and using already written libraries when they suite our
needs.
- It would occasionally be nice to have a way to remotely reconfigure
a (Debian based) machine, wouldn't it? What if rather than having
to log onto each machine in turn to affect a same or similar
configuration modification, you instead made the change in one
place -- perhaps in LDAP -- and the machines then reconfigured
themselves?
Has anyone got notes and lists of URL's relevant to this general topic
that I may have? Will you put them up and reply here with a URL
please?
------------------------------------------------------------------------
Also see:
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200101/msg01640.html>
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200101/msg01852.html>
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200101/msg02021.html>
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200102/msg00973.html>
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200102/msg01090.html>
<URL:http://lists.debian.org/debian-devel/2001/debian-devel-200102/msg01142.html>
<URL:http://lists.ximian.com/archives/public/setup-tool-hackers/>
<URL:http://lists.debian.org/debian-admintool/>
Footnotes:
[1] And don't you DARE tell me "that's beyond the scope of the Debian
project". What BS that would be! Many of us will, given the
opportunity, utilize the Debian "kit" to provide solutions to our
customers and clients. I did not say "proprietary", I said
"commercial". <URL:http://www.netule.com/>, <URL:http://www.netule.org/>
[2] See <URL:http://www.crocodile.org/lord/XML_XSLT_Configuration.html>
There was also some work done wrt data structure / object schema
migration, that I recall, by the CLOS people; see the Sonya Keene
book on CLOS (IIRC) for some examples of how CLOS classes can be
programmed to manage schema migration in a "lazy" (when used)
fashion.
[3] This still leaves us with the interactivity during upgrades
involving conffile schema updates problem that developers of
"turn-key" commercial solutions must face! It would be most
awesome to have a generally useable solution in Debian + apt-get +
debconf. Corel faced this, IIRC, and that is what impelled them
to invent that "debwrap" kludge hammer.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Reply to: