[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Use and abuse of debconf (revisited)

>From the list archive, at:

Luca De_Vitis <devitis@students.cs.unibo.it> wrote last January:
> But, what I mean with template system to configure, is for first
> configuration or inexpert user configuration. If you are talking
> about expert users that open configuration file and add or remove
> fields and play with configuration, you probably are talking about
> people who do not need for a tool that configure their application,
> and would hit "take may old configuration" when upgrading.

What about the case (eg) where you have a machine, at an ISP, that
performs virtual hosting ...  You construct an interface (sort of a
webmin but hopefully a little better and debconf compatible) for your
customers and clients that allows them to have control only over their
rented peice of the system.  That interface's internals needs to
modify daemon configuration files, BUT you want it to do so IN A WAY
THAT IS COMPATIBLE WITH HAND EDITTING of those configuration files
(including comment preservation please), in case you need to do some
custom setup, above and beyond the out-of-the-box (turn-key) default.

Ideally, (daemon) software upgrades will not only preserve the
configuration settings, but will perform the conffile upgrade as well,
and also ideally, in some "turn-key" commercial applications I can
envision[1], that upgrade can proceed without any operator
intervention.  A great example of a program that does this well is
"fwbuilder"[2], which uses an XSLT sheet to bring all conffiles (which
are XML with accompanying strict DTD) from one version to the next.
(I believe Mac OS-X uses XML for all conffiles) Another interesting
"template with central meta-conffile" based approach may be found at:


The diagram there gives me a few interesting ideas worth discussing.
(... and if you know of previous discussion we may not have read,
 please reply here with a list archives or web site URL!)

 - What if Debian packages, instead of installing the conffiles
   themselves, installed instead conffile templates.  (Perhaps those
   templates can be XML documents?  (much complexity)) "debconf" could
   ask for the fill-in confdata, then the postinst can run the
   template processor to generate the real conffile.  Perhaps not all
   subsystems will require this, since some of them can just use a
   shell snippet in "/etc/default", for instance.

 - Perhaps "dpkg" could handle conffile templates as a special case of
   conffile?  Perhaps this is part of the reconciliation of the
   "package cannot self-modify it's conffiles" quandary?

   o As an operator or consultant, I may want to modify the template,
     but when that package is upgraded, I need to know if a new
     version of that template has been installed.  I don't want to
     lose my changes, and I don't want the daemon to suddenly stop
     working.  In the case of a "turn-key" commercial product, I don't
     want customers to have to hire an expensive consultant every time
     they use an update service.  And what if there's several thousand
     machines out there in need of an update that requires operator
     intervention? It would be really dreamland cool if a template
     merge could be made reliably automatic!

 - What ever happened to the idea of registering "extrafiles" or
   "autofiles" with dpkg?

 - This idea seems to imply a standardized templating system.  eperl?
   template toolkit?  autogen? m4? cpp? Imakefile + xmkmf?

 - Maybe this would implement a total Rube Goldberg contraption that
   would prove to be quite fragile -- easy to break and difficult to
   dpkg manage.  Or perhaps it would be too complex for most people to

 - Not all system software will require a setup this complex.

 - We already have similar systems in Debian, but from upstream

       o Sendmail

       o autoconf + automake


In the original debian-devel thread abour "Use and Abuse of debconf",
at one point, "libconfmodule-perl" was mentioned.  ConfModule.pm
operates, presently, by working only on a chunk of the conffile that
is delimitted by special comment markers.  It's quick and dirty and
works fine for some things, but is not quite what I want.

What if the ConfModule.pm was expanded upon, and turned into a more
complete framework?  It could conceivably, in my mind, provide objects
for each major type of configuration file, with reader and writer
methods.  That could eliminate the need for conffile templates, since
the native conffiles would be manipulated directly[3].

A set of shell script functions could be created also, to use from
maintainer scripts.  They would be implemented by creating some
command line tool perl programs that use the ConfModule.pm framework,
similar to "debhelper".

This framework could build an abstract syntax tree of the conffile
it's reading; a DOM or DOM-like representation might be ideal.  It
would then be capable of reading the conffile's native format
(httpd.conf, for instance) into memory, manipulating it, then writing
it back out, like the Lisp reader/writer.

 - Values filled in from the "debconf" managed conffdata should be
   clearly marked in the output, for the benefit of the advanced
   system administrator or consultant who is performing customized
   hand configuration.

 - When the conffile is hand-editted and one of those values changed,
   the next time that the debconfiscated configuration machinery reads
   that conffile in, some conflict resolution must take place and the
   debconf database reconciled with the actual conffile.  Generally,
   the value from the conffile should be put into the debconf
   database, overwriting what's there, prior to presentation by the
   management user interface.  There may be cases where that is not
   the right thing to do though.  What if the configuration has been
   hand-broken, and the operator wants to restore it via "dpkg

 - What if an overseer daemon, perhaps one that utilizes "fam", kept
   an eye on certain conffiles and listened for communications from
   "debconf", and performed reconfiguration initiation at the
   appropriate times?

 - When a conffdata/conffile schema migration is necessary, there must
   be a confmodule subclass for the old format and one for the new,
   and a script that can read in the old format and write it back out
   in the new format; and/or merge a new version with an old.  What I
   have in mind is something like what is done by the debianized
   PostgreSQL upgrade setup.

 - Something must be done in the case where a new confdata item has
   been added to the daemon's schema, but the value to be filled in
   cannot be defaulted or computed automaticly.  Potentially there
   will be situations where there's no way around a requirement for
   operator interaction during an upgrade.

 - There is also the case where one package must perform some
   configuration of another package's software.  For example,
   installing a chroot BIND v9 "named" requires that the syslogd be
   informed of a log socket inside the "named" chroot.  Note that
   "sysklogd" does not provide an API for this task... (but read on)

    + Debian Policy states that no package may modify another
      package's conffile, and that any package that may need to be
      configured by another package must provide an API for doing so.
      But that requirement is not always forseen.  (It occurs to me
      that the designers of "debconf" must have already thought of

      o The thing is, once the conffile has been modified, even
        automaticly, it's md5sum changes, and during future upgrades,
        this will cause "dpkg" to prompt whether it should replace or
        keep that conffile.

      o It would be better if that conffile is not changed, but
        confdata is instead updated via "debconf", which provides a
        general purpose and common API.

      o In the case of "sysklogd" above, what is required is a
        "debconf" slot to hold the value of an "OPTS_syslogd"
        variable.  That value then must be written, during "postinst
        configure", to a "/etc/default/sysklogd" shell snippet, for
        the init.d script to source.  What remains is a way for the
        other program to append to that value in a portable fashion.

 - That is all very ad-hoc... it all suggests to me that a standard
   intermediary form would simplify, especially, the schema migration
   and fully automated upgrade.  That implies XML and XSLT, where
   those who have gone before us have developed and tested such
   systems already (libfwbuilder).

The subthread beginning at:
... brings up an important point -- migration of a service from one
machine to another is more difficult if that service's configuration
data is kept in a "registry" database.  "debconf" is not meant to be a
full on registry or directory service, but is apparently meant for use
by maintainer scripts only; that is, for initial and minimal

Native XML conffiles

An eventual migration to XML as the native conffile format for all
daemons would be awesome.  (Isn't that what Apple has done with Mac
OS-X?  They use LDAP too, right?  I cannot afford to purchase it and
find out for myself -- that is why I must ask.)  A co-ordinated team
of Good Programmers could patch the daemons and #ifdef around the new
code, perhaps, leaving the old confcode in place for users who want to
do it the old way.  This is really the best long-term solution we have
available, yous must agree!

 - The library that supports native XML conffiles (or directory
   service based confdata) should also provide the mechanism for
   confdata reload signalling.  In the case of the directory service
   based system, the DS ought to notify it's registered clients
   (gconf, PostgreSQL "LISTEN" and "NOTIFY" triggers).

 - There must be a way for it to check not only the syntax, but the
   semantic consistency of the updated conffile, prior to actually
   reloading it, to avoid crashing the software and interrupting
   service.  This point ties in with the one about "conflict
   resolution", above.

 - That implies transaction support or an "RMON polka", so that there
   is never an incomplete and inconsistent complex-confdata set pushed
   to a peice of software.  (You can't fill out only part of a struct
   and expect the software to reload, for example.)

Other Issues

We must also consider conffdata scoping -- I mean "site-wide",
"machine-wide" scoping here.  It should be possible for the box to
look up site-wide configuration information from a central source when
one is available. (cf: Active Directory; LDAP; debconf; DHCP)

Another important issue is standardization and registration of
(deb)confdata keys (and menu headings, docu and wm).  For instance, I
don't think I should have to tell "libnss-ldap" (let's pick on him
again) the base DN of my LDAP server -- it should have the ability to
look that up and offer a default (or list of them?) somehow.  It's not
a simple problem, and designing a good solution will require much
study and discussion.

 - When a meta-package that ties together sundry subsystems --
   "sourceforge" is a prime example -- is installed, there ought to be
   a meta-debconfiscation of some kind, to try and "druidize" the
   installation and configuration of those interacting subsystems.

I believe that these issues have all been faced by other project
groups in the computer industry.  Whenever possible, we should strive
to learn from their solutions and mistakes, and to try hard to be
compatible and interoperative with them.  This probably means
utilizing standards such as "SLP", "LDAP", "SNMP", "WBEM", and "CIM",
for instance, and using already written libraries when they suite our

 - It would occasionally be nice to have a way to remotely reconfigure
   a (Debian based) machine, wouldn't it?  What if rather than having
   to log onto each machine in turn to affect a same or similar
   configuration modification, you instead made the change in one
   place -- perhaps in LDAP -- and the machines then reconfigured

Has anyone got notes and lists of URL's relevant to this general topic
that I may have?  Will you put them up and reply here with a URL

Also see:

[1] And don't you DARE tell me "that's beyond the scope of the Debian
    project".  What BS that would be!  Many of us will, given the
    opportunity, utilize the Debian "kit" to provide solutions to our
    customers and clients.  I did not say "proprietary", I said
    "commercial". <URL:http://www.netule.com/>, <URL:http://www.netule.org/>

[2] See <URL:http://www.crocodile.org/lord/XML_XSLT_Configuration.html>
    There was also some work done wrt data structure / object schema
    migration, that I recall, by the CLOS people; see the Sonya Keene
    book on CLOS (IIRC) for some examples of how CLOS classes can be
    programmed to manage schema migration in a "lazy" (when used)

[3] This still leaves us with the interactivity during upgrades
    involving conffile schema updates problem that developers of
    "turn-key" commercial solutions must face!  It would be most
    awesome to have a generally useable solution in Debian + apt-get +
    debconf.  Corel faced this, IIRC, and that is what impelled them
    to invent that "debwrap" kludge hammer.

To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Reply to: