[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

dotdee: a proposal for improving conffile management in Debian

Howdy debian-dpkg!

First, please forgive that this email comes from a Debian-unknown, and
a Canonical/Ubuntu core developer at that.  But I promise I've been
around Debian packaging, conffile management, and archive
administration for a few years now ;-)


Having read through some of debian-dpkg's mailing list archives, I
think the issues around conffile management are quite well understood
by this audience.  For those lacking background, I have encapsulated
this same proposal in a detailed blog post (the relevant bits of which
I will duplicate here):
 * http://blog.dustinkirkland.com/2011/04/dotdee-modern-proposal-for-improving.html

As Ubuntu Server Core Developer and Packager, I'm quite keen on
improving two problems we encounter frequently, with respect to

  1. Particularly in modern, massive Debian/Ubuntu deployments (e.g.
Cloud, Grid, and Cluster computing), it's no longer humans that are
managing individual systems, but rather systems (puppet, chef,
cfengine, et al.) managing systems.  (Insert your Skynet jokes
here...)  As such, it's difficult (if not impossible) for a Debian
package or a distribution to make configuration changes to another
package's conffiles without violating Debian policy -- even when the
change might objectively be the "right" or the "best" thing to do, in
terms of end user experience and ensuring proper package operation
(especially when the given system is *only* ever managed by such a
configuration management system).

  2. In other cases, one local package has a run-time dependency on a
second package on the local system, but requires the package it
depends on to be configured in a particular way.  Again, if that
configuration lives in a conffile owned by the second package, the
first package cannot automatically make that configuration change
without violating said policy.

As an exception, rather than the rule, there are a couple of very
mature packages that provide smart frameworks enabling system
administrators, packagers, and distributions to configure them all
within policy.

A good example would be apache2's /etc/apache2 directory, which allows
for admins, packagers, and distributions to insert their own
configuration modifications as files (or symbolic links) into sourced
directories such as /etc/apache2/conf.d, /etc/apache2/mods-available,
and /etc/apache2/sites-available.

The concept of a ".d" directory in /etc is very well understood in
most Linux/UNIX circles, actually.


Here, I'm proposing a tool called "dotdee" that I think would greatly
benefit Debian and Debian-derived distributions such as Ubuntu.  Its
stated goal is to turn any given flat file on your system to a
dynamically concatenated flat file generated from a unique ".d"
directory dedicated to that particular file.  With such a dedicated
and well-formed directory, system administrators, Debian packagers,
and distributions could conveniently place additional configuration
snippets in particular conffile's dedicated ".d" directory.

I have written a first prototype of the tool dotdee (snapshot
attached), or to view the latest under revision control, see:
 * http://bazaar.launchpad.net/~kirkland/+junk/dotdee/view/head:/dotdee

It's a very simple, straightforward shell script, inspired a bit by
Debian's incredibly useful update-alternatives tool.  I've only
invested a couple of hours into it at this point; just enough to prove
the concept and demonstrate its operation.  Should we agree upon a
complete design or blueprint, I would gladly rewrite it as necessary
(in shell, python, or C).

The script runs in 3 different modes:

   1. sudo dotdee --setup /etc/path/to/some.conf
   2. sudo dotdee --update /etc/path/to/some.conf
   3. sudo dotdee --undo /etc/path/to/some.conf


First the setup mode takes a flat file as a target.  Assuming the file
is not already managed by dotdee, a new directory structure is created
under /etc/dotdee.  In the example above, that would be
/etc/dotdee/etc/path/to/some.conf.d.  So "/etc/dotdee" is prepended,
and ".d" is appended to the path which is to be managed.  It's trivial
to get back to the name of the managed file by stripping /etc/dotdee
from the head, and .d from the tail of the string.

Next, the actual managed flat file is moved from
/etc/path/to/some.conf to
/etc/dotdee/path/to/some.conf.d/50-dpkg-original.  Again, this a
well-formed path, with "/etc/dotee" prepended, a ".d" appended, and
the file itself is renamed to "50-dpkg-original".  This is intended to
clearly denote that this is the original, base file, as installed by
dpkg itself.  The number "50" is precisely halfway between "00" and
"99", leaving plenty of remove for other file snippets to be placed in
ordered positions before and/or after the original file.

After this, we run the update function, which will concatenate in
alphanumeric order all of the files in
/etc/dotdee/etc/path/to/some.conf.d/* and write the output into

Finally, the update-alternatives system is used to place a symlink at
the original location, /etc/path/to/some.conf, pointing to
/etc/dotdee/etc/path/to/some.conf.  Additionally, a second,
lower-priority alternative is also set, pointing to dpkg's original at


As mentioned above, the update function performs the concatenation
immediately, building the targeted path from its dotdee managed
collection of snippets.  This should be run anytime a file is added,
removed, or modified in the dotdee directory for a given managed file.
 As a convenience, this could easily and automatically be performed by
an inotify watch of the /etc/dotdee directory.  That, itself, would be
a dotdee configuration option.


The undo function is something I added for my own development and
debugging while working on the tool, however I quickly realized that
it might be an important tool for other system administrators and
packagers (think, postrm maintenance scripts!).


This approach would require some integration with dpkg itself.  On
package upgrade/installation, dpkg would need to need to detect when
the target path of a file it wants to create/update is in fact a
symbolic link referencing an /etc/dotdee path.  It would need to drill
down into that path and place the file it wants to write on top of the
50-dpkg-original file instead.


So what would this look like in practice?

Once integrated with dpkg, I'd like dotdee to be a utility that human
system administrators could run to manually turn a generic conffile
into a ".d" style configuration directory, such that they could append
their own changes to some numbered file in the dotdee directory, avoid
the interactive dpkg-conffile-changed prompts.

More importantly, I would like one package's postinst maintainer
script to be able take another package that it depends upon and turn
its conffile into a dotdee managed file, such that it could append or
prepend configuration information necessary for proper operation.
This, of course, would require significant buy-in from Debian, and
entail various appropriate policy updates.

In terms of supported configuration file types, dotdee would
eventually need some code for handling some particular configuration
file types.  The current, basic implementation works well for
sequentially evaluated file types (like sourced shell scripts), python
config parser, and even windows ini file syntax.  On the other hand,
something like XML would not immediately work in the current dotdee
implementation.  For that, I've thought a bit about a similar
approach, constructing the conffile from a quilt directory of numbered
patch files.  If you're interested in this approach, I can describe
that in more detail, too, and perhaps implement another prototype.


I plan to lead a session on this topic at the Ubuntu Developer Summit
in May 2011 in Budapest, and will certainly capture that feedback and
summarize it here for this audience.

But in the mean time, what do you think?  Have you encountered similar
problems before?  What other approaches have been taken to try and
solve this?  What parts of this proposal do you think are reasonable?
Are any parts completely unreasonable?  Are there extensions or
changes you propose?


Dustin Kirkland
Ubuntu Core Developer

Attachment: dotdee
Description: Binary data

Reply to: