Howdy debian-dpkg! First, please forgive that this email comes from a Debian-unknown, and a Canonical/Ubuntu core developer at that. But I promise I've been around Debian packaging, conffile management, and archive administration for a few years now ;-) BACKGROUND Having read through some of debian-dpkg's mailing list archives, I think the issues around conffile management are quite well understood by this audience. For those lacking background, I have encapsulated this same proposal in a detailed blog post (the relevant bits of which I will duplicate here): * http://blog.dustinkirkland.com/2011/04/dotdee-modern-proposal-for-improving.html As Ubuntu Server Core Developer and Packager, I'm quite keen on improving two problems we encounter frequently, with respect to conffiles: 1. Particularly in modern, massive Debian/Ubuntu deployments (e.g. Cloud, Grid, and Cluster computing), it's no longer humans that are managing individual systems, but rather systems (puppet, chef, cfengine, et al.) managing systems. (Insert your Skynet jokes here...) As such, it's difficult (if not impossible) for a Debian package or a distribution to make configuration changes to another package's conffiles without violating Debian policy -- even when the change might objectively be the "right" or the "best" thing to do, in terms of end user experience and ensuring proper package operation (especially when the given system is *only* ever managed by such a configuration management system). 2. In other cases, one local package has a run-time dependency on a second package on the local system, but requires the package it depends on to be configured in a particular way. Again, if that configuration lives in a conffile owned by the second package, the first package cannot automatically make that configuration change without violating said policy. As an exception, rather than the rule, there are a couple of very mature packages that provide smart frameworks enabling system administrators, packagers, and distributions to configure them all within policy. A good example would be apache2's /etc/apache2 directory, which allows for admins, packagers, and distributions to insert their own configuration modifications as files (or symbolic links) into sourced directories such as /etc/apache2/conf.d, /etc/apache2/mods-available, and /etc/apache2/sites-available. The concept of a ".d" directory in /etc is very well understood in most Linux/UNIX circles, actually. PROPOSAL Here, I'm proposing a tool called "dotdee" that I think would greatly benefit Debian and Debian-derived distributions such as Ubuntu. Its stated goal is to turn any given flat file on your system to a dynamically concatenated flat file generated from a unique ".d" directory dedicated to that particular file. With such a dedicated and well-formed directory, system administrators, Debian packagers, and distributions could conveniently place additional configuration snippets in particular conffile's dedicated ".d" directory. I have written a first prototype of the tool dotdee (snapshot attached), or to view the latest under revision control, see: * http://bazaar.launchpad.net/~kirkland/+junk/dotdee/view/head:/dotdee It's a very simple, straightforward shell script, inspired a bit by Debian's incredibly useful update-alternatives tool. I've only invested a couple of hours into it at this point; just enough to prove the concept and demonstrate its operation. Should we agree upon a complete design or blueprint, I would gladly rewrite it as necessary (in shell, python, or C). The script runs in 3 different modes: 1. sudo dotdee --setup /etc/path/to/some.conf 2. sudo dotdee --update /etc/path/to/some.conf 3. sudo dotdee --undo /etc/path/to/some.conf SETUP MODE First the setup mode takes a flat file as a target. Assuming the file is not already managed by dotdee, a new directory structure is created under /etc/dotdee. In the example above, that would be /etc/dotdee/etc/path/to/some.conf.d. So "/etc/dotdee" is prepended, and ".d" is appended to the path which is to be managed. It's trivial to get back to the name of the managed file by stripping /etc/dotdee from the head, and .d from the tail of the string. Next, the actual managed flat file is moved from /etc/path/to/some.conf to /etc/dotdee/path/to/some.conf.d/50-dpkg-original. Again, this a well-formed path, with "/etc/dotee" prepended, a ".d" appended, and the file itself is renamed to "50-dpkg-original". This is intended to clearly denote that this is the original, base file, as installed by dpkg itself. The number "50" is precisely halfway between "00" and "99", leaving plenty of remove for other file snippets to be placed in ordered positions before and/or after the original file. After this, we run the update function, which will concatenate in alphanumeric order all of the files in /etc/dotdee/etc/path/to/some.conf.d/* and write the output into /etc/dotdee/etc/path/to/some.conf. Finally, the update-alternatives system is used to place a symlink at the original location, /etc/path/to/some.conf, pointing to /etc/dotdee/etc/path/to/some.conf. Additionally, a second, lower-priority alternative is also set, pointing to dpkg's original at /etc/dotdee/path/to/some.conf.d/50-dpkg-original. UPDATE MODE As mentioned above, the update function performs the concatenation immediately, building the targeted path from its dotdee managed collection of snippets. This should be run anytime a file is added, removed, or modified in the dotdee directory for a given managed file. As a convenience, this could easily and automatically be performed by an inotify watch of the /etc/dotdee directory. That, itself, would be a dotdee configuration option. UNDO MODE The undo function is something I added for my own development and debugging while working on the tool, however I quickly realized that it might be an important tool for other system administrators and packagers (think, postrm maintenance scripts!). DPKG INTEGRATION This approach would require some integration with dpkg itself. On package upgrade/installation, dpkg would need to need to detect when the target path of a file it wants to create/update is in fact a symbolic link referencing an /etc/dotdee path. It would need to drill down into that path and place the file it wants to write on top of the 50-dpkg-original file instead. IN PRACTICE So what would this look like in practice? Once integrated with dpkg, I'd like dotdee to be a utility that human system administrators could run to manually turn a generic conffile into a ".d" style configuration directory, such that they could append their own changes to some numbered file in the dotdee directory, avoid the interactive dpkg-conffile-changed prompts. More importantly, I would like one package's postinst maintainer script to be able take another package that it depends upon and turn its conffile into a dotdee managed file, such that it could append or prepend configuration information necessary for proper operation. This, of course, would require significant buy-in from Debian, and entail various appropriate policy updates. In terms of supported configuration file types, dotdee would eventually need some code for handling some particular configuration file types. The current, basic implementation works well for sequentially evaluated file types (like sourced shell scripts), python config parser, and even windows ini file syntax. On the other hand, something like XML would not immediately work in the current dotdee implementation. For that, I've thought a bit about a similar approach, constructing the conffile from a quilt directory of numbered patch files. If you're interested in this approach, I can describe that in more detail, too, and perhaps implement another prototype. COMMENTS? I plan to lead a session on this topic at the Ubuntu Developer Summit in May 2011 in Budapest, and will certainly capture that feedback and summarize it here for this audience. But in the mean time, what do you think? Have you encountered similar problems before? What other approaches have been taken to try and solve this? What parts of this proposal do you think are reasonable? Are any parts completely unreasonable? Are there extensions or changes you propose? Cheers, -- :-Dustin Dustin Kirkland Ubuntu Core Developer
Attachment:
dotdee
Description: Binary data