Dear list members
-----------------
Some months ago I asked the list about my idea to package all sorts of
system customization (programs, scripts, configuration, installed
packages) using the Debian packaging system [1]. I have implemented this
idea and tested it -- now I thought it might be a good idea to share the
results for everyone interested in setting up a similar system.
My use of Debian packaging to distribute system customization
-------------------------------------------------------------
I started with [2] and packaged some applications which are a
"classical" case for packaging: They are built using the normal "make"
and "make install" invocations. Such programs only make a small fraction
of my system customization which is why I quickly started to package
something different: A large collection of HTML documentation was to be
packaged. The problem with "debuild" (which was the recommended
packaging program when following [2]) was that it checked all files with
"lintian" but as the documentation consisted of about 60,000 files this
took too much time. Therefore I used another technique to package
architecture-independent files: I directly created .deb files out of a
prepared directory structure. These packages are not really ready to be
uploaded by a Debian Developer (missing copyright files and such) but
they are way faster to create. In the end, I created six directories for
different types of packaging:
arch
For binary packages which should contain different files
depending on the architecture (i386 or amd64). Example: nanozip
debuild
Packages to be compiled with debuild. Of course, the binaries
vary on different architectures but it is easy to compile first
on native amd64 and then in an i386 chroot. Example: dhex
foreign
Packages in .deb form from various sources. Example: virtualgl.
meta
Packages which only consist of dependencies.
Example: mdvl-base-minimal (depends on what I consider a minimal
useful Debian installation for desktop and development usage)
raw
For packages which consist of configuration files and other
architecture independent data. These are directly compressed to
.deb files. Example: mdvl-ial (HTML documentation collection)
special
A few things that do not fit into any other category. MDPC has
to be edited in order to do anything with "special" packages.
Example: dxx-rebirth (a game for which I want to compile
mutliple binaries for the same architecture).
In the course of setting up the packaging system I decided to use
reprepro to manage the repository: It could be used with simple commands
and there were good usage examples available. Also, I created several
scripts to automate the process, start applications I used in
conjunction with the packaging and setup a suitable chroot to
cross-compile for i386.
At first, these scripts were loosely connected and could be invoked via
several Makefiles distributed over various directories. To further
improve the situation I started to merge all scripts into one central
interface I called MDPC (MDVL Debian Packaging Control Script). This
script can be used to newly create packages, update them or to enter a
specially created chroot and all the other functions which were
previously managed by separate scripts. With MDPC it is not necessary to
remember any specific commands of the Debian Packaging system (like
debuild, dpkg-deb or reprepro) -- all commands are simply invoked with
$ mdpc and suitable arguments.
In case anybody wants to implement a similar system, I have attached
MDPC [3] to this mail. To use it, it is required to generate
archive-signing keys, change the configuration and probably adjust some
settings I made directly in the source code (like the server's
IP-address -- 192.168.1.16 if you want to grep for it). Also, many
"Copyright (c) 2013 Ma_Sys.ma" lines are hard-coded because MDPC was
developed for a specific need. Therefore MDPC can not serve as a
solution to setup an own Debian packaging infrastructure with
zero-effort -- it's use for the community should rather be the source
code which shows how to generate (minimal and simple) Debian packages.
Advantages of using Debian packaging to backup system customization
-------------------------------------------------------------------
* The system can be re-created by installing a single "root" package
which pulls in the whole system as dependencies on a minimal Debian
installation.
* It is not required to backup machines by copying their whole HDD. The
repository, user data and a Debian netinst CD are all that is
required.
* Systems' configuration and software (selections) can be upgraded via
the normal "aptitude update && aptitude full-upgrade".
* Customization integrates well with the rest of the system. Everything
is at it's standard location.
* System-specific configuration can be managed by if-conditions in
installation and configuration scripts. Also, there can be specific
packages which are only installed on special systems (e.g.
mdvl-special-client for all clients, mdvl-special-server for the main
server)
* A management scripts automates most interactions with the packaging
system: I.e. Commands one might foreget are usually automated. Adding
new packages is rather simple.
* The "source-code" of the repository is a great documentation of /all/
system configuration which needs to be installed on systems. Unlike
package-lists it can never be outdated because all systems are
upgraded from it and as long as all changes to configuration files
are recorded in the packagaging system it is not necessary to keep
track of them by different means (e.g. lists of changed configuration
files which often become outdated).
* Ideally, no old/unused/irrelevant configuration files are backed up
and distributed which can often happen when updating to newer
releases and doing "full" backups of e.g. /etc.
* Packages outside the repositories can be seamlessly integrated once
an own repository was set up.
Disadvantages
-------------
* Upgrading to a new Debian version is difficult. It might take very
long to resolve the dependencies and the
package-management-utilities' suggestions are not always the way to
go. Sometimes, packages have to be temporarily removed and installed
afterwards again.
This is the mayor and really annoying disadvantage. Especially when
using aptitude, it can be a challenging game to satisfy the
dependencies eventhough a test in a VM has shown that the
dependencies /can/ be resolved. apt-get on the other hand often
forces some packages to be removed which can be installed afterwards
again.
* When packages are changed, the changing person has to keep oddities
of the package-management in mind: E.g. when changing a package with
symlinks problems might arise that require the package to be removed
with dpkg --force-depends --purg PACKAGE and installed back with
aptitude -f afterwards.
* Moving files between packages is a minor issue: The changed packages
need to get a suitable "Conflicts:" field -- otherwise the upgrade
fails.
* If a single user is required to remove a package which was part of
the "root" package's dependencies the package-management-software
suggests the user to remove almost the whole system because the root
package is removed and all "automatically installed packgages"
(dependencies of the root package) are removed with it. However, it
is also possible to mark the packages as manually installed in such
situations.
Evaluation
----------
I am proud of having all system customization in packages: It simplifies
things a lot and whenever I am at a machine which is not the main system
a simple package upgrade installs all the newest configuration files and
programs.
But I would not recommend such a setup for everyone: It does probably
not make sense when there are very few machines (eventhough with four
computers it already pays off in my opinion). Also, if each user does
his/her own heavy system customization the package-based setup becomes
hard to manage because to simplyfy installation of a new client there
are many dependencies on a single "root" package.
Finally, the setup of such a system is a lot of work. Even though my
script might help one or the other to get started it sometimes requires
accessing the packaging system directly and it was not designed to be
sort of "user-friendly". Invalid actions might just yield no output
without further notice. For most users who do not regularily setup new
systems (e.g. in virtual machines) the best solution is a normal
HDD-backup by either rsyncing (or something more advanced of that sort)
or even imaging which has the advantage of being easily and safely
restored without any additional thoughts on a complex packaging system.
For those whith less customization, just backing up /etc and /home might
be enough.
For those interested in learning about their system, tinkering with a
lot of scripts or those who need to manage many (might also be virtual)
computers or frequently setup new systems I can definitly recommend a
solution based on or built around Debian packaging.
The "work once, enjoy afterwards" ideal is almost implemented but still
there can be some work once the system is up (especially upon release
upgrade / a test with at least one VM is absolutely required).
Resources
---------
[1] Original thread on debian-user
http://lists.debian.org/debian-user/2013/01/msg01060.html
[2] Introduction to Debian Packaging
https://wiki.debian.org/IntroDebianPackaging
[3] Ma_Sys.ma Development Linux Debian Packaging Control Script
Attachment "mdpc_1.0.1.7.tar.gz" (19 KiB)
I hope that this mail was useful to read for at least some members of
the community and that the code of MDPC is also useful for someone apart
from myself.
Thank you for reading that (hopefully not too longish) story.
Linux-Fan
--
http://masysma.ohost.de/
Attachment:
mdpc_1.0.1.7.tar.gz
Description: GNU Zip compressed data
Attachment:
signature.asc
Description: OpenPGP digital signature