[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Grip, components and Sections

I'm considering a useful change in how Grip works at the repository
level. Currently, a brute force method is used to put any package with
a name ending in '-dev' into a 'dev' component and ending in '-doc'
into the 'doc' component. This will be retained, such that ordinary
users have a sources line of:

deb http://www.emdebian.org/grip squeeze main

Those who want to build packages on Grip will then add:

deb http://www.emdebian.org/grip squeeze dev
deb-src http://www.emdebian.org/grip squeeze main
deb-src http://www.emdebian.org/grip squeeze dev

(doc is largely redundant currently, as -doc packages themselves are
nearly empty but this change puts tools handling documentation into the
doc component as well as the -doc packages themselves.)

This allows apt to download the Packages file that allows installation
of the -dev packages without bloating the 'main' Packages file with
the details of those -dev packages as well as the sources need to
build the packages themselves.

I'm looking at extending this to make it easier to handle a larger
number of packages as well as making the relocation easier to

It's a simple change in the code, instead of always setting '-C main'
when calling reprepro, I'd call '-C $section' where $section holds the
name of the Section: listed for that package in the Debian Packages
file (this uses the overrides set by ftpmaster.debian.org, not
necessarily the value encoded in the debian/control file within the

This does mean that some packages that are currently in 'main' for Grip
will move into 'dev' or 'doc' - check the Section listings for more


Each Grip repository could then choose which components to activate -
as soon as reprepro is configured to use the component, packages will
be redirected into it. Name matching will be case-insensitive

One exception, both the 'devel' and 'libdevel' Sections will end up in
the 'dev' component.

Some Section:s already listed for Grip include:
admin base comm database debian-installer devel doc editors embedded
fonts games gnome graphics interpreters libdevel libs mail math misc
net oldlibs otherosfs perl python ruby shells sound tex text utils vcs
video web x11 xfce

Of those, xfce|gnome could be useful components, debian-installer is a
mystery and probably needs to be fixed.

deb http://www.emdebian.org/grip squeeze main
deb http://www.emdebian.org/grip squeeze xfce


deb http://www.emdebian.org/grip squeeze main
deb http://www.emdebian.org/grip squeeze gnome


The Debian archive maintainers provide the authoritative list of
sections. At present, they are: admin, cli-mono, comm, database, devel,
debug, doc, editors, electronics, embedded, fonts, games, gnome,
graphics, gnu-r, gnustep, hamradio, haskell, httpd, interpreters, java,
kde, kernel, libs, libdevel, lisp, localization, mail, math, misc, net,
news, ocaml, oldlibs, otherosfs, perl, php, python, ruby, science,
shells, sound, tex, text, utils, vcs, video, web, x11, xfce, zope.
(Policy 2.4)

So 'kde' could be handled in the same way as xfce or gnome.

The important point here is that only the components configured by the
admin of that individual repository will be managed in this way, such
that each repository can adjust the relative sizes of each component.
In effect, a Section: becomes a discrete component within the
distribution that can be added or removed by users. Once removed,
aptitude will list packages installed from that component as
"obsolete", allowing easy removal. Which Sections are in use needs to
be clearly explained when setting out the apt sources lines to be used
with the repository.

This isn't about creating 20 components and having users specify some
combination of 12 different ones that may or may not end up as fully
installable. It is meant to allow repository admins to cherry-pick a
handful of components and reliably apportion packages into those
components whilst preserving the core packages in main.

The emphasis is to provide components where the package sets are
roughly equivalent, whereby xfce|kde|gnome|foo is a good example. It is
also needed to ensure that packages like 'automake' reliably end up in
'dev' without needing complex regular expression matching and without
dumping hundreds of development tool packages into 'main'. If the admin
does not configure the component, the Section: variable is ignored and
'main' is used.

I'm undecided whether to enable 'xfce' for www.emdebian.org/grip or
whether to leave xfce as the de facto default and offer other
environments as their own components. I'm also undecided whether Grip
ever wants a complete KDE or complete GNOME environment or whether we
offer some GNOME/KDE packages to complement the more slimline apps from
other environments.

I will be enabling the 'dev' and 'doc' changes to keep things like
automake out of Grip/main. I'll also enable 'gnome' and 'kde' at

Users should not be expected to need more than three or four components
to get all the binary packages they need (excluding deb-src lines). It
is also part of the drive to encourage others to create their own Grip
repositories with their own configurations and their own package lists.
Just because we have www.emdebian.org/grip does not mean that all Grip
installations need to come from that one machine. This is particularly
useful for in-house development where packages that are not available
publicly can be added to http://localhost repositories and multistrap
etc. used from those. Such local repositories can be full or partial
mirrors of publicly available repositories, that's all down to the
reprepro configuration.

Overall, this will allow Grip to jump from 1,000 packages to 8,000
packages without a noticeable increase in the size of the downloaded
Packages files. The Emdebian Grip Packages.gz file for main is currently
381Kb compared to 7.3Mb for Debian. For Grip 2.0, I'd expect a
compressed Packages.gz file of sub 1Mb, hopefully around 600Kb or ~8%
of the size of the equivalent Debian file with a commensurate
reduction in the size of the apt caches.

e.g. Debian:
$ ls -hl /var/cache/apt/
total 26M
drwxr-xr-x 3 root root 88K 2009-08-08 08:04 archives
-rw-r--r-- 1 root root 14M 2009-08-08 08:04 pkgcache.bin
-rw-r--r-- 1 root root 13M 2009-08-08 07:59 srcpkgcache.bin

$ ls -hl /var/cache/apt/
total 1.7M
drwxr-xr-x 3 root root  36K 2009-08-05 20:17 archives
-rw-r--r-- 1 root root 833K 2009-08-05 20:16 pkgcache.bin
-rw-r--r-- 1 root root 832K 2009-08-05 20:03 srcpkgcache.bin

(So Grip caches are ~5% of the size of the Debian caches.)


(With changes intended to be made for Squeeze release goals, the size
of the Packages.gz file could be substantially reduced again but
these changes will have *no* effect on either the size of the
Sources.gz file or the srcpkgcache.bin file. That's the penalty of
having deb-src lines in your apt sources.)

Overrides will still be supported (by reprepro) for individual
packages (e.g. localhost packages). There is no support for repository
management programs other than reprepro in emdebian-grip-server. Some
overrides might be necessary to ensure that 'main' remains installable
without needing extra components.

Note that the 'dev' component is aimed at building normal Debian
packages which can then be gripped - i.e. NATIVE builds. There is no
expectation of cross-compilers being available for Emdebian Grip.

However, it should be possible to build packages on Emdebian Grip 2.0
and then use 'emgrip' to prepare those packages for installation on a
machine *of the same architecture* running Emdebian Crush or for
copying to a repository for the creation of a rootfs of Emdebian Crush
*of the same architecture* as the machine running Grip. (Just as it
will be possible to upload packages built on Grip to a repository of
other Grip packages - of that architecture.) There are no changes in the
source packages themselves between Emdebian Grip and Debian - only the
binaries are 'gripped'. In these situations, there is no need for
emdebuild or other scripts from the 'emdebian-tools' package - building
will be done via dkpg-buildpackage and then use emgrip afterwards. A
simple wrapper script can be created for such builds, with DEB_VENDOR
support etc..



Neil Williams

Attachment: pgpxEYxpb6rU_.pgp
Description: PGP signature

Reply to: