I'm considering a useful change in how Grip works at the repository level. Currently, a brute force method is used to put any package with a name ending in '-dev' into a 'dev' component and ending in '-doc' into the 'doc' component. This will be retained, such that ordinary users have a sources line of: deb http://www.emdebian.org/grip squeeze main Those who want to build packages on Grip will then add: deb http://www.emdebian.org/grip squeeze dev deb-src http://www.emdebian.org/grip squeeze main deb-src http://www.emdebian.org/grip squeeze dev (doc is largely redundant currently, as -doc packages themselves are nearly empty but this change puts tools handling documentation into the doc component as well as the -doc packages themselves.) This allows apt to download the Packages file that allows installation of the -dev packages without bloating the 'main' Packages file with the details of those -dev packages as well as the sources need to build the packages themselves. I'm looking at extending this to make it easier to handle a larger number of packages as well as making the relocation easier to understand. It's a simple change in the code, instead of always setting '-C main' when calling reprepro, I'd call '-C $section' where $section holds the name of the Section: listed for that package in the Debian Packages file (this uses the overrides set by ftpmaster.debian.org, not necessarily the value encoded in the debian/control file within the sources). This does mean that some packages that are currently in 'main' for Grip will move into 'dev' or 'doc' - check the Section listings for more information: http://www.emdebian.org/grip/search.php Each Grip repository could then choose which components to activate - as soon as reprepro is configured to use the component, packages will be redirected into it. Name matching will be case-insensitive One exception, both the 'devel' and 'libdevel' Sections will end up in the 'dev' component. Some Section:s already listed for Grip include: admin base comm database debian-installer devel doc editors embedded fonts games gnome graphics interpreters libdevel libs mail math misc net oldlibs otherosfs perl python ruby shells sound tex text utils vcs video web x11 xfce Of those, xfce|gnome could be useful components, debian-installer is a mystery and probably needs to be fixed. deb http://www.emdebian.org/grip squeeze main deb http://www.emdebian.org/grip squeeze xfce or deb http://www.emdebian.org/grip squeeze main deb http://www.emdebian.org/grip squeeze gnome etc. The Debian archive maintainers provide the authoritative list of sections. At present, they are: admin, cli-mono, comm, database, devel, debug, doc, editors, electronics, embedded, fonts, games, gnome, graphics, gnu-r, gnustep, hamradio, haskell, httpd, interpreters, java, kde, kernel, libs, libdevel, lisp, localization, mail, math, misc, net, news, ocaml, oldlibs, otherosfs, perl, php, python, ruby, science, shells, sound, tex, text, utils, vcs, video, web, x11, xfce, zope. (Policy 2.4) So 'kde' could be handled in the same way as xfce or gnome. The important point here is that only the components configured by the admin of that individual repository will be managed in this way, such that each repository can adjust the relative sizes of each component. In effect, a Section: becomes a discrete component within the distribution that can be added or removed by users. Once removed, aptitude will list packages installed from that component as "obsolete", allowing easy removal. Which Sections are in use needs to be clearly explained when setting out the apt sources lines to be used with the repository. This isn't about creating 20 components and having users specify some combination of 12 different ones that may or may not end up as fully installable. It is meant to allow repository admins to cherry-pick a handful of components and reliably apportion packages into those components whilst preserving the core packages in main. The emphasis is to provide components where the package sets are roughly equivalent, whereby xfce|kde|gnome|foo is a good example. It is also needed to ensure that packages like 'automake' reliably end up in 'dev' without needing complex regular expression matching and without dumping hundreds of development tool packages into 'main'. If the admin does not configure the component, the Section: variable is ignored and 'main' is used. I'm undecided whether to enable 'xfce' for www.emdebian.org/grip or whether to leave xfce as the de facto default and offer other environments as their own components. I'm also undecided whether Grip ever wants a complete KDE or complete GNOME environment or whether we offer some GNOME/KDE packages to complement the more slimline apps from other environments. I will be enabling the 'dev' and 'doc' changes to keep things like automake out of Grip/main. I'll also enable 'gnome' and 'kde' at www.emdebian.org/grip. Users should not be expected to need more than three or four components to get all the binary packages they need (excluding deb-src lines). It is also part of the drive to encourage others to create their own Grip repositories with their own configurations and their own package lists. Just because we have www.emdebian.org/grip does not mean that all Grip installations need to come from that one machine. This is particularly useful for in-house development where packages that are not available publicly can be added to http://localhost repositories and multistrap etc. used from those. Such local repositories can be full or partial mirrors of publicly available repositories, that's all down to the reprepro configuration. Overall, this will allow Grip to jump from 1,000 packages to 8,000 packages without a noticeable increase in the size of the downloaded Packages files. The Emdebian Grip Packages.gz file for main is currently 381Kb compared to 7.3Mb for Debian. For Grip 2.0, I'd expect a compressed Packages.gz file of sub 1Mb, hopefully around 600Kb or ~8% of the size of the equivalent Debian file with a commensurate reduction in the size of the apt caches. e.g. Debian: $ ls -hl /var/cache/apt/ total 26M drwxr-xr-x 3 root root 88K 2009-08-08 08:04 archives -rw-r--r-- 1 root root 14M 2009-08-08 08:04 pkgcache.bin -rw-r--r-- 1 root root 13M 2009-08-08 07:59 srcpkgcache.bin Grip: $ ls -hl /var/cache/apt/ total 1.7M drwxr-xr-x 3 root root 36K 2009-08-05 20:17 archives -rw-r--r-- 1 root root 833K 2009-08-05 20:16 pkgcache.bin -rw-r--r-- 1 root root 832K 2009-08-05 20:03 srcpkgcache.bin (So Grip caches are ~5% of the size of the Debian caches.) http://www.emdebian.org/grip/dists/unstable/main/binary-armel/ http://ftp.debian.org/debian/dists/unstable/main/binary-armel/ (With changes intended to be made for Squeeze release goals, the size of the Packages.gz file could be substantially reduced again but these changes will have *no* effect on either the size of the Sources.gz file or the srcpkgcache.bin file. That's the penalty of having deb-src lines in your apt sources.) Overrides will still be supported (by reprepro) for individual packages (e.g. localhost packages). There is no support for repository management programs other than reprepro in emdebian-grip-server. Some overrides might be necessary to ensure that 'main' remains installable without needing extra components. Note that the 'dev' component is aimed at building normal Debian packages which can then be gripped - i.e. NATIVE builds. There is no expectation of cross-compilers being available for Emdebian Grip. However, it should be possible to build packages on Emdebian Grip 2.0 and then use 'emgrip' to prepare those packages for installation on a machine *of the same architecture* running Emdebian Crush or for copying to a repository for the creation of a rootfs of Emdebian Crush *of the same architecture* as the machine running Grip. (Just as it will be possible to upload packages built on Grip to a repository of other Grip packages - of that architecture.) There are no changes in the source packages themselves between Emdebian Grip and Debian - only the binaries are 'gripped'. In these situations, there is no need for emdebuild or other scripts from the 'emdebian-tools' package - building will be done via dkpg-buildpackage and then use emgrip afterwards. A simple wrapper script can be created for such builds, with DEB_VENDOR support etc.. Comments? -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/
Attachment:
pgpxEYxpb6rU_.pgp
Description: PGP signature