On Wed, 22 Apr 2009 12:06:35 +0100 Wookey <wookey@wookware.org> wrote: > +++ Neil Williams [2009-04-21 20:01 +0100]: > > On Tue, 21 Apr 2009 20:33:57 +0200 > > Hector Oron <hector.oron@gmail.com> wrote: > > > > > Are there any plans to create/modify an apt-get suitable for embedded > > > systems? I mean without that much perl dependencies and without that > > > oversized apt cache. > > > > As far as the size of the apt cache is concerned, Grip has that problem > > solved - filtering the archive. > > That does help enormously, but only so long as you can operate without > Debian itself. As soon as you need one package from debian that > implies a full debian package list/cache download. OK, so maybe a script is needed that is part of the emdebian-grip package. Users can choose whether to install it on the Grip device or on a nearby desktop/laptop but the process needs some way of permanently adding the package to the Grip filter list. There are already scripts in the emdebian-grip-server package that could form the basis of the new script. I'm adding dozens of new packages to Grip at the moment (nearly 1,000 in total now) and I'm wondering how to arrange the "request a new package" functionality. 1. New script, apt-grip, that does none of the complicated apt-cross stuff, it works more like multistrap in that it gets the Packages file from Debian to some temporary directory, downloads the package and dependencies using apt to a temporary directory, removes all the Packages and cache data, calls emgrip against all the .debs in the temporary download area, bins the downloaded ones, installs the grip ones and cleans up after itself. (Note how the planned script deletes as-it-goes to save total space requirements. I need to find time to get that incremental apt support implemented and the new script could work with that very simply but if the new script runs principally on a Debian box or server, it wouldn't matter anyway. Space is cheap on Debian boxes, don't you know. :-)) 2. Some method of getting that added to Grip long term. Now, we could use the BTS (requires working email on the device running apt-grip) or we could use a web form (security problems, spamming problems, junk requests and other pointlessness). Other methods? What I need is a good way of looking up the data for the packages without needing the entire Packages file on the Grip device itself. That way, we minimise the amount of data to be downloaded and processed on the Grip device. If someone fancies writing a client:server application? (Possibly best done in perl because that's how all the other code is currently.) The Packages file could live on a machine with lots of room (like a server somewhere) and when it receives a query from the client it can do the look up locally, get the local apt installation to do the dependency stuff and actually download the packages - maybe even grip them on-the-fly - and then give the client a URL to go-get-and-install. So the server part would go into the emdebian-grip-server package in Debian and the client into emdebian-grip. I've trimmed the dependencies of emdebian-grip in 1.9.1 - it should be less of a burden to install on a Grip device now but the client could also run on a laptop running standard Debian and just copy the gripped .debs onto a USB stick etc. Could be a very useful service if it could run on emdebian.org - it would also make it easy to keep a log of requested packages for addition to the main Grip filter. Alternatively, make the whole thing only run on standard Debian machines, do away with the server:client architecture from the start and use some other way to get the list of added packages into Grip itself. (A standard Debian box could easily use the BTS as email is very likely to be properly configured and internet accessible etc.) The questions that remain are where we draw the lines - e.g. should Grip even contain exim? JRE? (I'm expecting to add a complete build environment in the next week or so - I want to be able to compile something for XFCE on a Grip device using the dev component in the repository: deb http://www.emdebian.org/grip sid main dev but that is a separate Packages file so it's less of an issue. exim and JRE would end up in the main Packages file.) How many packages do we want in Grip *main* in total ? It'll be over 1,000 by the time this current run is done and I know that is smaller than it needs to be. Should we go to 5,000 or try to stay below 2,000? Above 5,000 is where the size of the Grip Packages file will start to become noticeable again, above 10,000 in one place and we lose some of the benefits of the filtering. -doc and -dev packages are now automatically filtered into doc and dev components respectively (although there are very few actual packages in existence so far). Those are easy because we can rely on the package name for filtering. It gets harder if we want packages like exim to be in alternative archive components to avoid cluttering main. An alternative is that the emdebian-grip-server package is constantly updated to exactly what is running on www.emdebian.org/grip/ so any server anywhere can create a secondary repository that just adds a few packages - resulting in say 3,000 basic packages at www.emdebian.org and another 3,000 spread over three or four other servers elsewhere and people can take their pick. The principle would be that the secondary servers host *alternative* packages - e.g. a different basic environment, so that users would only need to add at most one secondary. There need to be systems to organise such things and there need to be volunteers to write some new code and host some repositories. (New scripts for inclusion into emdebian-* packages should be GPL-3+ but other than that, anyone on the list should feel free to contribute. Most of the ideas above are only incremental changes from existing examples in the emdebian-grip-server package. Check /usr/share/emdebian-tools/) > On my grip systems > doing 'apt-get update' makes my 56Mb initial install become a 120 MB > install. By the end of the current set of updates, you should find that Emdebian Grip sid and squeeze contain the packages you need - if not, let me know. > in tmpfs so they are lost on reboot). Maybe we could arrange to to > grip-style description filtering on downloaded lists? Probably easier to not download the Packages file from Debian in the first place. > Thee are definately changes which would allow the correct behaviour > and functionality of apt/dpkg without _quite_ so much space > requirement. Usualy at the expense of speed, but actually, filtering > the package file probably makes it faster overall. Even gzipping might > depending on relaitve speeds of flash and CPU. If we could run most of the processing on a server (or some other remote machine), it wouldn't cost anything in terms of speed. There are options available based on Grip - as long as we stay with the Grip methods of architecture-neutrality and post-processing. It's when we need to recompile stuff that it all gets hard. It's time to get more people involved in Grip, so this is the call for new contributors. I'm not planning on writing the scripts outlined above (shortage of time) but I will respond and fix bugs that may arise when new scripts need to use the existing code as part of the ongoing development of the code behind Grip. (That's why the outline is vague and open-ended.) -- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/
Attachment:
pgpsVbD_4RKjy.pgp
Description: PGP signature