list2cd rewrite
Over the past two weeks, I've rewritten list2cd from scratch to
implement some improvements (of course, YMMV):
- It now knows about the isofs sector size of 2048 bytes. All file
sizes are rounded up to the next 2k boundary. This is configurable;
if for some reason you want to create a UDF or ext2 CD, you can
change it.
- The CD stuffing algorithm has been changed. If a package won't fit
on a CD, we dump it into a queue of files to be retried on the next
CD, then grab the next package and try to fit it on. Repeat until
we run out of packages or space on the CD.
- You can limit the number of CD list files that are produced. This
is useful if you want to build an "oddly-sized" CD that you'd never
want a second (or third or thirteenth) CD for. Since nobody is
going to install from 18 MiniCDs, we can save a lot of time by just
building the package tree for the first one.
- The whole dependency logic has been simplified, and now you can
specify whether you want depends, recommends, and suggests to be
respected when adding packages to CDs.
Together, this approach makes it possible to create "full" CDs, and
might even be able to shave a CD off a big set. However, there are
some drawbacks (or caveats at least):
- It's written in Python. If you don't like Python, this might be a
problem for you. It's actually Python 2.2 code, since for readability
it uses a lot of the cleanups in the 2.x series. (dict.has_key(x)
is written as 'x in dict', and it makes use of nested scopes, list
comprehensions, and 'true' division.)
- It exposes problems with the rest of the CD build process;
specifically, poor estimates of how big the CD is going to be. This
problem is particularly egregious with the HFS-hybrid CDs, because
the HFS stuff adds a lot of overhead to the disc. Ideally, we
should be at the point where you can say "MAX=700" and get images
that fill a 700MB blank; instead, you can end up with discs in the
750MB range. This is particularly critical with the CD stuffing
logic, because you can't depend on some huge package at the end of
the CD saving your butt anymore. (The upside is that the CD sizes
should be stable for any reasonably large package set, since there's
always some lower-popularity package that's smaller. It removes
some of the black magic, which is good.)
- There's currently no accounting for the fact that directories take
up space on the destination filesystem. (The old program didn't do
that either, but until woody it really didn't matter... but the
pools add a lot of small directories to the disc, which may be a Bad
Thing space-wise... could be around 2MB per disc.)
- It depends on accurate information about what's on the CDs. One
solution that may be helpful is to fill the temporary area for each
CD with the "non-package" stuff first, du --blocksize=2048 -s the
area for each CD, and then work from that... which would invert the
building order and mess up existing build scripts that do make
bootable after make list. (I haven't actually tried this yet, it
just occurred to me. But it would probably work better.)
- Some of the other nooks and crannies of debian-cd aren't quite set
up yet for woody; the PPC boot stuff appears somehwat broken, for
example, and the .calc bits aren't there. The latter should be a
non-issue after the item above is fixed.
- It doesn't directly drop in yet. It behaves the same in terms of
environment variables, but its command line options aren't quite the
same. For now I've called it "list2cds-cram" since it crams things
onto the CDs.
- I haven't tested all of the NONUS and NONFREE bits yet. They're
written, and the code looks like it will work, but I haven't
bothered yet (and I haven't gotten a local non-US mirror set up yet
either).
- I don't know what happens when attempts are made to build the source
CDs. It could be "bad."
Anyway, the net result is that it can build reasonable CDs from woody
in both MiniCD and 650 MB sizes, but it still needs work, and I don't
know if I should commit it to CVS (under a different name, of course)
until I get all this business worked out with the makefile and getting
more accurate sizing info.
Chris
--
Chris Lawrence <lawrencc@debian.org> - http://www.lordsutch.com/chris/
Reply to: