Re: Rsync and incremental updates on top of Original CDs

["J.A. Bezemer" <costar@panic.et.tudelft.nl>]

On Tue, 20 Feb 2001, Richard Atterer wrote:

> Hi Anne,
> 
> first of all, I should probably have mentioned the name that I have
> /finally/ decided on: Jigsaw Download, or short jido. (I just hope
> that "jido" doesn't happen to mean anything naughty in any language -
> you never know... :)

Googling voor "jido" reveals mostly martial arts sites, so I guess that won't
be a problem.

> On Tue, Feb 20, 2001 at 12:58:42PM +0100, J.A. Bezemer wrote:
> > On Sun, 18 Feb 2001, Richard Atterer wrote:
> > 
> > > Note that I'm currently working on a tool which will allow you to say:
> > > "Here's a CD full of files and an 'image template' (created for a new
> > > CD, also by my tool) - please fetch any new deb's and re-use the other
> > > files from the old CD."
> > 
> > Cool!
> >
> > You mention "the" old CD as if it were only one (which is very nice
> > already), but since packages tend to get shuffled around a bit each
> > time, it would be most useful to allow multiple local "repositories"
> > (mounted or copied CDs, and /var/cache/apt/archives/ for example). 
> > (And in the extreme case: multiple CDs in one drive to be changed on
> > request a la apt; should be possible in at least C which allows you
> > to skip stuff when writing and re-write it later. Which would also
> > be useful to concentrate all FTP/HTTP traffic in an
> > as-short-as-possible online period.)
> 
> Hm, all this could be done, but you'll need to add it to
> pseudo-image-kit yourself! :-) 

The problem is that the current make-pseudo-image is a bash script, and AFAIK
bash/cmdline-utils can't do middle-of-the-file addressing. I wouldn't mind
having a C equivalent, but my time is extremely limited.

> In true Unix fashion, "jido-file" is just a command line utility which
> tries to do one thing well; in this case, work on image template
> files. It doesn't contain any networking code. (A second download tool
> "jido" will eventually do that, it could completely replace the
> pseudo-image-kit.)
> 
> This is /roughly/ how it could work:
[...]

If I understand it correctly, you first want to get all needed files on some
local filesystem, and assemble them to a real image later. This does seem a
duplication of effort, since when "jido-file print-matching" knows an md5sum
of some file, it also knows its place in the CD image and can easily copy it
there directly (assuming jido-file is not a shell script ;-), as in:

 <insert and mount CD>
 find /cdrom -type f \
 | jido-file copy-matching \
     --files-from=- --template=potato.iso.template --output=potato.iso \
     --already-done=potato.iso.already-done

where already-done would contain md5sums of files that were already copied to
the .iso earlier (something like your --cache).

Followed by either

a) jido-file print-missing \
     --template=potato.iso.template --jido=potato.iso.jido \
     --already-done=potato.iso.already-done \
   > list-of-files-to-get

   Get needed files to some local directory with for example a modified
   make-pseudo-image

   Repeat "jido-file copy-matching" as above

or for example
b) jido-net get-and-copy-missing
     --template=potato.iso.template --jido=potato.iso.jido \
     --already-done=potato.iso.already-done --output=potato.iso
     --ftp-mirror-list=/README.mirrors

   which would do the getting and copying at once.

And either at the start or halfway through or at the end some command to fill
in the stuff that's specific to the CD (binary data from the template).

I think that would be the easiest way to do things.

I get the (maybe incorrect) impression that you think files can only be
created linearly, but nothing is less true. Try this:

-----------------------------
#include <stdio.h>

int
main (void)
{
  FILE *testfile;
  char part1[] = "a test\n";
  char part2[] = "this is ";

  testfile = fopen ("testfile", "a");	/* Make sure file exists */
  fclose (testfile);			/* otherwise r+ won't work */

  testfile = fopen ("testfile", "r+b");	/* b for Windows compat */

  fseek (testfile, 8, SEEK_SET);	/* or 800000000 ;-) */
  fwrite (part1, 1, 7, testfile);

  fseek (testfile, 0, SEEK_SET);
  fwrite (part2, 1, 8, testfile);

  fclose (testfile);

  return 0;
}
-----------------------------

This also works in Windows, except that you probably shouldn't try 800000000
there if you're low on disk space. 


>   Would it be OK for you to update the kit once
> jido-file is in a working state?

You mean to create a modified make-pseudo-image to get missing files to some
local dir? That shouldn't be too hard for me ;-)


Regards,
  Anne Bezemer