Bug#796641: cdimage.debian.org: jigdo scanning from local media is slower than dirt

[Mateusz Poszwa <old4@o2.pl>]

Je 2015-08-23 00:13:00
Richard Jasmin <frazzledjazz@gmail.com> skribis:

> when using local media to scan for packages needed, it is horrendously slow.
> 
> This could be sped up by using a local package database or csv file or the like
> on each install medium.Then it would be a breeze to scan that file and compare
> it to the jigdo template. The entire medium, in my case BDROM, would not need
> to be scanned.
> 
> Since the entire jigdo process is all about knowing what packages are where,
> this should be an easy fix.

Hello.

This is a very nice idea. It’s not a cdimage.d.o problem though.

If I remember correctly, JigDo keeps a cache with md5sums of files
it has already scanned in ~/.jigdo-file-cache.db.
It should be possible to extend it with appropriate entries,
given the template (which contains, among other information,
md5sums and paths) and the mountpoint.

For now I have some workarounds for you.
As I said, JigDo maintains a cache of scanned files,
so you can pre-scan your media (or a loopmounted image)
in spare time as soon as you get them,
so you don’t have to do it when you create a new image.
Of course you will still need them for the packages,
but at least JigDo will already know where to find them.

Second workaround requires a lot of spare disk space.
Long ago (oh dear; it was 5 years ago) I’ve written many
JigDo-related tools. Among them was a script for finding
and copying files needed by a given JigDo file.
It’s faster than JigDo, because it makes use of filenames,
and checks only required files (unless it doesn’t find all of them
(which happens because images usually don’t share the same kernels),
in which case it falls back to scanning everything, but you can
interrupt the script at this point and just make JigDo download
the remaining files). You can use it to extract needed files
to a temporary folder, and then point JigDo to scan that folder.
Even though you essentially scan the needed files twice,
I think it’s still worthwhile if you make ,e.g., a DVD out of a BD.
If you make a BD out of BD, you don’t need all this in the first place.

You can find the script at http://f8l.netne.net (jigdo-files.sh).

If you want even faster method, I could modify another script of mine
which extracts packages from images based on information found in
templates. It extracts all files now, but I can make it read
another template to extract only files found in both templates.
Let me know if you want that. This script is much shorter and clearer,
so if you don’t trust 5-years-ago-me, this is a better way.

I have more JigDo tricks up my sleeve (like jigdo.php).
One day I could port what I described here into JigDo itself.
No additional files are needed, as the information is already there.

I’m always happy to help.
Best regards.

-- 
Mateusz Poszwa