|
I'm in the middle of building a patch for 718225, I'm having to
think carefully about caching at this point, and I think the area of
caching could do with a review (or at least a little input or
agreement with my proposals/plans from Daniel). Obviously its too late to get too much improvement in this area into jessie, but knowing the intended direction of improvement for a future version could be very helpful in building this patch, (as well as shaping the next version). (tl;dr summary at end.) Brief caching background: ------------------------------------- The set of cachable files includes downloaded: packages (deb & udeb), installer files (vmlinuz & initrd), and distribution information files (release, packages.gz and contents-[arch].gz files). Additionally, there's possible caching of certain completed build stages, but this is mainly useful only for development of LB, with the exception of caching the basic bootstrap chroot which is required by the current design of the build process. Caching scenarios include: During a single clean build (don't download the same thing multiple times); Re-running a build process in a directory where a build has previously taken place, without cleaning it out, so potential for many files to be retrieved from the cache; and doing an offline build, where absolutely everything will and must come from the cache (a variation of building in a previously used directory; and not to be confused with using your own local mirror, a completely different thing!). One further situation complicates things further. The description of the --cache-packages parameter describes that disabling it is not recommended, but in rare setups it is actually faster to re-download (from a local mirror) rather than hit the disk. Offline building: ------------------------------------- The only reference to offline building I've seen in LB documentation is within the description for the --cache-indices parameter ("would allow to rebuild an image completely offline, however, you would not get updates anymore", which was introduced in the v1.0 live-helper days. If enabled, a few apt data files are retrieved from a cached copy if they exist, rather than re-installing a few local keys and key packages (if you have any in your config), and an install of aptitude is possibly avoided. To me this param and bit of code are a little puzzling, adding complexity to avoid very little work, and it's not even entirely clear to me how it helps "offline" capabilities, unless only in relation to the possible install of aptitude, which surely could be handled much more cleanly. I can find nothing through google that suggests this parameter is actually needed by anyone, just someone happening to notice it break in an ancient bug report, and a mention of it in an old article. Furthermore, offline building has actually been broken for at least two and a half years now, since "support for including firmware packages automatically" was included in v3.0-a47, unless you disable inclusion of firmware packages (set --firmware-binary and --firmware-chroot to false). That code has always lacked caching support, preventing offline building. Clearly there seems to be no serious use for it. My partially complete 718225 work does actually fix the lack of caching in the 'firmware' related code, and thus it's possible that offline building could be workable once again, but should I bother paying it any thought? I'm still trying to figure out how I might best use caching in implementing this patch; if we agreed on offline support being unnecessary and ditching it, it might possibly make things a little less complicated (and --cache-indices could perhaps be ripped out later inline with that). Update: Just noticed, downloading of mirror 'trace' files (placed in the image as .disk/archive_trace) does not use caching, and has been in place since v1.0.5-2, so offline building can't have worked since then! (Unless perhaps that just silently fails). No Caching (of package files) ------------------------------------- (Downloading being faster than retrieving from disk). Apparently a rarely needed capability, which has existed since around v1.0-a22. Nothing on google about it, though I'm sure it does work, so maybe there are people using it, but is anyone? Obviously there's going to be a fair bit of disk activity during the build process anyway, this just reduces it a little. Can the existence of this functionality really be justified? It would be nice to remove unnecessary things like this to keep the code cleaner, so perhaps it could be removed in the next version. Does anyone seriously require this? Freshness ------------------------------------- For the installer stage of the build process, everything is retrieved from the cache if a copy is already there, except (currently) the contents-[arch].gz files used to get a list of firmware packages to download. If you check out the installer images (e.g. http://ftp.debian.org/debian/dists/sid/main/installer-amd64/), i.e. the vmlinuz and initrd files used by the install process, you'll see that they rarely, but occasionally are updated. Obviously if you use the daily build it's more frequent. There's actually a security risk in using the daily build (inadequate info files to securely download, which my 718225 patch highlights to the user), so perhaps that's to be avoided, but there's still a decent need for a user to want to replace their cached installer files. In terms of distribution information files used in a secure-wget-download verification process in my 718225 patch, I intend to use the cached copies, but if a failure occurs, download fresh copies of any cached files used as a second chance, and only really fail if the verification check after that fails also. So this will work perfectly fine in all scenarios. Contents-[arch].gz files used to get a list of firmware packages, and Packages.gz, used to get a list of udebs may possibly need to be updated, and also trace files (if caching where implemented for them). So certain files discussed here should be used from the cache during a build, to avoid unnecessary repeat downloads (e.g. contents-[arch].gz is downloaded twice, once during chroot_firmware, and once during installer_debian-installer), but the user may need choice of whether to refresh them at the start of a new build, or whether to allow the cached copies to be used. How should this best be approached?? There are a few options:
Obviously the third is the simplest for users, but least
efficient, and the fourth perhaps requires the most forethought
for each build. I'm not certain which I'd consider best, but I
think I'd lean towards one of the first two. As for implementing it, rather than having the first block of
code downloading a dist-info file doing a flush if required, it
would be cleaner to have a simple 'init' stage to the build
process which does that I think. Or possibly an init script
executed at the start of the bootstrap script. Summary (tl;dr)
|