Bug#771002: Fix save to wrong cache
Package: live-build
Version: 4.0.3-1
Tags: patch
tl;dr short version:
--------------------
The chroot_package-lists script caches downloaded package files to the packages.binary cache instead of the packages.chroot cache. The result (as explained more thoroughly below) results in a lot of needless duplication of files between these two caches, wasting disk-space, and also slowing down the build process a little.
Patch attached (built against debian-next).
Longer explanation:
-------------------
Quick overview of problem area:
1) Upon instigating 'lb build', the first sub-stage to the build process (bootstrap) creates a basic installation in the 'chroot' directory.
2) The second stage ('chroot') is the focus of the issue, this breaks down as follows (unimportant details skipped)...
2a) A process is gone through of configuring apt, and generating a list of packages to download and install. This list includes any 'local' packages provided within your config (for which a local repository is created and linked to); linux-image packages; and firmware packages.
2b) The 'chroot_package-lists' script is executed (in 'install' mode) to get any additional packages specified in list files with your configuration and add them to this list.
2c) The 'chroot_install-packages' script reads the list and installs all of these packages. The packages are saved into the packages.chroot cache. The list is deleted at the end. A listing of all packages installed in chroot is generated and saved (to be saved on the disk for use by live).
2d) The 'chroot_package-lists' script is executed once again, this time in 'live' mode. This causes it to read any lists in your config with names ending in '-live'.
2e) The 'chroot_install-packages' script is executed to install these packages. Some hooks/hacks/etc are run. An updated list of installed packages is generated and saved as a different file (live needs to know the difference, presumably so if persisting a live OS to disk, after copying itself to disk, it can uninstall those packages only intended to be run from live media).
The problem:
- Prior to step 2b, the following packages may possibly be held within the packages.chroot cache: aptitude; keyring packages (debian-keyring, or others specified in config/whatever); and apt-utils. Note that you may not see any of these if you observe this process, whether they are downloaded depends on various factors.
- Step 2b recovers (copies) any packages cached in the packages.chroot cache to apt's cache in chroot (so that apt can use cached copies of packages rather than re-downloading), and downloads package dctrl-tools. (A dependency required to be installed for it to complete its job). All packages (up to four now) in apt's cache in chroot are then backed up (moved) to the packages.binary cache.
- Step 2c recovers (copies) all packages from the packages.chroot cache, and now downloads a whole bunch of additional packages as specified in the list. There were about 60 downloaded in an example basic config I tried, which included no optional extras like a DE. These are backed up (moved) to the packages.chroot cache.
- Step 2d recovers (copies) all packages (now a lot) from packages.chroot. It has to re-download dctrl-tools. After processing any additional lists, it then saves (moves) all packages to the packages.binary cache.
So the packages.binary cache now contains a duplicate copy of most of (if not all of) the packages in the packages.chroot cache. Moving forward through the entire process, it did not appear that a copy of these is actually useful to any later code that actually intentionally makes use of the packages.binary cache. Thus it wastes disk space, and the extra burden of copying these packages back and forth for later parts of the build process that use the packages.binary cache also slows down the build process a little.
From 2fc67f16e0bef68613e9b3d0d0db23beaf7d2c14 Mon Sep 17 00:00:00 2001
From: jnqnfe <jnqnfe@gmail.com>
Date: Tue, 25 Nov 2014 19:28:13 +0000
Subject: [PATCH] Fix chroot_package-lists script saving to wrong cache
---
scripts/build/chroot_package-lists | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/build/chroot_package-lists b/scripts/build/chroot_package-lists
index ba808bd..fb0c3ee 100755
--- a/scripts/build/chroot_package-lists
+++ b/scripts/build/chroot_package-lists
@@ -112,7 +112,7 @@ case "${LB_BUILD_WITH_CHROOT}" in
fi
# Saving cache
- Save_cache cache/packages.binary
+ Save_cache cache/packages.chroot
;;
esac
--
1.9.4.msysgit.0
Reply to: