Re: Re: live-build: LB_APT_INDICES="false" does not remove indices
I've been spending time familiarising myself with the code for this tool
over the past few days, and I have a fairly good grasp of it now.
I believe the problem lies within the chroot_archives script, and is
caused by the check on lines 539-547.
Those who already know how the tool works can feel free skip over the
following background, otherwise please bare with me as you need to read
this if you're to understand the cause of the problem (and I'd
appreciate you doing so after taking the time to investigate and then
write all this up :p )...
Background:
---------------------
This script is used a few times throughout the build process, and its
purpose is basically to setup the archive configuration for apt, and
toggles it between a 'chroot' and a 'binary' configuration. It can be
run in 'install' mode which configures apt in 'chroot' mode (using the
mirror specified per your configuration for the building of the disk
image), and 'remove' mode, which toggles the configuration to 'binary'
mode, configuring apt to use the 'binary' mirror, etc, as per your
configuration. The binary configuration is the configuration that will
actually be built into the live OS you run from the disc, while the
chroot configration, as already mentioned, is for the build process.
Before reading any further, understand that the build process (by
default at least) prefers at various points, to use (through 'chroot
execution) the binaries within a new installation it sets up within a
directory it creates named 'chroot', rather than the binaries of the
host. This installation will also become the file system of the live OS
used from the disk.
The build process is broken down into five bigs steps. The first creates
a 'chroot' directory with a very basic installation, and creates a
backup of this. The second step installs into this chroot installation
any packages that should go into the live OS (setting up the live OS).
The third retrieves any packages needed for an installer if your
configuration asked for one, which are saved into a 'binary' directory
(external to 'chroot'), into which everything that will become the
contents of the disk is placed. The fourth moves the 'chroot' directory
to one side (which is the actual live OS file system at this point), and
now uses the backup taken earlier, to then create a virtual disk file
(if it's correct to call it that, a file containing another filesystem
anyway), into which it copies the live OS 'chroot' directory contents.
This virtual disk file is saved into the 'binary' directory, other files
are added to complete the disk contents, and then these are wrapped up
into an iso. The fifth step generates an optional extra iso containing
source packages.
Each of the latter four stages begin by executing some scripts to
'prepare' the chroot directory for use by the build process, e.g. to
block starting/stopping of daemons during package installation, and to
enable necessary mounts for running binaries in the special 'chroot'
mode (I'll leave you to learn about chroot through google if you don't
already know). After this they do their work, and then undo the earlier
prep work. The exception is the fourth step which at the very beginning
must move the chroot directory (now representing the completed
filesystem for the live OS) to one side, and get a new chroot directory
to work from, by copying from the backup taken at the beginning of the
process.
One of these prep scripts is this chroot_archive script. So at the
beginning of a step, this script is 'installed', setting up apt within
the chroot directory to retrieve packages from the 'chroot' mirror. Work
is done, then this script is executed again in remove/uninstall mode,
which reconfigures apt to retrieve packages from the 'binary' mirror.
I.e. you can, through your configuration parameters, have the build
process use one mirror for building the disk ('chroot' mirror), and
setup a different mirror ('binary' mirror) in the live OS filesystem
that is run from the disk you've created.
The cause of the problem:
------------------------------------
To reduce wasted effort, under certain conditions, when run in remove
mode, the scripts exits early. Specifically, if it sees that the binary
mirrors are identical to the chroot mirrors, there's no point in it
continuing. There is one exception to this, during install mode it would
have taken any packages within certain directories in your config
directory and built a local repository with them, within
chroot/root/packages/; The remove process should remove this, so if this
directory exists, it will continue on ahead and remove it.
The problem is that this early exit implementation is flawed. There are
some steps during the removal process which really shouldn't be skipped.
For example, the step which removes apt indices if it is supposed to.
So if your configuration has identical mirrors for both chroot and
binary stages, and this 'chroot/root/packages/' directory does not exist
(because you did not add any packages that would have been placed in it
into your config), then this early exit occurs in this script, and thus
some things are not done.
***Another possible issue (I need feedback):
------------------------------------------------------------
The bit of code that is there for removing apt indices, only clears out
chroot/var/lib/apt/lists. It doesn't touch any of the files in the
directory above this. Now I'm not nearly familiar enough with the files
here to know which should and should not be deleted when the parameter
in question is used, but the message I'm responding to suggests that
possibly some of the files in chroot/var/lib/apt should also be deleted.
Files here touched elsewhere in the code (by a caching mechanism) include:
- secring.gpg*
- trusted.gpg*
- pkgcache.bin
- srcpkgcache.bin
- *_Packages
- *_Sources
- *Release*
I would guess that the first two should be left alone, perhaps the
latter three also. Elsewhere in the code the *bin files are deleted
before then using apt update & apt upgrade, which suggests that they're
recreatable indices, thus something that should be deleted in this context.
I may assume this is so in the patch I build, but I would really like
some feedback to confirm it.
Solution:
-------------------------
I'm going to build a patch (which I'll submit attached to a bug report)
which fixes some issues with this script.
The bulk of the code for the removal section is focused firstly on
generating the source list file, and then removing 'chroot'
config/pref/local-source configuration files and keys, replacing them
with 'binary' ones. I understand it skipping creation of the source list
file if the mirrors are the same, but not skipping the rest. Skipping
the small routines at the end, like removing apt indices, may have been
an oversight, but skipping the swapping of other apt config components
would seem intentional. I don't understand that intention though. If the
user of the tool is not just using the same mirror, but also wanted to
use other configuration files, and identical ones for both chroot and
binary use, they would create one copy of the files with generic names
(rather than postfix their names with '.chroot' or '.binary'), and of
course theirs no garuantee that these files would be identical, even if
using the same mirrors. This removal code does not dis-configure such
generic config components; it only removes any '.chroot' specific ones
and installs '.binary' specific ones, but this is skipped by early exit,
which makes no sense.
Furthermore, the install side of this script does not remove '.binary'
configurations, despite this script being used to toggle the config
between chroot and binary several times in the build process, they
should be always completely replacing each other.
One further issue I see is that is any '.chroot' keys are installed,
these are never removed by the removal process; there's no code for it.
Thus also there is no code for removing '.binary' keys during the
install process.
So to summarise the changes I will implement in a patch (or a set of
patches):
1) Fix this early exit behaviour to ensure that things that shouldn't be
skipped, like clearing apt indices, are not skipped. I.e. I may only
allow it to skip generation of the source list file.
2) Add code for the install process to remove '.binary' config
components (to account for the fact that this scripts is used over and
over again through the process, and steps after the first use of this
scripts should not have the chroot environment they execute on/in
contaminated with '.binary' config components'.
3) Add code for removal of '.chroot' keys installed during the install
mode, and similarly the opposite for install mode, as per the above.
4) Expand the apt indice removal code to remove more indice files (when
user supplied config/parameters instruct the script to), if it is
correct that more files are deleted than currently.
I'll try to build this patch and submit it in a bug report tomorrow.
It'll probably take less time than writing this message.
Alternate solution:
--------------------------
Perhaps the following would actually be a nicer/better approach than
described above. The second major stage of the build process results in
the completed live OS filesystem within the chroot directory. The first
use of this chroot_archives script is here; During the 'prep' stage, it
configures apt into chroot mode, ready to then fetch and install
packages, building the live OS filesystem, then at the end, in the
'undo-prep' stage, it results in apt being configured in binary mode.
Currently we then go through the whole installer step (which ultimately
makes no changes), before moving this aside and getting a new chroot
(from a backup) to continue working with. This script is again executed
in prep and undo-prep stages of the installer step, meaning it must
carefully toggle between chroot and binary apt configurations. What
would be better would be if we put to one side the chroot directory as
soon as we've done building it; we get a new chroot from backup, and
make sure that the chroot_archive stage file is removed. The following
steps (installer, binary, source) then don;t need to worry so much about
mucking this one up. They may need to prep and un-prep the scripts that
mount and unmount directories for chroot execution use, to support the
steps being executed in isolation, but otherwise, they can simply apply
pre steps, such as executing chroot_archive in install mode, without
bothering to un-prep, and thus keeping the chroot_archive code more
simple than otherwise (no need for the install stage to remove '.binary'
config).
I think I might build a patch built upon this idea instead. Hopefully
Daniel will be okay with these changes.
P.s. Appologies for such a long message for people to read; I do try to
be clear and thorough, which sometimes means a lot of text.
Reply to: