--- Begin Message ---
Package: live-build
Version: 3.0~a21-1
Severity: wishlist
Tags: patch
User: ubuntu-devel@lists.ubuntu.com
Usertags: origin-ubuntu ubuntu-patch oneiric
While trying to squeeze more space out of the Ubuntu live CDs, we
discovered that byte-compiled Python modules were using something on the
order of 12MB compressed. This is clearly not to be sneezed at, and
we'd like to remove the byte-compiled versions and arrange to have them
recompiled when installing the system to disk, since the CPU time
required to byte-compile on the fly is unlikely to dominate I/O when
running from a live image. I've attached a patch that adds a hook to do
this.
The only complications I saw in live-build were (a) in some ways it
isn't ideal to do this as a hook, but it seems a bit too hacky to do in
any other way; (b) some care needs to be taken if you accept something
like my patch to run apt-xapian-index in #627716 to avoid it
re-byte-compiling some modules (obviously feel free to adjust the
attached patch if you apply this one first).
The installer side of this is more complicated than might be considered
ideal due to the profusion of Python helpers. For your amusement and/or
in case you want to do something like this in live-installer, here's my
current code from ubiquity:
def configure_python(self):
"""Byte-compile Python modules.
To save space, Ubuntu excludes .pyc files from the live filesystem.
Recreate them now to restore the appearance of a system installed
from .debs."""
cache = Cache()
# Python standard library.
re_minimal = re.compile('^python\d+\.\d+-minimal$')
python_installed = sorted(
[pkg[:-8] for pkg in cache.keys()
if re_minimal.match(pkg) and cache[pkg].is_installed])
for python in python_installed:
re_file = re.compile('^/usr/lib/%s/.*\.py$' % python)
files = [f for f in cache['%s-minimal' % python].installed_files
if re_file.match(f) and
not os.path.exists(os.path.join(self.target,
'%sc' % f[1:]))]
install_misc.chrex(self.target, python,
'/usr/lib/%s/py_compile.py' % python, *files)
files = [f for f in cache[python].installed_files
if re_file.match(f) and
not os.path.exists(os.path.join(self.target,
'%sc' % f[1:]))]
install_misc.chrex(self.target, python,
'/usr/lib/%s/py_compile.py' % python, *files)
# Modules provided by the core Debian Python packages.
default = subprocess.Popen(
['chroot', self.target, 'pyversions', '-d'],
stdout=subprocess.PIPE).communicate()[0].rstrip('\n')
if default:
install_misc.chrex(self.target, default, '-m', 'compileall',
'/usr/share/python/')
if osextras.find_on_path_root(self.target, 'py3compile'):
install_misc.chrex(self.target, 'py3compile', '-p', 'python3',
'/usr/share/python3/')
def run_hooks(path, *args):
for hook in osextras.glob_root(self.target, path):
if not os.access(os.path.join(self.target, hook[1:]), os.X_OK):
continue
install_misc.chrex(self.target, hook, *args)
# Public and private modules provided by other packages.
install_misc.chroot_setup(self.target)
try:
if osextras.find_on_path_root(self.target, 'pyversions'):
supported = subprocess.Popen(
['chroot', self.target, 'pyversions', '-s'],
stdout=subprocess.PIPE).communicate()[0].rstrip('\n')
for python in supported.split():
cachedpython = cache['%s-minimal' % python]
if not cachedpython.is_installed:
continue
version = cachedpython.installed.version
run_hooks('/usr/share/python/runtime.d/*.rtinstall',
'rtinstall', python, '', version)
run_hooks('/usr/share/python/runtime.d/*.rtupdate',
'pre-rtupdate', python, python)
run_hooks('/usr/share/python/runtime.d/*.rtupdate',
'rtupdate', python, python)
run_hooks('/usr/share/python/runtime.d/*.rtupdate',
'post-rtupdate', python, python)
if osextras.find_on_path_root(self.target, 'py3versions'):
supported = subprocess.Popen(
['chroot', self.target, 'py3versions', '-s'],
stdout=subprocess.PIPE).communicate()[0].rstrip('\n')
for python in supported.split():
cachedpython = cache['%s-minimal' % python]
if not cachedpython.is_installed:
continue
version = cachedpython.installed.version
run_hooks('/usr/share/python3/runtime.d/*.rtinstall',
'rtinstall', python, '', version)
run_hooks('/usr/share/python3/runtime.d/*.rtupdate',
'pre-rtupdate', python, python)
run_hooks('/usr/share/python3/runtime.d/*.rtupdate',
'rtupdate', python, python)
run_hooks('/usr/share/python3/runtime.d/*.rtupdate',
'post-rtupdate', python, python)
finally:
install_misc.chroot_cleanup(self.target)
Thanks,
--
Colin Watson [cjwatson@ubuntu.com]
>From b9ed5879f2f293dbde9d4ee8af7a1c495028abcf Mon Sep 17 00:00:00 2001
From: Colin Watson <cjwatson@canonical.com>
Date: Wed, 22 Jun 2011 15:20:17 +0100
Subject: [PATCH] Add hook to remove .pyc files.
---
examples/hooks/all_chroot_pyc-purge.sh | 7 +++++++
scripts/build/lb_chroot_hacks | 2 +-
2 files changed, 8 insertions(+), 1 deletions(-)
create mode 100755 examples/hooks/all_chroot_pyc-purge.sh
diff --git a/examples/hooks/all_chroot_pyc-purge.sh b/examples/hooks/all_chroot_pyc-purge.sh
new file mode 100755
index 0000000..e74820a
--- /dev/null
+++ b/examples/hooks/all_chroot_pyc-purge.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+# This is a hook for live-build(7) to remove byte-compiled Python modules.
+# To enable it, copy or symlink this hook into your config/chroot_local-hooks
+# directory.
+
+find /usr -name \*.pyc -print0 | xargs -0r rm -f
diff --git a/scripts/build/lb_chroot_hacks b/scripts/build/lb_chroot_hacks
index f9e11dd..3f1e582 100755
--- a/scripts/build/lb_chroot_hacks
+++ b/scripts/build/lb_chroot_hacks
@@ -227,7 +227,7 @@ fi
# build the index in the background which will be racy in the context of
# live-build.
if [ -x chroot/usr/sbin/update-apt-xapian-index ]; then
- Chroot chroot /usr/sbin/update-apt-xapian-index --force --quiet
+ Chroot chroot PYTHONDONTWRITEBYTECODE=1 /usr/sbin/update-apt-xapian-index --force --quiet
fi
# Remove build systems clock drift
--
1.7.5.4
--- End Message ---
--- Begin Message ---
Source: live-build
Source-Version: 3.0~a26-1
We believe that the bug you reported is fixed in the latest version of
live-build, which is due to be installed in the Debian FTP archive:
live-build-cgi_3.0~a26-1_all.deb
to main/l/live-build/live-build-cgi_3.0~a26-1_all.deb
live-build-cron_3.0~a26-1_all.deb
to main/l/live-build/live-build-cron_3.0~a26-1_all.deb
live-build_3.0~a26-1.debian.tar.gz
to main/l/live-build/live-build_3.0~a26-1.debian.tar.gz
live-build_3.0~a26-1.dsc
to main/l/live-build/live-build_3.0~a26-1.dsc
live-build_3.0~a26-1_all.deb
to main/l/live-build/live-build_3.0~a26-1_all.deb
live-build_3.0~a26.orig.tar.gz
to main/l/live-build/live-build_3.0~a26.orig.tar.gz
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 631281@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Daniel Baumann <daniel@debian.org> (supplier of updated live-build package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Format: 1.8
Date: Thu, 04 Aug 2011 21:51:37 +0200
Source: live-build
Binary: live-build live-build-cgi live-build-cron
Architecture: source all
Version: 3.0~a26-1
Distribution: experimental
Urgency: low
Maintainer: Debian Live Project <debian-live@lists.debian.org>
Changed-By: Daniel Baumann <daniel@debian.org>
Description:
live-build - Debian Live - System Build Scripts
live-build-cgi - Debian Live - System Build Scripts (cgi frontend)
live-build-cron - Debian Live - System Build Scripts (cron autobuilder)
Closes: 586929 607225 627332 627716 630088 631281 633052 633356 636208 636440
Changes:
live-build (3.0~a26-1) experimental; urgency=low
.
[ Richard Nelson ]
* Update VERSION in common.sh to match last release.
.
[ Daniel Baumann ]
* Skipping binary_linux-image if no kernel was choosen to be installed
deliberately, thanks to Adam Conrad <adconrad@ubuntu.com>.
* Moving archives meta data arround in source tree.
* Correcting filename of live.debian.net archive file.
* Adding hook for calling update.mlocate, thanks to Colin Watson
<cjwatson@ubuntu.com> (Closes: #627332).
* Adding hook for calling update-apt-xapian-index, thanks to Colin
Watson <cjwatson@ubuntu.com> (Closes: #627716).
* Adding hook for removing gnome icon cache, thanks to Colin Watson
<cjwatson@ubuntu.com> (Closes: #630088).
* Adding hook to remove python py files, thanks to Colin Watson
<cjwatson@ubuntu.com> (Closes: #631281).
* Using --no-check-gpg option when using debootstrap >= 1.0.30 and apt
secure is disabled, thanks to Eugenio Paolantonio <me@medesimo.eu>
(Closes: #633356).
* Also unmounting /run in lb_clean eventhough it's not really needed
anyway but let's be on the caucious side (Closes: #633052).
* Adding kubuntu mode in order to have different defadefault hooks
enabled for ubuntu and kubuntu (needed for #630088).
* Consolidating local hooks for chroot and binary into
config/hooks/*.{chroot,binary}.
* Numbering embedded hooks.
* Saving packages in cache even when failing or interrupting a build
(Closes: #586929).
* Enabling hooks based on the mode we're building for.
* Calling dpkg-divert with --quiet in chroot_dpkg.
* Correcting comments in chroot_hooks to match usage.
* Reworking binary hooks helper.
.
[ Cody A.W. Somerville ]
* Fixing corruption of usb binary images caused by trying to modify
flags in partition table when image is mounted with offset.
* Updating lb_binary_linux-image to only copy over kernel and initrd
if LB_LINUX_PACKAGES doesn't equal 'none'.
* Appending 'linux32' to LB_ROOT_COMMAND if 'uname -m' is x86_64 and
LB_ARCHITECTURE is i386.
.
[ Daniel Baumann ]
* Updating cron scripts.
* Compacting copyright file.
* Reworking debconf stuff of merging former live-autobuild-manual and
live-autobuild-images into a single live-build-cron.
* Avoid to hijack root-command with linux32, rather set use it in the
chroot function only.
* Adding a comment to the chroot calls in excludes handling of
binary_chroot whey the chroot function is not used.
.
[ Cody A.W. Somerville ]
* Setting partition flags in source_usb after creating the filesystem
in order to avoid bug in parted that makes them forget the flags.
* Adding function for more resilient detachment of loopback devices.
.
[ Daniel Baumann ]
* Calling mkdir with -p to avoid build failures with multiple kernels
on disk info in ubuntu mode, thanks to Cody A.W. Somerville
<cody.somerville@canonical.com>.
* Correcting uuid extraction from initrds in ubuntu mode to work from
within binary so that it is cleaned up by lb_clean on failed builds.
* Setting default compression levels to -6 for gzip, bzip, lzip and
lzma tarballs.
.
[ Cody A.W. Somerville ]
* Adding paths for remote debian-installer images on armel.
* Making config tree available to chroot hooks.
.
[ Daniel Baumann ]
* Updating derivatives handling for d-i mirror defaults to better
match the reality that by default, most derivatives are not going to
actually rebuild d-i.
* Adding mdadm manually to extra packages for binary pool for being
used by debian-installer (Closes: #607225).
* Add forgotten handling of already set chroot_hooks in defaults.
* Updating grub packages list in binary_debian-installer.
* Using 'live' rather than 'incomplete' for cd_type in .disk on images
that include debian-installer with live-installer udeb.
* Simplify bind mounting of the config tree during local hooks are run
by using read-only bind mounts.
* Removing unmaintained studio package lists.
* Correcting debconf variables.
* Updating TODO for live-build-cron.
* Correcting typo in lb_config for program variable, thanks to
intrigeri <intrigeri@boum.org>.
* Adding virtual-hdd in lb_config manpage as binary image type.
* Handle package-lists parameter for lb_config in live-build-cron-
images script for both lb version 2.x and 3.x.
* Setting default kernel flavours for armel to all available flavour
due to the nature of armel images (single rootfs image, multiple
kernel images).
* Removing debian-instaler-distribution selection from live-build-
cron-images for the time being.
.
[ Ben Armstrong ]
* Fixing/adding cross-references for live-boot and live-config.
.
[ Daniel Baumann ]
* Correcting cross-references to live-boot and live-config in
lb_config manpage.
* Adjusting check for target directory in live-build-cron scripts to
fail if non-directories are used, thanks to Ben Armstrong
<synrg@debian.org>.
.
[ intrigeri ]
* Cleaning up temporary directories at the same time as other cruft.
.
[ Daniel Baumann ]
* Removing -r from short options in lb_config (Closes: #636208).
* Adding hook to update apt-file cache (Closes: #636440).
* Renumbering hook files.
* Adding missing 'set -e' in remove-python-py.chroot hook.
* Correcting indentiation in hook defaults assignment.
Checksums-Sha1:
cea6a8370850f331bb41d7c14e6a1073bf169982 1299 live-build_3.0~a26-1.dsc
ef3e544cb6208b01b4e43919d17b81f936e93a5d 1871295 live-build_3.0~a26.orig.tar.gz
76bb2f310e11d78b64c42d599ed9521456e59a59 47378 live-build_3.0~a26-1.debian.tar.gz
c3f8ab5570c0ff8a66b2591bbacdf19889784cb9 1136604 live-build_3.0~a26-1_all.deb
48e3fa890c4da6e3fa873a003f12e7a71fa54ad2 60754 live-build-cgi_3.0~a26-1_all.deb
fd95849570e8120d911955b6b9dadec8a6739631 60772 live-build-cron_3.0~a26-1_all.deb
Checksums-Sha256:
b4a40025d6b176864a887d5abd6c6c771c6a124c718387bce3e21fbdf7b69c9e 1299 live-build_3.0~a26-1.dsc
02c23763237a137455844f6ef0626f735e1d70e11b14f52ce622238f6bc7be19 1871295 live-build_3.0~a26.orig.tar.gz
30785186808273c3a61bb7020527c19ab90afba6ce88d8fa33c6056c10e9f343 47378 live-build_3.0~a26-1.debian.tar.gz
77d7a25cfe20e93a867a232b72ea6a7cb05cff4df40616bf4e790f03c7d42e3e 1136604 live-build_3.0~a26-1_all.deb
2d247bd7b42cf5c10787ef7e0bf82cfb215b6f0906029117e57acee8e2621287 60754 live-build-cgi_3.0~a26-1_all.deb
c6e13f93375a4fc82df0ab4027ed2f2984a4032a3cf118e761978a941f38a787 60772 live-build-cron_3.0~a26-1_all.deb
Files:
0382e832aa44a068c21bde942e1cd271 1299 misc optional live-build_3.0~a26-1.dsc
b622c099838a6a00f3d2288c29270205 1871295 misc optional live-build_3.0~a26.orig.tar.gz
d82b446062e1de54b27e51ab7e962730 47378 misc optional live-build_3.0~a26-1.debian.tar.gz
d5aca919b8ba0a020e1d8e67b075a82a 1136604 misc optional live-build_3.0~a26-1_all.deb
d04e86ff88f3bedb7f5b30ce1dc4722a 60754 misc optional live-build-cgi_3.0~a26-1_all.deb
7b12f869311214368767c32525fa4766 60772 misc optional live-build-cron_3.0~a26-1_all.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEARECAAYFAk47iZcACgkQ+C5cwEsrK55A3QCglv6HDJazJMpFNy8rJGjpzjvM
UNgAoJDk3DMmXr+djagq15x2iOE248tf
=ygZ/
-----END PGP SIGNATURE-----
--- End Message ---