python packaging infrastructure
After a long silence (and some work on "unrelated" C++ transitions ;),
an update to the python packaging infrastructure.
We want to change the default python version used to the recent
python-2.4.2 release, and change it again to 2.5.0, when/if this
version is released before etch freezes. Furthermore the release team
requested a reduction of the available python versions in etch and and
an easier upgrade procedure when changing the default python version.
The need to support more than one python version? In the past, some
applications didn't work with newer python versions, for sarge these
were zope (2.6) using python2.2 and jython using the python2.1 library
and modules. At least jython isn't an issue anymore (orphaned, and
new upstream snapshots available), and the latest Zope2 release
started to use python2.4. So looking at the final product (the
release), there's no need for more than one supported version. There
are reasons to support more than one version:
- It makes a transition easier (when forward conflicts can be
avoided), if some applications can stay a little bit longer with the
previous release. This doesn't seem to be an issue with the
transition to 2.4 anymore as this version was released some time ago,
but we can expect these incompatibilities when transitioning to a
freshly released 2.5 version.
- Many developers like to have all (well, maybe the recent ones)
python versions available to be able to test their application with
more than one python version. That's ok, but maybe doesn't directly
belong in a Debian release. Design the packaging in such a way that
it can be used the minimal set of python versions in a Debian and
release and be used to automatically rebuild packages for more python
versions would be a compromise. Now that binary NMU's are a reality,
uploads adding or dropping a python version could be done this way as
well.
Other implementations should fit into the packaging as well, although
there seems a lack of interest for Jython in Debian, and IronPython
seems to depend on the newest .net features not yet available in the
free .net implementations.
Removing explicit dependencies on python versions seems to be the
goal, when we persue the goals outlined above. To be able to rebuild
packages mostly automagically, some things have to be done:
- Remove explicit build dependencies. These can be replaced by the
new python-all or python-all-dev packages, and explicit version
information can be extracted from the python-all dependencies.
python-all_2.3.5-4 (currently in incoming) depends on python2.3 and
python2.4.
- Drop the packages having an explicit python version in it's name
(pythonX.Y-foo). Instead of directly installing for a python
version, a packaging infrastructure cares about making a package
available for a specific python version. From a user point of view,
there is no need to make a distinction between modules and
extensions. Small extensions may be shipped in the same package,
larger extensions may be kept in their own known python version
specific packages.
- Packages beeing installed outside the default sys.path and
providing python modules have to be re-byte-compiled, when a python
version changes.
- Installing a python implementation / python version makes all
installed modules/packages available for the new python version.
The above requires some changes in the python policy, what follows is
a simplified proposal how to change the policy:
- For architecture independent packages:
- build one binary-all package and specify which python implementations
it works with
TODO later: if different versions are required to run with packaged
python implementations, package different versions of the module in
different source packages.
Example: the latest version of foo uses generator expressions,
which cannot be used with jython.
- modules are installed into a directory not directly in sys.path.
- Architecture-dependent packages:
- Small extension modules (.so file smaller than some threshold
value, 50k?): The extensions for all compatible python
implementations/versions are put into the same binary package.
- Large extension modules: Do as before: put them in a separate
binary package (python2.y-foo), add a dependency package.
- Add two packages 'python-all' and 'python-all-dev'.
- These packages depend on all supported runtime / development
versions supported by the distribution. Use these packages only as
build dependencies for python modules, to determine for which
python implementations/versions a module package is supposed to
build. Using this information, a Python module package can extract
the default Python version, and all Python versions it needs to
build for, from the build environment. When generating the control
file, the module's package does not need to be modified when the
default Python version is changed or support for a new version
and/or implementation is added.
- Add a new field to the control file for binary packages,
XB-Python-Version. The field should list all python versions which
the module/extension is supposed to install to.
A special value of "all" causes installation to all Python versions
(so that architecture-independent modules do not have to be
re-uploaded when a new Python implementation/version enters the
distribution). We have to use a new field, because we cannot
determine all installed python modules when installing a new Python
version.
The installation and removal of packages is supported by some support
scripts / registry script / name-it-whatever-scripts:
- Python modules are installed in a directory tree which is managed
by the registry.
- No directory of the registry is part of sys.path in any python
implementation.
- On package installation: modules are made available to every
compatible installed Python implementation on the system
(technically: copy the directory tree and symlink every file).
The files are then byte-compiled.
- On package removal: remove files/directories created during
installation.
Delete byte-compiled files.
- When installing a new python implementation, all installed python
modules are made available to it. If some modules applications
cannot be upgraded (according to the "Python-Version" field), then
the update of the new python version fails.
Then, byte-compile the files.
Other/unrelated issues:
- Wether to provide "optimized" byte-compiled files. Drop it / keep
it, make it a configuration option.
- Adjust the installed size field. Currently the size of
byte-compiled files is not mentioned in the Installed-Size field.
Get the value at build time, either by a test compilation or an
estimation (double the size of the .py files).
Implementation: There currently exists more than one implementation,
the goal definitely is to have only implementation handling all the
packaging issues mentioned above.
- python-central (0.1-0.3), done by Bastian Kleinadam some years ago,
only handles binary-all modules.
- python-support, done by Josselin Mouette, available in the archives,
only handling binary-all modules, supported by an updated dh_python.
- python-central (0.4), a reimplementation of Bastian Kleinadam's
work, handling python extensions and applications/modules outside
the default sys.path. To not interfer with python-support's
changed dh_python, this has a renamed dh_pycentral. Can be found at
deb http://people.debian.org/~doko/python ./
The package changing the default python version is at
deb http://people.debian.org/~doko/python-default ./
I'm currently unhappy that the new dh_python breaks current packaging
practice, please let us come to a common approach how to handle the
packaging infrastructure. As dh_python currently violates the python
policy, I filed a RC report to debhelper and python-support to prevent
them entering the testing distribution.
Some things about python-support, others may comment about python-central:
- Some concerns were raised by the release team, that python-support
tracks it's own state instead of using the dpkg database. To keep
track of usable python versions, python-central uses a field
Python-Version in the control file (having values like "2.3, 2.4",
"all", ">= 2.3")
I don't mind a separate '.pyvers' file in a directory, but that information
should be generated from the control file. Having the available tools
be able to deal with the additional information is a benefit.
- It doesn't have yet support for extensions and modules outside the
default sys.path. although it should be easy to add these.
- python-support's dh_python currently does more on it's own than
dh_pycentral does.
I'm in favour of supporting an infrastructure handling extensions and
modules outside the default sys.path well, which ever this
implementation will look like.
Matthias
Reply to: