[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

python packaging infrastructure

After a long silence (and some work on "unrelated" C++ transitions ;),
an update to the python packaging infrastructure.

We want to change the default python version used to the recent
python-2.4.2 release, and change it again to 2.5.0, when/if this
version is released before etch freezes.  Furthermore the release team
requested a reduction of the available python versions in etch and and
an easier upgrade procedure when changing the default python version.

The need to support more than one python version? In the past, some
applications didn't work with newer python versions, for sarge these
were zope (2.6) using python2.2 and jython using the python2.1 library
and modules.  At least jython isn't an issue anymore (orphaned, and
new upstream snapshots available), and the latest Zope2 release
started to use python2.4.  So looking at the final product (the
release), there's no need for more than one supported version.  There
are reasons to support more than one version:

 - It makes a transition easier (when forward conflicts can be
 avoided), if some applications can stay a little bit longer with the
 previous release.  This doesn't seem to be an issue with the
 transition to 2.4 anymore as this version was released some time ago,
 but we can expect these incompatibilities when transitioning to a
 freshly released 2.5 version.

 - Many developers like to have all (well, maybe the recent ones)
 python versions available to be able to test their application with
 more than one python version.  That's ok, but maybe doesn't directly
 belong in a Debian release.  Design the packaging in such a way that
 it can be used the minimal set of python versions in a Debian and
 release and be used to automatically rebuild packages for more python
 versions would be a compromise. Now that binary NMU's are a reality,
 uploads adding or dropping a python version could be done this way as

Other implementations should fit into the packaging as well, although
there seems a lack of interest for Jython in Debian, and IronPython
seems to depend on the newest .net features not yet available in the
free .net implementations.

Removing explicit dependencies on python versions seems to be the
goal, when we persue the goals outlined above. To be able to rebuild
packages mostly automagically, some things have to be done:

 - Remove explicit build dependencies.  These can be replaced by the
 new python-all or python-all-dev packages, and explicit version
 information can be extracted from the python-all dependencies.
 python-all_2.3.5-4 (currently in incoming) depends on python2.3 and

 - Drop the packages having an explicit python version in it's name
 (pythonX.Y-foo).  Instead of directly installing for a python
 version, a packaging infrastructure cares about making a package
 available for a specific python version.  From a user point of view,
 there is no need to make a distinction between modules and
 extensions.  Small extensions may be shipped in the same package,
 larger extensions may be kept in their own known python version
 specific packages.

 - Packages beeing installed outside the default sys.path and
 providing python modules have to be re-byte-compiled, when a python
 version changes.

 - Installing a python implementation / python version makes all
 installed modules/packages available for the new python version.

The above requires some changes in the python policy, what follows is
a simplified proposal how to change the policy:

 - For architecture independent packages:

   - build one binary-all package and specify which python implementations
   it works with

   TODO later: if different versions are required to run with packaged
   python implementations, package different versions of the module in
   different source packages.

   Example: the latest version of foo uses generator expressions,
   which cannot be used with jython.

   - modules are installed into a directory not directly in sys.path.

 - Architecture-dependent packages:

   - Small extension modules (.so file smaller than some threshold
   value, 50k?): The extensions for all compatible python
   implementations/versions are put into the same binary package.

   - Large extension modules: Do as before: put them in a separate
   binary package (python2.y-foo), add a dependency package.

 - Add two packages 'python-all' and 'python-all-dev'.

   - These packages depend on all supported runtime / development
   versions supported by the distribution. Use these packages only as
   build dependencies for python modules, to determine for which
   python implementations/versions a module package is supposed to
   build. Using this information, a Python module package can extract
   the default Python version, and all Python versions it needs to
   build for, from the build environment. When generating the control
   file, the module's package does not need to be modified when the
   default Python version is changed or support for a new version
   and/or implementation is added.

 - Add a new field to the control file for binary packages,
 XB-Python-Version. The field should list all python versions which
 the module/extension is supposed to install to.

 A special value of "all" causes installation to all Python versions
 (so that architecture-independent modules do not have to be
 re-uploaded when a new Python implementation/version enters the
 distribution). We have to use a new field, because we cannot
 determine all installed python modules when installing a new Python

The installation and removal of packages is supported by some support
scripts / registry script / name-it-whatever-scripts:

 - Python modules are installed in a directory tree which is managed
 by the registry.

 - No directory of the registry is part of sys.path in any python

 - On package installation: modules are made available to every
 compatible installed Python implementation on the system
 (technically: copy the directory tree and symlink every file).
 The files are then byte-compiled.

 - On package removal: remove files/directories created during
 Delete byte-compiled files.

 - When installing a new python implementation, all installed python
 modules are made available to it.  If some modules applications
 cannot be upgraded (according to the "Python-Version" field), then
 the update of the new python version fails.
 Then, byte-compile the files.

Other/unrelated issues:

 - Wether to provide "optimized" byte-compiled files. Drop it / keep
 it, make it a configuration option.

 - Adjust the installed size field. Currently the size of
 byte-compiled files is not mentioned in the Installed-Size field.
 Get the value at build time, either by a test compilation or an
 estimation (double the size of the .py files).

Implementation: There currently exists more than one implementation,
the goal definitely is to have only implementation handling all the
packaging issues mentioned above.

- python-central (0.1-0.3), done by Bastian Kleinadam some years ago,
  only handles binary-all modules.

- python-support, done by Josselin Mouette, available in the archives,
  only handling binary-all modules, supported by an updated dh_python.

- python-central (0.4), a reimplementation of Bastian Kleinadam's
  work, handling python extensions and applications/modules outside
  the default sys.path.  To not interfer with python-support's
  changed dh_python, this has a renamed dh_pycentral. Can be found at
     deb http://people.debian.org/~doko/python ./
  The package changing the default python version is at
     deb http://people.debian.org/~doko/python-default ./

I'm currently unhappy that the new dh_python breaks current packaging
practice, please let us come to a common approach how to handle the
packaging infrastructure.  As dh_python currently violates the python
policy, I filed a RC report to debhelper and python-support to prevent
them entering the testing distribution.

Some things about python-support, others may comment about python-central:

 - Some concerns were raised by the release team, that python-support
 tracks it's own state instead of using the dpkg database. To keep
 track of usable python versions, python-central uses a field
 Python-Version in the control file (having values like "2.3, 2.4",
 "all", ">= 2.3")
 I don't mind a separate '.pyvers' file in a directory, but that information
 should be generated from the control file.  Having the available tools
 be able to deal with the additional information is a benefit.

 - It doesn't have yet support for extensions and modules outside the
 default sys.path. although it should be easy to add these.

 - python-support's dh_python currently does more on it's own than
 dh_pycentral does.

I'm in favour of supporting an infrastructure handling extensions and
modules outside the default sys.path well, which ever this
implementation will look like.


Reply to: