[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Python related changes for unstable/squeeze

Besides the "normal" pending update of the python version for the
unstable distribution, there will be more changes around python
packaging, including the introduction of python-3.x and addressing
some packaging issues.

Python versions

 - 2.4 is still used by zope-2.x and depending packages like plone.
   There was a SoC project which intended to update zope-2.x to use a
   newer python version, but afaik this didn't reach a stable state
   yet. It doesn't make sense to provide a full set of third party
   python modules and extensions for 2.4.  If it is forseeable that
   zope-2.x still requires 2.4 when the freeze for the next Debian
   release approaches, maybe package only 2.4 modules for zope-2.x's
   and plone's dependencies. An alternative might be not to
   distribute zope-2.x and plone with the next stable release.

 - 2.5 is superseded by 2.6; currently there doesn't seem to be
   a reason to ship 2.5 and modules for 2.5 with the next stable
   release. The upstream 2.5 maintainance branch doesn't see bug
   fixes anymore, only security releases will be made from this

 - 2.6 is the current stable release which should enter unstable
   soon and become the default after extension packages supporting
   more than one python version are built and migrated to testing.

 - I do not want to speculate about a 2.7 release and the upstream
   version freeze for the next Debian release. If the 2.7 release
   is well ahead of the Debian freeze, then there should be no
   problem to include it.

 - 3.0/3.1: I do not plan to upload 3.0 to unstable or experimental,
   but will prepare 3.1 packages for experimental and upload those
   to unstable with the final release or a late release candidate.
   The 3.1 release is planned for April 2009.

Packaging for Python 3.x

Python 3.x is not upward compatible to 2.x and requires changes to the
sources. Almost no file can be used unmodified. A large part of the
changes can be done automatically with the help of the 2to3 utility.
Porting resources can be found at

 - http://docs.python.org/3.0/whatsnew/3.0.html#porting-to-python-3-0
 - http://docs.python.org/3.0/howto/cporting.html#cporting-howto
 - http://docs.python.org/3.0/library/2to3.html

Upstream does expect Python-2.x to stay around for a while; Python-3.x
will need some time until extensions and modules are ported. For this
reason I do plan to have both Python-2.x and Python-3.x in the next
stable Debian release.

Some third party modules and extensions may be released in a form
where the code for 3.x can be auto-generated from the 2.x code with
the 2to3 utility, some upstreams may decide to stop 2.x support with
one version and provide 3.x support with another version. Debian has
to support both approaches.  The currently used packaging methods only
allow packaging of one version for all Python version (installing in a
shared area and providing this version for each Python version).
Binary packages will double in size when providing both 2.x and 3.x
compatible code, so it does make sense to provide separate binary
packages for 2.x and 3.x.  These new packages should be prefixed with
`python3-' instead of `python-'.  It's up to the package maintainer if
these packages are built from one or two source packages.

There will be a `python3' package and similiar packages built out of a
python3-defaults, which will provide the `python3' binary.

Local installation path

When installing Python modules using distutils, the resulting files
end up in the same location wether they are installed by a Debian
package, or by a local user or administrator, unless the installation
path is overwritten on the command line.  Compare this with most
software based on autoconf, where an explicit prefix has to be
provided for the packaging, while the default install installs into
/usr/local.  For new Python versions packaged in Debian this will
change so that an installation into /usr (not /usr/local) requires an
extra option to distutils install command (--install-layout=deb).  To
avoid breaking the packaging of existing code the distutils install
command for 2.4 and 2.5 will just accept this option and ignore it.
For the majority of packages we won't see changes in the packaging,
provided that the python packaging helpers can find the files in the
right location and move it to the expected target path.

A second issue raised by developers was the clash of modules and
extensions installed by a local python installation (with default
prefix /usr/local) with the modules provided by Debian packages
(/usr/local/lib/pythonX.Y/site-packages shared by the patched "system"
python and the locally installed python.  To avoid this clash the
directory `site-packages' should be renamed to `dist-packages' in
both locations:

 - /usr/lib/pythonX.Y/dist-packages (installation location for code
   packaged for Debian)
 - /usr/local/lib/pythonX.Y/dist-packages (installation location
   for locally installed code using distutils install without

The path /usr/lib/pythonX.Y/site-packages is not found on sys.path

About the name: Discussed this with Barry Warsaw and Martin v. Loewis,
and we came to the conclusion that using the same directory name for
both locations would be the most consistent way.

This change should make the request to conditionalize the inclusion of
/usr/local/lib/pythonX.Y/site-packages into sys.path obsolete.

If needed we can provide a symlink /usr/lib/pythonX.Y/site-packages
pointing to dist-packages.

Common installation location for shared code

The source files for code (public modules) which is shared between
Python versions are currently located in different paths:

 - /usr/share/pycentral/<binary package name>
 - /usr/share/pysupport/<binary package name>
 - /usr/share/pyshared

With this approach the Debian packaging tools (dpkg) cannot handle
file conflicts and replaces themself anymore. Errors are delayed to
the python packaging tools which either ignore these kind of errors or
have to replicate the conflicts/replaces checks.

The use of a common location was proposed in 2006, but didn't get any
feedback at this time.  Last summer I did choose a name for this
location, and a large part of packages already install into this
location `/usr/share/pyshared'.  For code shared between Python-3.x
installations I propose to use `/usr/share/py3shared'.

Use of exactly one location in sys.path

Depending on the python packaging helper, public modules are available
in different locations on sys.path.  Having parts of python packages
or upstream plugin systems installed into different locations may
break the python package or the plugin system. To avoid this situation
we do have the policy (section 2.1, second paragraph) that such
packages must use the same location (means: the same python packaging
helper).  This part of the policy tends to be ignored.

 - The import order is changed; this issue was raised earlier. See
   The use of relative imports (introduced in 2.5) make this more

 - Plugin systems derive the location for plugins from their
   installation location. It is not an upstream bug if Debian
   packages different plugins with different python packaging helpers.

The choice of packaging helper should not add additional constraints
on upstream software.  To avoid these problems in the future, exactly
one location for public python modules shold be used.

Avoid runtime dependencies on python packaging helpers

During upgrades or distribution upgrades, public python modules are
not available for use, or they may be in an inconsistent state.  This
is the time between removal of old version / unpacking new version and
the postinst configuration, when the python packaging helpers are run
and the public modules made available in sys.path.  There are
currently work arounds implemented to shorten the window of

 - Not removing symlinked files and not removing byte-compiled files
   during upgrades. This only works reliable if the new version
   does neither remove nor add files, or else an inconsistent set of
   files in a package may be imported during an upgrade.

 - Already create the symlinked files in the preinst step, because
   the time between running the preinst and unpacking the new version
   is much shorter.

Ideally I would like to see public python modules available after
unpack without the help of any helper application.  This is possible
by creating the symlinks while creating the package and including
these in the package instead of creating these at installation time.
The only configuration needed at installation time would be

The advantage would be a more robust upgrade path.

The disadvantage of this approach is the limitation to a specific set
of python versions (the package will have a dependency of the form
"python (<< X.Y)", which we already have for architecture dependent
packages containing extensions.  This would add this kind of
dependency to all architecture independent python-* modules and then
requires an upload of these packages for each python transition as
well.  Afaics this would not affect a transition or a set of
transitions to testing, because no new dependencies on new versions of
shared libraries are introduced during these uploads. This approach
certainly has an impact on the ftp archive and its mirrors, but I
can't say if this would be an issue or not.

If this cannot be done for all packages, this should be done for a
subset of "core" modules and extensions.

There is still the issue of handling name space packages based on
setuptools. Ideally existing techniques like diversions should be used
for this, even if it looks loke overkill to divert the same empty
__init__.py file.


There are other things which may be worth a look.

 - The current policy advises packages to byte-compile only the
   files which a package owns itself. This is not done by several
   packages, with the result that an upgrade caused by a byte-
   compilation error may be attributed to the wrong package (makes
   the upgrade of an otherwise fine package fail).

 - The removal of a python version will cause the need for massive
   rebuilds. because many python extensions currently have
   dependencies of the form pythonX.Y-foo.  There is nothing what
   can be done now for the upcoming removal, but those dependencies
   should not be there by default.  This is 2.4 of the python policy,
   but many packages tend to ignore that.

I will start uploading python2.6 and related packages, then proceed
with python3.x in the next weeks.


Reply to: