[debian-python@l.d.o added to To and Reply-To, citing whole mail for those who don't read -devel, me included ] First of all: thanks Matthias for your work on Python package(s) [Matthias Klose, 2009-02-16] > Besides the "normal" pending update of the python version for the > unstable distribution, there will be more changes around python > packaging, including the introduction of python-3.x and addressing > some packaging issues. > > > Python versions > --------------- > > - 2.4 is still used by zope-2.x and depending packages like plone. > There was a SoC project which intended to update zope-2.x to use a > newer python version, but afaik this didn't reach a stable state > yet. It doesn't make sense to provide a full set of third party > python modules and extensions for 2.4. If it is forseeable that > zope-2.x still requires 2.4 when the freeze for the next Debian > release approaches, maybe package only 2.4 modules for zope-2.x's > and plone's dependencies. An alternative might be not to > distribute zope-2.x and plone with the next stable release. > > - 2.5 is superseded by 2.6; currently there doesn't seem to be > a reason to ship 2.5 and modules for 2.5 with the next stable > release. The upstream 2.5 maintainance branch doesn't see bug > fixes anymore, only security releases will be made from this > branch. What about a smooth upgrade path for those who use Python2.5 for their (3rd party) applications? I think Python 2.6 should be default in Squeeze and Python 2.4&2.5 in supported. > - 2.6 is the current stable release which should enter unstable > soon and become the default after extension packages supporting > more than one python version are built and migrated to testing. > > - I do not want to speculate about a 2.7 release and the upstream > version freeze for the next Debian release. If the 2.7 release > is well ahead of the Debian freeze, then there should be no > problem to include it. > > - 3.0/3.1: I do not plan to upload 3.0 to unstable or experimental, > but will prepare 3.1 packages for experimental and upload those > to unstable with the final release or a late release candidate. > The 3.1 release is planned for April 2009. > > Packaging for Python 3.x > ------------------------ > > Python 3.x is not upward compatible to 2.x and requires changes to the > sources. Almost no file can be used unmodified. A large part of the > changes can be done automatically with the help of the 2to3 utility. > Porting resources can be found at > > - http://docs.python.org/3.0/whatsnew/3.0.html#porting-to-python-3-0 > - http://docs.python.org/3.0/howto/cporting.html#cporting-howto > - http://docs.python.org/3.0/library/2to3.html > > Upstream does expect Python-2.x to stay around for a while; Python-3.x > will need some time until extensions and modules are ported. For this > reason I do plan to have both Python-2.x and Python-3.x in the next > stable Debian release. > > Some third party modules and extensions may be released in a form > where the code for 3.x can be auto-generated from the 2.x code with > the 2to3 utility, some upstreams may decide to stop 2.x support with > one version and provide 3.x support with another version. Debian has > to support both approaches. The currently used packaging methods only > allow packaging of one version for all Python version (installing in a > shared area and providing this version for each Python version). > Binary packages will double in size when providing both 2.x and 3.x > compatible code, so it does make sense to provide separate binary > packages for 2.x and 3.x. These new packages should be prefixed with > `python3-' instead of `python-'. It's up to the package maintainer if > these packages are built from one or two source packages. > > There will be a `python3' package and similiar packages built out of a > python3-defaults, which will provide the `python3' binary. > > > Local installation path > ----------------------- > > When installing Python modules using distutils, the resulting files > end up in the same location wether they are installed by a Debian > package, or by a local user or administrator, unless the installation > path is overwritten on the command line. Compare this with most > software based on autoconf, where an explicit prefix has to be > provided for the packaging, while the default install installs into > /usr/local. For new Python versions packaged in Debian this will > change so that an installation into /usr (not /usr/local) requires an > extra option to distutils install command (--install-layout=deb). To > avoid breaking the packaging of existing code the distutils install > command for 2.4 and 2.5 will just accept this option and ignore it. > For the majority of packages we won't see changes in the packaging, > provided that the python packaging helpers can find the files in the > right location and move it to the expected target path. > > A second issue raised by developers was the clash of modules and > extensions installed by a local python installation (with default > prefix /usr/local) with the modules provided by Debian packages > (/usr/local/lib/pythonX.Y/site-packages shared by the patched "system" > python and the locally installed python. To avoid this clash the > directory `site-packages' should be renamed to `dist-packages' in > both locations: > > - /usr/lib/pythonX.Y/dist-packages (installation location for code > packaged for Debian) great idea > - /usr/local/lib/pythonX.Y/dist-packages (installation location > for locally installed code using distutils install without > options). > > The path /usr/lib/pythonX.Y/site-packages is not found on sys.path > anymore. is "local/" missing in this path or do you want to simply rename site-packages into dist-packages? > > About the name: Discussed this with Barry Warsaw and Martin v. Loewis, > and we came to the conclusion that using the same directory name for > both locations would be the most consistent way. > > This change should make the request to conditionalize the inclusion of > /usr/local/lib/pythonX.Y/site-packages into sys.path obsolete. if I understand this correctly, local installations (including ez_install ones) will use /usr/local/.../site-packages and the ones used by administrators (who know about --install-layout=deb) /usr/local/.../dict-packages, and the site-packages will not be in default sys.path, right? If yes, then although I love the idea of "solving" the `sudo easy-install MyModule` really annoying issue (last time I raised it here[1], see also Christoph's "Why you should not use easy_install carelessly on Debian"[2]), what's a purpose of having local site-packages if installing there will have no effect. Or do you want to keep it for local *Python* interpreter installations? [1] http://groups.google.com/group/pylons-discuss/browse_thread/thread/86c713cdf9c7a54d/a53a64d8670f76ac [2] http://workaround.org/easy-install-debian > If needed we can provide a symlink /usr/lib/pythonX.Y/site-packages > pointing to dist-packages. > > > Common installation location for shared code > -------------------------------------------- > > The source files for code (public modules) which is shared between > Python versions are currently located in different paths: > > - /usr/share/pycentral/<binary package name> > - /usr/share/pysupport/<binary package name> > - /usr/share/pyshared > > With this approach the Debian packaging tools (dpkg) cannot handle > file conflicts and replaces themself anymore. Errors are delayed to > the python packaging tools which either ignore these kind of errors or > have to replicate the conflicts/replaces checks. > > The use of a common location was proposed in 2006, but didn't get any > feedback at this time. Last summer I did choose a name for this > location, and a large part of packages already install into this > location `/usr/share/pyshared'. For code shared between Python-3.x > installations I propose to use `/usr/share/py3shared'. I really like the idea of using the same location for both tools, please note that you'll have to change pycentral to use something like /usr/lib/pyshared (for Python extensions) or I will not even try to convince Joss to fix #478178[3] ;-P > Use of exactly one location in sys.path > --------------------------------------- > > Depending on the python packaging helper, public modules are available > in different locations on sys.path. Having parts of python packages > or upstream plugin systems installed into different locations may > break the python package or the plugin system. To avoid this situation > we do have the policy (section 2.1, second paragraph) that such > packages must use the same location (means: the same python packaging > helper). This part of the policy tends to be ignored. > > - The import order is changed; this issue was raised earlier. See > http://lists.debian.org/debian-python/2006/01/msg00065.html > The use of relative imports (introduced in 2.5) make this more > obvious. > > - Plugin systems derive the location for plugins from their > installation location. It is not an upstream bug if Debian > packages different plugins with different python packaging helpers. > > The choice of packaging helper should not add additional constraints > on upstream software. To avoid these problems in the future, exactly > one location for public python modules shold be used. yes, that's a problem, but I think we should do it step by step, first: all packages should ship .py and .so files in the same directories (so that dpkg can handle conflicts again), then we'll think about using common entry in sys.path (and/or merging helper tools or making it possible to choose one of them to do all the stuff) > Avoid runtime dependencies on python packaging helpers > ------------------------------------------------------ > > During upgrades or distribution upgrades, public python modules are > not available for use, or they may be in an inconsistent state. This > is the time between removal of old version / unpacking new version and > the postinst configuration, when the python packaging helpers are run > and the public modules made available in sys.path. There are > currently work arounds implemented to shorten the window of > unavailability: > > - Not removing symlinked files and not removing byte-compiled files > during upgrades. This only works reliable if the new version > does neither remove nor add files, or else an inconsistent set of > files in a package may be imported during an upgrade. > > - Already create the symlinked files in the preinst step, because > the time between running the preinst and unpacking the new version > is much shorter. > > Ideally I would like to see public python modules available after > unpack without the help of any helper application. This is possible > by creating the symlinks while creating the package and including > these in the package instead of creating these at installation time. > The only configuration needed at installation time would be > byte-compilation. > > The advantage would be a more robust upgrade path. > > The disadvantage of this approach is the limitation to a specific set > of python versions (the package will have a dependency of the form > "python (<< X.Y)", which we already have for architecture dependent > packages containing extensions. This would add this kind of > dependency to all architecture independent python-* modules and then > requires an upload of these packages for each python transition as > well. Afaics this would not affect a transition or a set of > transitions to testing, because no new dependencies on new versions of > shared libraries are introduced during these uploads. This approach > certainly has an impact on the ftp archive and its mirrors, but I > can't say if this would be an issue or not. I don't like python (<< X.Y) dependencies, it's so... old-policy like. Not a good idea, IMHO > > If this cannot be done for all packages, this should be done for a > subset of "core" modules and extensions. > > There is still the issue of handling name space packages based on > setuptools. Ideally existing techniques like diversions should be used > for this, even if it looks loke overkill to divert the same empty > __init__.py file. > > > Various > ------- > > There are other things which may be worth a look. > > - The current policy advises packages to byte-compile only the > files which a package owns itself. This is not done by several > packages, with the result that an upgrade caused by a byte- > compilation error may be attributed to the wrong package (makes > the upgrade of an otherwise fine package fail). > > - The removal of a python version will cause the need for massive > rebuilds. because many python extensions currently have > dependencies of the form pythonX.Y-foo. There is nothing what > can be done now for the upcoming removal, but those dependencies > should not be there by default. This is 2.4 of the python policy, > but many packages tend to ignore that. python-support supports namespace packages and it does it good. I didn't want it to be enabled by default but since Joss provided a way to disable it (see #459468) I think it's OK. python-central should implement the same behaviour, IMHO > > > I will start uploading python2.6 and related packages, then proceed > with python3.x in the next weeks. > > Matthias Just one more issue: what about "current" issue? Although I protested when others wanted to remove it, now I agree it's useless. All packages that depend on it (Python applications mostly) should use private directories and thus not pollute the global namespace (we should add this to the Python policy, IMO) -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=-
Attachment:
pgpplvQGWgtLJ.pgp
Description: PGP signature