My attempt to merge python-central and python-support failed few months ago, so
here's another proposal to improve the Python situation in Debian. It's nothing new
actually, just a compilation of previous Matthias' and Joss' ideas and few of my
own.
The main idea is to ship symlinks (not .pyc files!) in binary packages as
Matthias proposed back in February and thus get rid of both helper tools (most
problems with current tools occur at install / upgrade time). I want to keep it
as simple as possible and let maintainers customize it in maintainer or
rtinstall/rtupdate/rtremove scripts (if really needed).
Advantages:
* a lot less opportunities to break a system while installing / upgrading
Python packages,
* no more problems with packages that provide the same namespace and use
different helper tool,
* Python modules available out of the box (think about daemons),
* no more "which tool should I use?" or "how do they differ?" questions ;-)
The main disadvantage of this approach is that architecture independent packages
will have to be rebuilt once new Python version will be added to the supported ones
(I think we can avoid a rebuild if a version is removed from the supported ones).
Luk (our release manager) told me that binNMUs for arch:all packages will be
possible in Debian at some point, but it's a matter or months or even years, so
for now - manual NMUs will be needed. I think it will be easy to detect which
packages need one basing on Contents file and Depends field.
Another disadvantage is the fact that pysupport's namespace feature will not be
supported. If we want to keep it simple and let dpkg know about all the files -
new *-common package with all common __init__.py (and other if needed) files
will have to be provided, just like we did before pysupport 0.7. Otherwise
removing .pyc files will not be a trivial thing to do. Note that there are lots
of problems with this feature anyway (upstreams tend to use site-packages for
things that don't belong there), so removing it will improve the situation at
the end, IMHO.
I'll try to make dh_python a drop in replacement for dh_pycentral and
dh_pysupport (i.e. tools used at build time) for most packages, but I don't
consider it to be a main goal. Some packages will have to be manually updated
(f.e. the ones that use pysupport's namespace feature).
I want the tool to be team maintained, I didn't decide yet if new Alioth
project should be created or if we should use DPMT or PAPT repo for this,
though.
Short overview of how things will work with new dh_python:
==========================================================
build time:
^^^^^^^^^^^
* files installed into the standard location[1] or into private directory
* dh_python will:
- move .py files from SL[1] into /usr/share/py{,3}shared,
- move .so files to /usr/lib/pythonX.Y/*-packages if /usr/local was used,
- create symlinks for all py{,3}shared content in /usr/lib/pythonX.Y/*-packages
*if* files are the same for all Python versions in SL[1] (if not, leave
original files) where X.Y are requested Python versions,
- create simple maintainer scripts with pycompile and pyclean commands
and (if needed) rtupdate script (for private directories that use default
Python version). Private modules that cannot be used with default Python
version will get additional pycompile command with all needed arguments, like
minimum required Python version or hardcoded one in both, maintainer script
and rtupdate one
[1] SL = standard distutils/setuptools location:
/usr/{,local}/lib/pythonX.Y/{site,dist}-packages/
installation time:
^^^^^^^^^^^^^^^^^^
* maintainer script should bytecompile pyc files for all provided
symlinks / private directories if given Python version is installed
* user installs new pythonX.Y package:
- bytecompile related symlinks (pycompile -V X.Y)
* user removes pythonX.Y package:
- remove all pythonX.Y's .pyc files (pyclean -V X.Y)
- all packages with private directories that use this Python version will be removed
by dpkg (Depends field will do its job)
* default Python version changes:
- rtupdate scripts for packages with private modules that support it (i.e.
the ones without hardcoded Python version and without private extensions)
invoked
examples:
=========
example 1 - public modules/extensions only
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
$ python2.4 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py
debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py
debian/python-foo/usr/lib/python2.4/site-packages/spam.py
$ python2.5 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py
debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py
debian/python-foo/usr/lib/python2.5/site-packages/spam.py
$ python2.6 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/__init__.py
debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/_foo.so
debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/bar.py
debian/python-foo/usr/local/lib/python2.6/dist-packages/spam.py
$ python3.1 ./3.x/setup.py install --root=debian/python3-foo/
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py
$ dh_python
# gets minimum required Python version from debian/pyversions or from
# XS-Python-Version or (this part is new) from Build-Depends(-Indep);
# ignores private dirs
debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
debian/python-foo/usr/lib/python2.4/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py
debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
debian/python-foo/usr/lib/python2.5/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py
debian/python-foo/usr/lib/python2.6/dist-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.6/dist-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
debian/python-foo/usr/lib/python2.6/dist-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
debian/python-foo/usr/lib/python2.6/dist-packages/foo/spam.py # not a symlink as it differes from Python2.4 and 2.5's version
debian/python-foo/usr/share/pyshared/foo/__init__.py
debian/python-foo/usr/share/pyshared/foo/bar.py
debian/python-foo/usr/share/pyshared/spam.py
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py -> /usr/share/py3shared/foo/__init__.py
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py -> /usr/share/py3shared/foo/bar.py
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/spam.py -> /usr/share/py3shared/foo/spam.py
debian/python3-foo/usr/share/py3shared/foo/__init__.py
debian/python3-foo/usr/share/py3shared/foo/bar.py
debian/python3-foo/usr/share/py3shared/spam.py
$ grep foo debian/python-foo.postinst
pycompile foo spam
$ grep foo debian/python3-foo.postinst
py3compile foo spam
$ grep foo debian/python-foo.prerm
pyclean foo spam
$ grep foo debian/python3-foo.prerm
py3clean foo spam
example 2 - private modules/scripts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# bar uses default Python version
$ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir1/ --install-scripts=/usr/share/privatedir1
debian/foo/usr/share/privatedir1/bar/__init__.py
debian/foo/usr/share/privatedir1/run.py
# spam uses Python 2.4
$ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir2/ --install-scripts=/usr/share/privatedir2
debian/foo/usr/share/privatedir2/spam/__init__.py
debian/foo/usr/share/privatedir2/run.py
# egg uses Python >= 2.6
$ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir3/ --install-scripts=/usr/share/privatedir3
debian/foo/usr/lib/privatedir3/egg/__init__.py
debian/foo/usr/lib/privatedir3/run.py
# baz uses Python >= 2.5 and has private extension
$ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir4/ --install-scripts=/usr/share/privatedir4
debian/foo/usr/lib/privatedir4/baz/__init__.py
debian/foo/usr/lib/privatedir4/baz/baz.so
debian/foo/usr/lib/privatedir4/run.py
$ dh_python /usr/share/privatedir1 # ignores SL; reads debian/pyversions or XS-Python-Version as -V is not given
debian/foo/usr/share/privatedir1/bar/__init__.py
debian/foo/usr/share/privatedir1/run.py
# Python not hardcoded in shebang and -V not used so recompile once default
# Python version will change, minimum required version hardcoded in the script
debian/foo/usr/share/python/runtime.d/python-foo.rtupdate
$ grep python:Depends debian/python-foo.substvars
python:Depends=python
$ dh_python /usr/share/privatedir2 -V 2.4
debian/foo/usr/share/privatedir2/bar/__init__.py
debian/foo/usr/share/privatedir2/run.py
# note that rtupdate script is not touched, python2.4 is added to the Depends
# instead and maintainer script contains hardcoded python2.4
debian/foo/usr/share/python/data/python-foo
$ grep python:Depends debian/python-foo.substvars
python:Depends=python, python2.4
$ grep privatedir debian/python-foo.postinst
pycompile /usr/share/privatedir1
pycompile /usr/share/privatedir2 -V 2.4
$ grep privatedir debian/python-foo.prerm
pyclean /usr/share/privatedir1 /usr/share/privatedir2
$ dh_python /usr/share/privatedir3 -V 2.6+
debian/foo/usr/share/privatedir3/egg/__init__.py
debian/foo/usr/share/privatedir3/run.py
debian/foo/usr/share/python/runtime.d/python-foo.rtupdate
$ grep python:Depends debian/python-foo.substvars
python:Depends=python, python2.4, python (>=2.6) | python2.6
$ grep privatedir debian/python-foo.postinst
pycompile /usr/share/privatedir1
pycompile /usr/share/privatedir2 -V 2.4
pycompile /usr/share/privatedir3 -V 2.6+
$ grep privatedir debian/python-foo.prerm
pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3
$ dh_python /usr/share/privatedir4 -V 2.5+ # note the "+" here and lack of it later
debian/foo/usr/lib/privatedir4/baz/__init__.py
debian/foo/usr/lib/privatedir4/baz/baz.so
debian/foo/usr/lib/privatedir4/run.py
$ grep python:Depends debian/python-foo.substvars
python:Depends=python, python2.4, python (>=2.6) | python2.6, python2.5
$ grep privatedir debian/python-foo.postinst
pycompile /usr/share/privatedir1
pycompile /usr/share/privatedir2 -V 2.4
pycompile /usr/share/privatedir3 -V 2.6+
pycompile /usr/lib/privatedir4 -V 2.5 # note that there's no "+" here (due to private extension)
$ grep privatedir debian/python-foo.prerm
pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3 /usr/lib/privatedir4
--
-=[ Piotr Ożarowski ]=-
-=[ http://www.ozarowski.pl ]=-
Attachment:
signature.asc
Description: Digital signature