Re: new dh_python proposal
Hello, Piotr.
In short - your post is long. Brain is limited. Better place an
overview in wiki and discuss in parts.
Before discussing your proposal I would really appreciate if somebody
from insiders could describe situation in Debian Python in all
possible detail including the history of development. I believe that
this piece of work is absolutely necessary, because the problem of
repackaging Python packages for Debian and maintaining integrity
between installed packets, packages and Python versions is too complex
to keep every detail in mind unless you can devote your full time to
the problem.
One way to make the status more obvious is to answer to emails in this
group, and merge answers into wiki, so that the next time you can
point a person to the exact place of documentation.
You say that attempt to merge -cental and -support failed, but didn't
mention why. You say that current tools have problems that occur at
install/upgrade time, but do not mention these problems explicitly. So
it is impossible to say whatever your ideas solve current problems and
won't add new.
Referencing rtinstall/rtupdate/rtremove scripts is cool for a seasoned
Debian developer, but for a Python developer it means nothing. In
other words - not only Python things are obscure for Debian
developers, but Debian stuff is also a mystery for Python masters, so
these things should be explained symmetrically for both communities.
The chances for Debian Python Packaging experts to pop up are
magnitude less if we won't explain the situation in detail.
Now about the proposal (from newcomer's point of view):
dh_python is a shell script -- I have a strong belief that Python
package automation scripts should be written in Python, there is no
need to learn bashes when you program Python - do not expect that
package maintainers will be able to debug their scripts in shell
language.
The description that dh_python will do at a build time looks like a
solution, but again - it doesn't specify what problem is being solved.
Perhaps I expected to see more high-level definition.
May I ask a question - is there a difference between installation of
Python Modules and Python Applications? Does the problem set you are
trying to resolve with this proposal include the difference? Have you
considered distributing Python Applications via virtualenv? How this
proposal handles virtualenv installations?
In conclusion my opinion is that problem set is not defined well
enough to propose a solution or estimate if it will be effective both
for current flow and for future ideas. I would start from wiki.
--anatoly t.
On Sun, Aug 2, 2009 at 6:53 AM, Piotr Ożarowski<piotr@debian.org> wrote:
> My attempt to merge python-central and python-support failed few months ago, so
> here's another proposal to improve the Python situation in Debian. It's nothing new
> actually, just a compilation of previous Matthias' and Joss' ideas and few of my
> own.
>
> The main idea is to ship symlinks (not .pyc files!) in binary packages as
> Matthias proposed back in February and thus get rid of both helper tools (most
> problems with current tools occur at install / upgrade time). I want to keep it
> as simple as possible and let maintainers customize it in maintainer or
> rtinstall/rtupdate/rtremove scripts (if really needed).
>
> Advantages:
> * a lot less opportunities to break a system while installing / upgrading
> Python packages,
> * no more problems with packages that provide the same namespace and use
> different helper tool,
> * Python modules available out of the box (think about daemons),
> * no more "which tool should I use?" or "how do they differ?" questions ;-)
>
> The main disadvantage of this approach is that architecture independent packages
> will have to be rebuilt once new Python version will be added to the supported ones
> (I think we can avoid a rebuild if a version is removed from the supported ones).
> Luk (our release manager) told me that binNMUs for arch:all packages will be
> possible in Debian at some point, but it's a matter or months or even years, so
> for now - manual NMUs will be needed. I think it will be easy to detect which
> packages need one basing on Contents file and Depends field.
>
> Another disadvantage is the fact that pysupport's namespace feature will not be
> supported. If we want to keep it simple and let dpkg know about all the files -
> new *-common package with all common __init__.py (and other if needed) files
> will have to be provided, just like we did before pysupport 0.7. Otherwise
> removing .pyc files will not be a trivial thing to do. Note that there are lots
> of problems with this feature anyway (upstreams tend to use site-packages for
> things that don't belong there), so removing it will improve the situation at
> the end, IMHO.
>
> I'll try to make dh_python a drop in replacement for dh_pycentral and
> dh_pysupport (i.e. tools used at build time) for most packages, but I don't
> consider it to be a main goal. Some packages will have to be manually updated
> (f.e. the ones that use pysupport's namespace feature).
>
> I want the tool to be team maintained, I didn't decide yet if new Alioth
> project should be created or if we should use DPMT or PAPT repo for this,
> though.
>
>
> Short overview of how things will work with new dh_python:
> ==========================================================
>
> build time:
> ^^^^^^^^^^^
> * files installed into the standard location[1] or into private directory
> * dh_python will:
> - move .py files from SL[1] into /usr/share/py{,3}shared,
> - move .so files to /usr/lib/pythonX.Y/*-packages if /usr/local was used,
> - create symlinks for all py{,3}shared content in /usr/lib/pythonX.Y/*-packages
> *if* files are the same for all Python versions in SL[1] (if not, leave
> original files) where X.Y are requested Python versions,
> - create simple maintainer scripts with pycompile and pyclean commands
> and (if needed) rtupdate script (for private directories that use default
> Python version). Private modules that cannot be used with default Python
> version will get additional pycompile command with all needed arguments, like
> minimum required Python version or hardcoded one in both, maintainer script
> and rtupdate one
>
> [1] SL = standard distutils/setuptools location:
> /usr/{,local}/lib/pythonX.Y/{site,dist}-packages/
>
> installation time:
> ^^^^^^^^^^^^^^^^^^
> * maintainer script should bytecompile pyc files for all provided
> symlinks / private directories if given Python version is installed
> * user installs new pythonX.Y package:
> - bytecompile related symlinks (pycompile -V X.Y)
> * user removes pythonX.Y package:
> - remove all pythonX.Y's .pyc files (pyclean -V X.Y)
> - all packages with private directories that use this Python version will be removed
> by dpkg (Depends field will do its job)
> * default Python version changes:
> - rtupdate scripts for packages with private modules that support it (i.e.
> the ones without hardcoded Python version and without private extensions)
> invoked
>
>
> examples:
> =========
>
> example 1 - public modules/extensions only
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> $ python2.4 ./setup.py install --root=debian/python-foo/
> debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py
> debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
> debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py
> debian/python-foo/usr/lib/python2.4/site-packages/spam.py
>
> $ python2.5 ./setup.py install --root=debian/python-foo/
> debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py
> debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so
> debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py
> debian/python-foo/usr/lib/python2.5/site-packages/spam.py
>
> $ python2.6 ./setup.py install --root=debian/python-foo/
> debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/__init__.py
> debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/_foo.so
> debian/python-foo/usr/local/lib/python2.6/dist-packages/foo/bar.py
> debian/python-foo/usr/local/lib/python2.6/dist-packages/spam.py
>
> $ python3.1 ./3.x/setup.py install --root=debian/python3-foo/
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py
>
> $ dh_python
> # gets minimum required Python version from debian/pyversions or from
> # XS-Python-Version or (this part is new) from Build-Depends(-Indep);
> # ignores private dirs
> debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
> debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
> debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
> debian/python-foo/usr/lib/python2.4/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py
> debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so
> debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
> debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
> debian/python-foo/usr/lib/python2.5/site-packages/foo/spam.py -> /usr/share/pyshared/foo/spam.py
> debian/python-foo/usr/lib/python2.6/dist-packages/foo/_foo.so
> debian/python-foo/usr/lib/python2.6/dist-packages/foo/__init__.py -> /usr/share/pyshared/foo/__init__.py
> debian/python-foo/usr/lib/python2.6/dist-packages/foo/bar.py -> /usr/share/pyshared/foo/bar.py
> debian/python-foo/usr/lib/python2.6/dist-packages/foo/spam.py # not a symlink as it differes from Python2.4 and 2.5's version
> debian/python-foo/usr/share/pyshared/foo/__init__.py
> debian/python-foo/usr/share/pyshared/foo/bar.py
> debian/python-foo/usr/share/pyshared/spam.py
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py -> /usr/share/py3shared/foo/__init__.py
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py -> /usr/share/py3shared/foo/bar.py
> debian/python3-foo/usr/lib/python3.1/dist-packages/foo/spam.py -> /usr/share/py3shared/foo/spam.py
> debian/python3-foo/usr/share/py3shared/foo/__init__.py
> debian/python3-foo/usr/share/py3shared/foo/bar.py
> debian/python3-foo/usr/share/py3shared/spam.py
>
> $ grep foo debian/python-foo.postinst
> pycompile foo spam
>
> $ grep foo debian/python3-foo.postinst
> py3compile foo spam
>
> $ grep foo debian/python-foo.prerm
> pyclean foo spam
>
> $ grep foo debian/python3-foo.prerm
> py3clean foo spam
>
>
> example 2 - private modules/scripts
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> # bar uses default Python version
> $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir1/ --install-scripts=/usr/share/privatedir1
> debian/foo/usr/share/privatedir1/bar/__init__.py
> debian/foo/usr/share/privatedir1/run.py
>
> # spam uses Python 2.4
> $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir2/ --install-scripts=/usr/share/privatedir2
> debian/foo/usr/share/privatedir2/spam/__init__.py
> debian/foo/usr/share/privatedir2/run.py
>
> # egg uses Python >= 2.6
> $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir3/ --install-scripts=/usr/share/privatedir3
> debian/foo/usr/lib/privatedir3/egg/__init__.py
> debian/foo/usr/lib/privatedir3/run.py
>
> # baz uses Python >= 2.5 and has private extension
> $ python ./setup.py install --root=debian/foo/ --install-lib=/usr/share/privatedir4/ --install-scripts=/usr/share/privatedir4
> debian/foo/usr/lib/privatedir4/baz/__init__.py
> debian/foo/usr/lib/privatedir4/baz/baz.so
> debian/foo/usr/lib/privatedir4/run.py
>
> $ dh_python /usr/share/privatedir1 # ignores SL; reads debian/pyversions or XS-Python-Version as -V is not given
> debian/foo/usr/share/privatedir1/bar/__init__.py
> debian/foo/usr/share/privatedir1/run.py
> # Python not hardcoded in shebang and -V not used so recompile once default
> # Python version will change, minimum required version hardcoded in the script
> debian/foo/usr/share/python/runtime.d/python-foo.rtupdate
>
> $ grep python:Depends debian/python-foo.substvars
> python:Depends=python
>
> $ dh_python /usr/share/privatedir2 -V 2.4
> debian/foo/usr/share/privatedir2/bar/__init__.py
> debian/foo/usr/share/privatedir2/run.py
> # note that rtupdate script is not touched, python2.4 is added to the Depends
> # instead and maintainer script contains hardcoded python2.4
> debian/foo/usr/share/python/data/python-foo
>
> $ grep python:Depends debian/python-foo.substvars
> python:Depends=python, python2.4
>
> $ grep privatedir debian/python-foo.postinst
> pycompile /usr/share/privatedir1
> pycompile /usr/share/privatedir2 -V 2.4
>
> $ grep privatedir debian/python-foo.prerm
> pyclean /usr/share/privatedir1 /usr/share/privatedir2
>
> $ dh_python /usr/share/privatedir3 -V 2.6+
> debian/foo/usr/share/privatedir3/egg/__init__.py
> debian/foo/usr/share/privatedir3/run.py
> debian/foo/usr/share/python/runtime.d/python-foo.rtupdate
>
> $ grep python:Depends debian/python-foo.substvars
> python:Depends=python, python2.4, python (>=2.6) | python2.6
>
> $ grep privatedir debian/python-foo.postinst
> pycompile /usr/share/privatedir1
> pycompile /usr/share/privatedir2 -V 2.4
> pycompile /usr/share/privatedir3 -V 2.6+
>
> $ grep privatedir debian/python-foo.prerm
> pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3
>
> $ dh_python /usr/share/privatedir4 -V 2.5+ # note the "+" here and lack of it later
> debian/foo/usr/lib/privatedir4/baz/__init__.py
> debian/foo/usr/lib/privatedir4/baz/baz.so
> debian/foo/usr/lib/privatedir4/run.py
>
> $ grep python:Depends debian/python-foo.substvars
> python:Depends=python, python2.4, python (>=2.6) | python2.6, python2.5
>
> $ grep privatedir debian/python-foo.postinst
> pycompile /usr/share/privatedir1
> pycompile /usr/share/privatedir2 -V 2.4
> pycompile /usr/share/privatedir3 -V 2.6+
> pycompile /usr/lib/privatedir4 -V 2.5 # note that there's no "+" here (due to private extension)
>
> $ grep privatedir debian/python-foo.prerm
> pyclean /usr/share/privatedir1 /usr/share/privatedir2 /usr/share/privatedir3 /usr/lib/privatedir4
>
> --
> -=[ Piotr Ożarowski ]=-
> -=[ http://www.ozarowski.pl ]=-
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkp1cAsACgkQB01zfu119Zme1gCgs6i6uNMIyLSWT6BAb+05wdK5
> /7oAn00BAOduJLRdv64PfcnGGHVdsTdv
> =JLqk
> -----END PGP SIGNATURE-----
>
>
Reply to: