Re: new dh_python proposal
Le vendredi 15 janvier 2010 à 11:58 +0100, Piotr Ożarowski a écrit :
> Here's updated dh_python proposal. Main change since my last mail:
> pycompile/pyclean will no longer search for files to compile/remove
> in public directories as Matthias will fix #552595 using triggers so I don't
> have to care about pycentral bugs anymore and dpkg -L seems to be fast enough
> to be used in maintainer scripts (BTW: my previous proposal had a typo: "prerm"
> should be replaced with "postrm", I guess everyone figured that out from the
> context…).
There is not much change since the first proposal, so I’m just sending
an updated version of the comments I already sent privately.
> Again, the main idea is to ship files in official site/dist-packages location
> or symlink files from /usr/share/py{,3}shared directory there (if files can be
> shared).
>
> This means Architecture:all packages will need sourceful uploads once list of
> supported Python versions will change (binNMUs do not touch architecture
> independent packages and even if they'd do, that still leaves the
> "=${source:Version}" problem).
This is an *extremely* bad idea. It feels like a huge regression in
terms of testing migration. It means a lot of unnecessary work,
precisely the kind of work we want to avoid.
> Advantages:
> * dpkg is aware of all .py files (including symlinks) and their locations (it
> isn't true for pysupport/pycentral),
> * no need for helper in Depends and Build-Depends - I want dh_python and
> pycompile/pyclean to be shipped in python packages,
> * broken modules that use __file__ incorrectly will work without problems,
True, but the right thing to do in the long term is to fix them:
http://malsain.org/~joss/python.jpg
> * less opportunities to break a system while installing / upgrading Python
> packages:
> - problems with creating/removing symlinks in pycentral (see f.e. #552595),
And you will run into new problems with a new tool. Although this can be
better if the maintainer is simply responsive enough.
> - starting/stopping daemons problems (very long downtimes due to byte
> compilation via triggers) in pysupport,
This is only a problem for daemons using namespace packages, which you
can count with one hand’s fingers.
> * no more compile errors at install time in public directories, all Python
> versions tested at build time,
> * Python modules available out of the box (think about daemons and upgrades)
>
>
> Short overview of how things will work with new dh_python:
> ==========================================================
>
> build time:
> ^^^^^^^^^^^
> * files installed into the standard location[1] or into private directory.
> It's up to maintainer which files will be installed for which Python version,
> by default every package providing public module should build/install for all
> `pyversions -vs` versions.
> * dh_python will:
> - move files to /usr/lib/pythonX.Y/*-packages if /usr/local or
> /usr/lib/python2.X/site-packages (for X >= 6) is used
> (i.e. --install-layout or --prefix can be skipped in most cases),
> - copy all files that can be shared to /usr/share/py{,3}shared and replace
> the ones in *-packages with a symlink (if package supports only one Python
> version, there's no need to use pyshared),
The way you describe it makes it sound like architecture: all packages
will now need to depend on python-all-dev and install their files for
all supported Python versions so that symlinks are created this way.
This is another big regression for packages that don’t use distutils /
setuptools / whatever crap upstreams invent. Currently, for
architecture-independent packages, you can use whatever build system is
shipped, close your eyes, run dh_py* and get done with it. With your
proposal, it means that *all* archirecture-independent packages would
require adaptation to loop over supported Python versions.
This is a huge task, and a complete waste of time. Don’t forget that
most people that were motivated to work on Python packages are tired of
endless changes that had to be done in hundreds of packages because of
thoughtless changes in core Python packages.
> - create simple maintainer scripts with pycompile and pyclean
> commands and (if needed) rtupdate script (for private directories that
> use default Python version and will most probably work with next default
> version). Private modules that cannot be used with default Python version
> will get additional pycompile command with all needed arguments, like
> minimum required Python version or hardcoded version in both, maintainer
> and rtupdate scripts,
This part looks worrysome to me. Putting the complexity of the
installation/cleanup actions in auto-generated maintainer scripts (and
now auto-generated rtupdate scripts) sounds like reproducing one of the
core design errors of python-central. The more complexity you put in
auto-generated scripts, the less chances you leave for the system to
clean itself up by just upgrading the helper scripts in case of a bug.
This is the kind of mistakes that leads to unmaintainable messes like
defoma, to quote another one. The debhelper maintainer only accepts the
simplest of maintainer script snippets now, for this very reason.
> - add substvar for ${python:Versions} (used in XB-Python-Version, i.e. the
> "make release managers happy" field (this field will be used by RMs to
> track transitions)
I bet it will not, but you can still hope so :)
> - add substvar for ${python:Provides}
Don’t forget that we must get rid of the Provides: crap in most
packages. It’s okay to generate the substvar, though, but maybe with a
warning.
> example 1 - public modules/extensions only
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> $ grep foo debian/python-foo.postinst
> dpkg -L python-foo | pycompile
> $ grep foo debian/python-foo.prerm
> dpkg -L python-foo | pyclean
Ugh. Use 'pycompile python-foo', pretty please.
Also note that using dpkg is another source of inconsistencies because
of the way you filter out file listings. For example if you discover in
a new version that a directory has to be excluded from byte-compilation,
you will need more and more code to clean up.
A general issue is that there is no global state for byte-compilation,
which means you cannot simply clean-up the mess if there is a broken
version.
> example 2 - private modules/scripts
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> $ grep privatedir debian/mypackage.postinst
> dpkg -L mypackage | pycompile /usr/share/privatedir1
> dpkg -L mypackage | pycompile /usr/share/privatedir2 -V 2.4
Ugh again. How about that?
pycompile -p mypackage /usr/share/privatedir1
Or even better, always "pycompile mypackage", and ship the list of
directories to byte-compile in a file.
Overall, I’m still dissatisfied with this proposal.
First of all, for private modules, it has absolutely zero interest, and
only consists in regressions. I don’t see the point in the proposed
changes for private modules, apart from introducing a third helper tool
with less features than both existing ones.
For public modules, I’m more mitigated. For architecture: any packages,
it is certainly justified to treat the .py files shipped in a similar
way to the .so in the same package, as anyway the list of supported
Python versions is hardcoded. We would still need to fix the
byte-compilation issues, of course.
For architecture: all packages, I’m still not convinced. Having to
rebuild these Python packages means roughly 3 times as many packages to
rebuild during transitions as there are now. This is going to make
transitions a *lot* more complicated. This means much more work for
everyone, including the RT, at times of changing the list of supported
Python versions. We also lose the current potential to have a large
subset of the archive immediately available for a new Python version
when it becomes ready.
I’m also still wondering which exact problem this proposal is trying to
fix. Currently the only thing it does seem to address is a very specific
kind of issue that Loïc explained to me, which happens in Python-based
package managers: when you upgrade the Python packages the package
manager depends on, including a change in the default Python version (if
the default Python version doesn’t change, all modules remain available
during the upgrade, with both -central and -support), and the upgrade
process is interrupted, the package manager might not be able to
restart. There are other ways to fix this bug; the most obvious way
being to apply something similar to your proposal, but only to a very
small set of Python packages, namely those on which the package managers
depend. For the rest of the archive, I just don’t see the point.
Introducing this way of doing things in a subset of Python packages
sounds interesting - either for a small set, or for all
architecture-dependent packages. Doing it archive-wide sounds like going
straight into a wall to me.
Cheers,
--
.''`. Josselin Mouette
: :' :
`. `' “You did never write something helpful
`- besides your trolling” -- Jörg Schilling
Reply to: