Re: dh_python and python policy analysis
On Sat, 12 Aug 2006 17:34:06 +0200, Matthias Klose <doko@cs.tu-berlin.de> said:
> Manoj Srivastava writes:
>> policy document. The current version, and future updates, are to be
>> found at http://www.golden-gryphon.com/software/manoj-policy/
Unfortunately, you are commenting on an old version of the
document, so some comments refer to issue that have already been
addressed.
>> Extension Module
>>
>> Extensions are C code compiled and linked against a specific
>> version of the libpython library, and so can only be used by one
>> version of Python.
> There should be no reason to link the extension against the python
> library. Usually many extensions which are developed upstream on
> Windows do link by default to libpython. Other extensions linking
> against libpython are those with build infrastructure maybe
> predating distutils. python-semanage is an example (and should not
> link using -z defs).
I'll add these in general notes, or perhaps in the extension
specific sections (since compilation rules do not belong in
the classification section).
> Another thing to mention here is a "Python package", a directory
> containing an __init__.py file plus modules and extensions.
This is mentioned in general notes; I did not think it was a
useful fifth classification, since we do not actually treat it very
differently.
>> Public
>>
>> Public modules are available for use in other Python scripts or
>> modules using the import directive. They are installed in one of
>> the directories:
>>
> maybe shorten that to "all directories in sys.path"; not sure if an
> explicit list of directories is needed.
Not all paths in sys.path are oermitted in packages (like the
path ending in .zip).
>> Packages may install public Python modules in directories specific
>> to Python packaging utilities -- which specify the directories
>> under which such modules should be droppped, and the the structure
>> of these directories is defined by the utilities themselves. Please
>> note that these directories are not in the path for Python, and are
>> not available for modules to be imported from. At the time of
>> writing, such uility specific directories include:
> ^^
>>
>> /usr/share/pycentral /usr/share/python-support
> These location are tool specific and should not be referenced
> explicitely in the packaging scripts (debian/rules)
You have just made all my selinux python packages violate this
line :). I use update-python-modules in postinst and prerm, but I do
not use any debhelper modules, and I should not be required to do
so.
I don't think that implicitly mandating the use of the helper
scripts dh_(pysupport|pycentral|python) should belong in policy.
>> 2. Goals of the new Python policy
>>
> No, not the whole design goal. Although the document is titled
> "developer's view", the other goals should be mentioned as well.
> These are meant to work around processes in debian which are
> currently suboptimal, but unlikely to change.
[SNIP lots of good stuff]
I'll add these to the document.
>> New python version introduced
>>
>> * Most pure Python modules with no restrictions on the
>> version of Python supported, and those pure Python modules that
>> only have a lower bound on the versions of python supported (for
>> example, "2.3-", or "all"), would require no
> "2.3-" -> ">= 2.3", not "all". the range notation cannot express
> things like "2.2, >= 2.4". The use case for the latter is the jython
> package (now removed from testing) still at an implementation level
> of the corresponding cpython version, with i.e. 2.3 not a supported
> python version anymore. So in the following text "set of versions",
> instead of "range of versions" should be used.
OK, done.
>> The new policy also reduces the numbers of packages in the archive,
>> by supporting multiple versions of Python in the same binary
>> package (at the cost of increased size of that one package, but it
>> should still result in space saving.)
> Maybe mention the two cases, where the package size increases:
> - extension modules
> - pure modules where different versions of the upstream package are
> shipped and are directly installed into /usr/lib/pythonX.Y/
OK.
>> 3. Recipe for developers
>> 3.1.1. Python versions supported by the source
>>
>> The XS-Python-Version field in debian/control specifies the
>> versions of Python supported by the package [30][1]. While this is
>> a requirement only if using the utility package python-central
>> (python-support, for example, prefers debian/pyversions), setting
>> this is "appreciated" in any case, according to the [31]new policy
>> wiki[32][2]. This is used to track packages during Python
>> transitions.
> ... and test rebuilds.
OK.
>> This can be a single version, or one or more of a list of
>> non-overlapping ranges. The lowest range may optionally omit a low
>> end, and the highest range may optionally omit an upper end. In
>> other words, the overall range may be open ended. The ranges are
>> often matched to the set of all known Python version that have
>> existed, and the supported set is the intersection of the known
>> versions of python and the range specification.
> XS-Python-Version can have the values "current" and "current_ext" as
> well (plus the list of ranges), which will expand to "current", if
> the package does not have any extensions and can be used with
> another python default version without a new upload.
This seems like tool specific information to me. Am I wrong
that pycentral is the helper tool that utilizes this information?
> It's replaced by the version number of the current default version
> in the Python:Versions substitution variable.
Again, this document does not mention python:Versions filed,
that belongs in helper tool documentation.
> "current_ext" normally only needs to be used for packages having a
> private extension module. dh_pysupport doesn't use this information,
> but requires the developer to explicitely pass the directory
> containing the extension module.
So am I correct in understanding that "current" is a setting
used by pycentral internally? If so, there are footnotes that mention
the "current" setting in the appropriate locations, but I did not
include what I thought was an helper tool internal variable in
policy. I understand that the information is used by pycentral, by
it breaks the semantics of XS-Python-Version, and there are a number
of other reason that I found compelling in that section.
>> 1. If the current version of Python is supported by the package,
>> then:
>>
>> * For packages with private modules or private extensions
>> compiled for the current python version and for applications using
>> /usr/bin/python, this should be set to the string "all" (or "-", in
>> the case of debian/pyversions). [33][3] If the module doesn't work
>> with all Python version, the range of versions supported should be
>> used [34][4]
>>
>> * For packages with public modules, this should be set to the
>> string "all" (or "-", in the case of debian/pyversions), unless not
>> all versions of Python are supported (in which case the setting
>> should specify the versions or range of versions actually
>> supported, like ">= 2.4" or ">= 2.2, << 2.y".
> again, "current" is a legitimate value.
It is mentioned in a footnote that pycentral would use current
in this place. Do you think this is not enough?
>> 3.1.2. Depends:
>>
>> The rules for calculating the dependencies a package has are
>> simple.
>>
>> 1. If a script invokes /usr/bin/pythonX.Y, than the package must
>> depend
>> on pythonX.Y. This is because no amount of automatic byte compiling
>> would ever get rid of the requirement that /usr/bin/pythonX.Y has
>> to be present for the script to function.
> I think, that is too strict. The current behaviour is depending on
> the dh_python implementation scanning all files for that interpreter
> line. Consider a package with scripts in /usr/bin: foo, foo2.3,
> foo2.4, calling python, python2.3 and python2.4, which would lead to
> a dependency on all supported python versions. The scripts work for
> the default python version, for the non-default python versions only
> if the corresponding pythonX.Y package is installed.
I would consider a package buggy if it ships scripts that do
not work out of the box; if some of the scripts require an alternate
version of python, then by all means that should be expressed in a
dependency relationship.
So, if it is different version of the same script, compiled
for different version of python, I would suggest either extracting
the version dependent bits into a module, or having different
packages using alternatives.
If I have a tool that uses /usr/bin/foo, and the default
version of python changes, then /usr/bin/foo breaks, as does my
tool. This is a very fragile situation, and does nothing to help
transitions.
>> 2. For package that contains extensions, the range of Python
>> versions
>> required has to be restricted to Python versions for which the
>> extensions have been built and shipped in the package. For packages
>> with private extension modules, this means that the range of python
>> versions it depends on has to be set to whatever version of Python
>> was used during the build process (since private extension module
>> packages can only be compiled for one version of Python at any
>> time).
> not limited to extensions only, but applies to any other package
> hardcoding some kind of version information in the packaged files.
What kinds of hardcoding are you hinting at? We already take
care of scripts that invoke specific versions of python above, and
there is language that says that if a module in some manner depends
on specific versi0ons of python you must restrict the versions of
ython supported.
This clause is for cases when there is no other version based
dependency, but if we are shipping extensions (usually public), then
the versions of python the extension was compiled against
automsatically further limits the version we are supporting.
Public pure python modules are compiled on install, so that
does not have a bearing here, we already mention that private modules
restrict the supported version to a single version elsewhere.
What cases are missing?
>> 3.1.3. Provides
>>
>> Packages with public modules and extensions should be named, or
>> should provide, python-foo. Pure Python public modules that support
>> all Python versions need not have a Provides field.
> ... unless there is an application using a non-default python
> version using this module. or else you require the application
> depending on any indirect dependency of python-foo.
Hmm. Two things: if application X requires my pure python
public module (called, say, python-foo), and uses some specific
version of python, why can't it depend on just python-foo Why do I
have to provide pythonX.Y-foo?
Also, as a maintainer of python-foo, I can't know when such an
application would be created, and we are trying to minimize
reuploads of packages -- so either one provides all such
pythonX.Y-foo at the get go, and reupload at every new python version
or dropping of the old version -- or we upload every time some app is
uploaded that may require yet abother X.y, and when we drop a version
of Python.
So I think tis is not a good rule t change, unless there is a
use case that demonstrates why it is needed.
>> 3.1.4. Build-Depends:
>>
>> If the package provides public extension modules, then build
>> depending on "python-all-dev (>= 2.3.5-11)" shall ensure that all
>> the >pythonX.Y-dev packages are available during building.
> Limiting this to public extension modules is not robust, if a pure
> python module hardcodes version information.
A pure python module is not compiled at build time. Only
extensions need be the -dev package to link against.
>> 3.2. Deprecating "current" in versions supported
> There is currently no agreement about that.
This is not a document about current agreements. This is a
document that tries to do the right thing, and overloading
XS-Python-Version does not appear to be the way to go.
>> Currently, the string "current" in the field XS-Python-Version is
>> used by python-central to indicate that the package contains
>> private modules,
> that is wrong.
Does dh_pysupport also use it?
>> and explicitly state that the package is only built for the current
>> Python version, and not for any other version supported in
>> Debian. This is flawed, for the following reasons:
>>
>> * The value corresponds to the version of Python the package is
>> currently built for; but in all other cases the semantics of the
>> XS-Python-Version field is to indicate which versions of Python are
>> supported by the package, and indicates compatibility, not the
>> version it is currently built against. So this special case breaks
>> the semantics of the field.
> that is the reason that "current, >= 2.3" is supported.
And I think we mention this in footnotes, since it is helper
tool specific usage.
>> * The information conveyed by this field is redundant; it should be
>> clear that the package contains private modules, based on the
>> directories the modules are shipped in, and also the fact that it
>> ought to build depend on python-dev and not on python-all-dev.
> python-all-dev does not differentiate between "current" and
> "current_ext"; dh_pysupport has an implementation decision to hide
> that information as a parameter to dh_pysupport. That information
> should be visible in the control file.
Can you elaborate on why this should be in the control file?
>> * The semantics of "current" are not fixed, since they depend on
>> the
>> contents of the package python-defaults, and are ill suited for the
>> debian/control file.
> again, there's nothing wrong about that, as you can use current
> together with a list of support versions.
Well, since this is helper tool specific, I think it needs to
be moved to helper tool documentation.
>> 3.5. Private Extension
>>
>> These are compiled files linked to python libraries, and kept in a
> no, in most cases, any extension needs to be linked against
> libpython.
Compiled files that require a specific versions of the Python
shared library ....
>> --------------------------------------------------------------------------
>>
>> 3.5.1. Supported versions
>>
>> If a single version of Python is supported, then the versions
>> supported
> ^
> singular!
Thanks
>> by the binary package (XB-Python-Version field or the file
>> .versions) is set to that version (copied from XS-Python-Version).
>> If the current version is not supported, this field it set to the
>> minimum version actually supported by the module.
> what do you mean?
current is 2.3, module support 2.5-, we use 2.5.
> I'm adding a "failed-pre-rtupdate" hook for the next upload, which
> is run, if the pre-rtupdate hook fails (to allow the package to go
> to a sane state again).
> I do have a different view on raising build and support information
> to the Packages/Sources files; in other cases the document clarifies
> things well. IMO the current section 3.6 makes things more
> confusing than they are (at least for me). Would it be helpful to
> add paragraphs starting with "Example:" in sections where they are
> useful? I.e. most package maintainers won't need the rtupdate
> scripts, and therefore could skip reading when they don't need these
> scripts.
Could you review the new versions of the document?
manoj
--
"We learn from history that we learn nothing from history." George
Bernard Shaw
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/%7Esrivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Reply to: