Re: dh_python and python policy analysis
Hi,
Here is round two of my take on python policy. I have
incorporate the correction offered by various people, and read the
documents for python-central and python-support, and incorporated my
understanding of those into this document.
So, this is my take on the new python policy, based on the
analysis of the new Python policy draft and dh_python, and is
supposed to be a rough specification of what the python policy is
supposed to be (based on current dh_python behaviour). The current
analysis, and future updates, are to be found at
http://www.golden-gryphon.com/software/manoj-policy/
This can still use a lot of polishing.
I am including a text version below.
manoj
Packaging with the new Python policy
A package developers view
Manoj Srivastava
Copyright (c) 2006 Manoj Srivastava
Revision History
Revision 1.0 25 Jul 2006
Obstacles to conformance with the new python policy. While the new Python
policy, specifically the [1]"Packaged Modules" chapter, contains the
elements that must be present in the debian/control filename, it is not
very explicit about how the values are to be substituted. The Debian Wiki
falls back on calling dh_python, which is not helpful in understanding the
actual logic to be followed. This article is an attempt to correct this
gap in documentation.
--------------------------------------------------------------------------
Table of Contents
1. [2]Introduction
1.1. [3]Categorization of Python software
2. [4]Recipe for developers
2.1. [5]General Notes
2.1.1. [6]Python versions supported by the
source
2.1.2. [7]Supported versions in the binary
package
2.1.3. [8]Depends:
2.1.4. [9]Provides
2.1.5. [10]Build-Depends:
2.2. [11]Script
2.2.1. [12]Supported versions
2.2.2. [13]Depends:
2.3. [14]Private Pure Python Modules
2.3.1. [15]Byte compilation
2.3.2. [16]Supported versions
2.3.3. [17]Depends:
2.4. [18]Private Extension
2.4.1. [19]Supported versions
2.4.2. [20]Depends:
2.5. [21]Public pure Python Module
2.5.1. [22]Byte compiling
2.5.2. [23]Supported versions
2.5.3. [24]Depends:
2.5.4. [25]Provides:
2.6. [26]Public Extension
2.6.1. [27]Supported versions
2.6.2. [28]Depends:
2.6.3. [29]Provides
1. Introduction
While trying to update SELinux packages, I ran across problems in trying
to determine if my packages were complying with the new python policy: any
practical tips for packaging generally devolved to the statement "Oh, just
run dh_python". This is my attempt to offer more concrete tips for
packaging, by reverse engineering dh_python for the specifications and
tips.
--------------------------------------------------------------------------
1.1. Categorization of Python software
Program/script
This consists of software directly called by an end user of
external program, and is independently interpreted by the Python
interpreter. Usually starts with the magic bytes #!, with the
interpreter being /usr/bin/python* or /usr/bin/env python*.
Modules
This is code included in python "programs/scripts", and not
invoked directly (serving as library modules in compiled
languages).
Modules can be categorized under two orthogonal criteria: firstly, based
on the whether or not they are implemented purely in python, like so:
Pure Python Module
These are python source code, to be interpreted by the Python
interpreter just like program/script code is, and may work across
many versions of Python.
Extension Module
Extensions are C code compiled and linked against a specific
version of the libpython library, and so can only be used by one
version of Python.
Another way of categorizing modules is based on whether or not they are
available for use by third party scripts/modules.
Public
Public modules are available for use in other Python scripts or
modules using the import directive. They are installed in one of
the directories
/usr/lib/pythonX.Y/site-packages
/usr/lib/pythonX.Y
/var/lib/python-support/pythonX.Y
. Packages must not directly install anything in the directory
/var/lib/python-support/pythonX.Y, that is reserved for the
utility python-support.
Packages may install public Python modules in directories
specific to Python packaging utilities -- which specify the
directories under which such modules should be droppped, and the
the structure of these directories is defined by the utilities
themselves. Please note that these directories are not in the path
for Python, and are not available for modules to be imported from.
At the time of writing, such uility specific directories include:
/usr/share/pycentral
/usr/share/python-support
.
Private
Private modules are modules that are used internally by a
program, and are generally only accessible to that specific
program or suite of programs included in the same package. They
are not intended to be shared with third party programs. They are
installed in special directories, for example:
/usr/lib/<package>
/usr/share/<package>
/usr/lib/games/<package>
/usr/share/games/<package>
--------------------------------------------------------------------------
2. Recipe for developers
2.1. General Notes
This is essentially a restatement of policy and dh_python internals to
allow one to look at the type of python components being shipped in the
package, and get an idea of what the expectation of policy is, based on
both the policy and dh_python. My notes on the internal implementation of
dh_python are also [30]provided for reference.
There is certain amount of duplication in the sections below, but I have
elected to leave it in place, so each section on each type of module is
self contained.
One of the major features of Python packaging on Debian systems is that
pure Python modules are byte compiled for all supported versions of Python
that are available. There are several utilities available to aid in this
task (python-central and python-support, for example). However, these
utilities need information about modules in the package (for example,
whether a pure Python module supports all versions of Python or not).
--------------------------------------------------------------------------
2.1.1. Python versions supported by the source
The XS-Python-Version field in debian/control specifies the versions of
Python supported by the package. While this is a requirement only if using
the utility package python-central (python-support, for example, prefers
debian/pyversions), setting this is "appreciated" in any case, according
to the [31]new policy wiki[32][1]. This is used to track packages during
Python transitions.
This can be a single version, or one or more of a list of
non-overlapping ranges. The lowest range may optionally omit a low end,
and the highest range may optionally omit an upper end. In other words,
the overall range may be open ended. The ranges are often matched to the
set of all known Python version that have existed, and the supported set
is the intersection of the known versions of python and the range
specification.
1. For packages with private modules or private extensions compiled for
the current python version and for applications using /usr/bin/python,
this should be set to the string "current", or if the module doesn't
work with all Python version, one may specify something like "current,
>= 2.4" or even "current, >= 2.2, << 2.5" - as long as the current
version is supported.
2. For packages with public modules, this should be set to the string
"all", unless not all versions of Python are supported (in which case
the setting should specify the versions or range of versions actually
supported, like ">= 2.4" or ">= 2.2, << 2.y" - as long as the current
version is supported).
3. If the current version is not supported, or if the application uses
/usr/bin/pythonX.Y explicitly, then this should be set to the (list
of) version(s) supported.
--------------------------------------------------------------------------
2.1.2. Supported versions in the binary package
python-central uses XB-Python-Version field in the control file, which
is substituted in the binary package using the values from the
XS-Python-Version field, filtered according to various rules detailed in
the rest of this document. Again, this field is not mandatory unless the
utility python-central is used, but is recommended nevertheless.
python-support uses either the list of public extension versions i
/usr/lib/python-support/$package or the
/usr/share/python-support/$package/.version.
--------------------------------------------------------------------------
2.1.3. Depends:
Packaged modules available for the default Python version (or many
versions including the default) must depend on python (>= X.Y). If they
require other modules to work, they must depend on the corresponding
python-foo. They must not depend on any pythonX.Y-foo.
Packaged modules available for only one particular version of Python
(even if it is the default), on the other hand, must depend on the
corresponding pythonX.Y package instead. If they need other modules (say,
"bar"), they must depend on the corresponding pythonX.Y-bar packages, and
must not depend on any python-bar. For consistency, if the package ("foo")
provides several pythonX.Y-foo packages, and it needs the module "bar", it
must also depend on pythonX.Y-bar corresponding to each version "X.Y" for
the virtual packages pythonX.Y-foo that it provides.
--------------------------------------------------------------------------
2.1.4. Provides
Packages with public modules and extensions should be named, or should
provide, python-foo, if the package contains an extension for more than
one python version. Also, for every version of python supported the
package should provide pythonX.Y-foo packages. This assumes that the
package correctly depends on all the appropriate versions of any version
specific module that it itself requires.
--------------------------------------------------------------------------
2.1.5. Build-Depends:
If the package provides public extension modules, then build depending
on "python-all-dev (>= 2.3.5-11)" shall ensure that all the >pythonX.Y-dev
packages are available during building.
If you are using helper packages (CDBS, yada, debhelper, etc) you should
build depend on the appropriate version. This also applies if your build
process uses either python-central or python-support.
--------------------------------------------------------------------------
2.2. Script
These are executable scripts which start with the magic string #!.
--------------------------------------------------------------------------
2.2.1. Supported versions
If a script invokes /usr/bin/pythonX.Y, then the version supported by
the source package (XS-Python-Version or debian/pyversions) should be
restricted to X.Y, assuming that the field is being provided. Or else, it
should be set to the list of python versions that the script can support,
or "all".
If there is no other value for the versions (set by any modules
packaged), then the versions spported by the binary package
(XB-Python-Version or the file .versions) is the same as above
(XS-Python-Version, if XS-Python-Version is not empty, or "all", if it is
empty).
--------------------------------------------------------------------------
2.2.2. Depends:
If the script invokes a specific version of Python, the package must
depend on pythonX.Y. If there are separate scripts that invoke different
versions of Python, then all these versions must be in the Depends field
-- if you still want to continue packaging instead of just shooting the
upstream.
If the script just calls /usr/bin/python, and there is no other
restriction on the versions of Python supported, then the dependency is an
un-versioned dependency on "python". If there is a range of Python
versions supported, then the dependency has to be versioned, the process
is as follows.
1. If a lower bound to the range exists, then
a. If the lower bound is less than or equal to the current
version, then add a dependency on python (>= $min_version)
b. Or else depend on python (>= $min_version) |
python$min_version)
2. If the upper bound to the range exists, then the stop version is the
next minor version after the upper bound (found by incrementing the
minor version), whether or not it exists, and is form of Major.Minor.
a. If the stop version is strictly greater than the upper version,
then depend on python (<< $stop_version)
--------------------------------------------------------------------------
2.3. Private Pure Python Modules
These are python source code files that live in a private directory. If
a package contains a private pure Python module, only one Python version
may be supported, even if the private module can be recompiled for any new
version of Python.
--------------------------------------------------------------------------
2.3.1. Byte compilation
Any directory in which private pure Python modules are placed should be
remembered, and the modules byte-compiled on installation. This is done
automatically if one uses python-central, python-support uses the
information in the file /usr/share/python-support/$package.dirs for a list
of directories to scan.
--------------------------------------------------------------------------
2.3.2. Supported versions
The version supported by the source package (XS-Python-Version or
debian/pyversions) is either the specific version of Python supported, or
"current", if there are no specific restrictions based on Python version.
If the package contains private extention modules, then the rules for
setting XB-Python-Version for the private extention modules take
precedence.
This should be set to "current" if there are no specific restrictions
based on Python version. Note that presence of private extension modules
in the same package would mean that there are restrictions on the
XB-Python-Version field.
If only a subset of Python versions are supported by the package, then
if the current version is supported, then XB-Python-Version should be set
to "current".
If the current version is not supported, and if the minimum version
supported is strictly higher than the current version, the
XB-Python-Version field should be set to the minimum version supported.
Failing that, if the current version is not supported, and the max
version is lower than the current version, then this package should be
deprecated, and is obsolete.
--------------------------------------------------------------------------
2.3.3. Depends:
If there is no restriction on the versions of Python supported, then the
dependency is an un-versioned dependency on "python". If a specific
version of python is supported, the following dependencies are created:
1. If the current version is supported, then add a dependency on python
2. Or else if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
If a upper bound of the supported versions exists, then the stop
version is the next minor version after the maximum supported version
(found by incrementing the minor version), whether or not it exists,
and is form of Major.Minor. If there is an upper bound on the
supported versions, depend on python (<< $stop_version)
3. If the current version is higher than any supported version, then
this package is obsolete. However, the following dependencies are
still valid, if the corresponding lower and upper bounds of supported
versions exist. python (>= $min_version) | python$min_version,
python (<< $stop_version)
--------------------------------------------------------------------------
2.4. Private Extension
These are compiled files linked to python libraries, and kept in a
private directory. Since these files are compiled with one specific
version of python, and do not live in versioned directories, only one
version of python is supported at any given time.
--------------------------------------------------------------------------
2.4.1. Supported versions
The version supported by the source package (XS-Python-Version or
debian/pyversions) is either the specific version of Python supported, or
"current", if there are no specific restrictions based on Python version.
If a single version of Python is supported, then the versions supported
by the binary package (XB-Python-Version field or the file .versions) is
set to that version (copied from XS-Python-Version). If the current
version is not supported, this field it set to the minimum version
actually supported by the module. If the current version is supported (or
there are no restrictions on the version of python supported), then this
field is set to the current version. Note that this is different from the
case of the private pure Python modules, where the explicit string
"current" was used in similar situations.
--------------------------------------------------------------------------
2.4.2. Depends:
Since only one version can be supported at any given time, and the
version supported is recorded in the XB-Python-Version, we create stop
version to be the next minor version up from the supported version (by
incrementing the minor version). The following process is followed.
1. If the supported version is less than or equal to the current
version, then depend on python (>= $version)
2. Or else, depend on python (>= $version) | python$version
Also, since only one version is supported, depend on: python (<<
$stop_version)
--------------------------------------------------------------------------
2.5. Public pure Python Module
Public modules should be packaged with a name of python-foo, where foo
is the name of the module. Such a package should support the current
Debian Python version, and more if possible.
There are two kinds of public pure Python modules, the most common being
the variety that live in unversioned public module directories, and, in
rare cases, pure python modules that live in versioned public module
directories. The latter is usually the case when the pure Python module
imports an public extension module from the same directory, and thus the
public extension and pure python modules must be in the same directory.
Otherwise, pure python modules should live in an unversioned public module
directory.
Depending on the packaging utility used, the modules live in either
/usr/share/python-central or in /usr/share/python-support/$package.
Official pure Python modules generally live in a different set of
directories than unofficial ones, but are otherwise treated exactly like
other public pure Python Module which live in unversioned directories as
detailed below.
--------------------------------------------------------------------------
2.5.1. Byte compiling
In the common case of pure Python modules in unversioned public module
directories, tools exist to help byte compile the pure Python modules for
all versions of Python installed on the target system. In case of pure
Python modules in versioned public module directories, byte compilation is
up to the package scripts.
--------------------------------------------------------------------------
2.5.2. Supported versions
For the source package, this should be the version of Python supported,
or "all", if there are no specific restrictions based on Python version.
In case of pure Python modules in versioned public module directories, the
set of versions supported usually corresponds to the list of versioned
directories in which pure Python modules live.
The rules for determining the versions supported by the binary package
setting depend on whether the pure python module lives in a versioned
directory, or not.
1. In the common case of pure Python modules in unversioned public
module directories, any private modules have preference when it comes
to determining the value for this field. If there are no private
modules packaged with this package, this is set to the value of
XB-Python-Version, if present. If XB-Python-Version is null, then this
field is set to "all"
2. In case of pure Python modules in versioned public module
directories, this field is set to a list of versions corresponding to
the list of versioned public module directories in which pure Python
modules live.
--------------------------------------------------------------------------
2.5.3. Depends:
The rules for this field setting depend on whether the pure python
module lives in a versioned directory, or not.
1. In the common case of pure Python modules in unversioned public
module directories, we look to XB-Python-Version to see how to
proceed.
a. If there is no restriction on the version of python supported
(there is no lower or upper bound to the range, and the current
version is supported), then add a dependency on python.
[33]Note If there is no upper or lower bound, but the current
version is specifically excluded, it is unclear what to
do. Consider the range -2.2,2.4-.
b. Or else if the minimum version supported is greater than the
current versions, then depend on python (>= $min_version) |
python$min_version
If a upper bound of the supported versions exists, then the
stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. If there is
an upper bound on the supported versions, depend on python (<<
$stop_version)
c. If the current version is higher than any supported version,
then this package is obsolete. However, the following
dependencies are still valid, if the corresponding lower and
upper bounds of supported versions exist. python (>=
$min_version) | python$min_version, python (<< $stop_version)
2. For the case where pure Python modules in versioned public module
directories, the set of versions corresponding to the list of
versioned public module directories in which pure Python modules live
is used in lieu of the value of XS-Python-Version.
a. if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
The stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. Depend on
python (<< $stop_version)
--------------------------------------------------------------------------
2.5.4. Provides:
Dependencies for the package also differ slightly based on whether the
pure Python module is version dependent or not.
1. In the common case of pure Python modules in unversioned public
module directories,
a. If there is a minimum supported version,
i. If the minimum version supported is less than or equal to
the current version, then depend on python (>=
$min_version)
ii. Or else, if the minimum version supported is greater than
the current version, then depend on python (>=
$min_version) | python$min_version
If a upper bound of the supported versions exists, then the
stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. If there is
an upper bound on the supported versions, and the upper bround is
greater than or equal to the current version, then depend on
python (<< $stop_version)
b. If the current version is higher than any supported version,
then this package is obsolete. However, the following
dependencies are still valid, if the corresponding lower and
upper bounds of supported versions exist. python (>=
$min_version) | python$min_version, python (<< $stop_version)
2. In case of pure Python modules in versioned public module
directories, the set of versions corresponding to the list of
versioned public module directories in which pure Python modules live
is used in lieu of the value of XS-Python-Version.
a. If the lowest version in the set is less than or equal to the
current version, then depend on python (>= $min_version)
b. Or else depend on python (>= $min_version) |
python$min_version,
c. If the highest version in the set is greater than or equal to
the current version, then depend on python (<< $stop_version)
where the stop version is the next minor version greater than the
highest supported version.
d. For every version in the set, also depend on python$version
--------------------------------------------------------------------------
2.6. Public Extension
Public extensions should be packaged with a name of python-foo, where
foo is the name of the module. Such a package should support the current
Debian Python version, and more if possible.
Depending on the packaging utility used, the modules live in either
/usr/lib/pythonX.Y/site-packages or in
/usr/lib/python-support/$package/pythonX.Y, with proper symbolic links in
place.
--------------------------------------------------------------------------
2.6.1. Supported versions
The versions supported by the source and binary packages should be set
to the list of versions for which the compiled extension modules are
provided.
--------------------------------------------------------------------------
2.6.2. Depends:
The list of Python versions supported is used to determine the
dependency relationships as follows.
1. if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
The stop version is the next minor version after the maximum
supported version (found by incrementing the minor version), whether
or not it exists, and is form of Major.Minor. Depend on python (<<
$stop_version)
--------------------------------------------------------------------------
2.6.3. Provides
For every version of Python for which compiled extension modules are
provided, provide a package named pythonX.Y-foo (assuming that the package
name is python-foo).
Notes
[34][1] For compatibility, each of the tools (python-central and
python-support) are able to use each others data.
References
Visible links
1. http://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html
2. file:///tmp/html-n28320#AEN38
3. file:///tmp/html-n28320#AEN44
4. file:///tmp/html-n28320#AEN104
5. file:///tmp/html-n28320#AEN106
6. file:///tmp/html-n28320#AEN116
7. file:///tmp/html-n28320#AEN146
8. file:///tmp/html-n28320#AEN156
9. file:///tmp/html-n28320#AEN173
10. file:///tmp/html-n28320#AEN178
11. file:///tmp/html-n28320#AEN189
12. file:///tmp/html-n28320#AEN193
13. file:///tmp/html-n28320#AEN206
14. file:///tmp/html-n28320#AEN235
15. file:///tmp/html-n28320#AEN238
16. file:///tmp/html-n28320#AEN244
17. file:///tmp/html-n28320#AEN261
18. file:///tmp/html-n28320#AEN284
19. file:///tmp/html-n28320#AEN287
20. file:///tmp/html-n28320#AEN298
21. file:///tmp/html-n28320#AEN315
22. file:///tmp/html-n28320#AEN323
23. file:///tmp/html-n28320#AEN326
24. file:///tmp/html-n28320#AEN339
25. file:///tmp/html-n28320#AEN380
26. file:///tmp/html-n28320#AEN430
27. file:///tmp/html-n28320#AEN436
28. file:///tmp/html-n28320#AEN439
29. file:///tmp/html-n28320#AEN452
30. file:///tmp/python_policy.txt
31. http://wiki.debian.org/DebianPython/NewPolicy
32. file:///tmp/html-n28320#FTN.AEN126
34. file:///tmp/html-n28320#AEN126
--
A truly thoroughbred man (a Buddha) is hard to find. He is not born
anywhere, but where that seer is born, the people prosper. 193
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/%7Esrivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Reply to: