dh_python and python policy analysis
Hi,
I have finished my initial analysis of Python policy and
dh_python, and created a rough specification of what the python
policy is supposed to be (based on current dh_python behaviour). The
current analysis, and future updates, are to be found at
http://www.golden-gryphon.com/software/manoj-policy/
The document is a draft, since I have not been involved in
Python development, it may have flaws, and I am hoping that people
more conversant with Python development would point them out to me.
The document could also stand some polishing; and since it was
written piecemeal, continuity leaves much to be desired as yet.
I am including a text version below.
manoj
Packaging with the new Python policy
A package developers view
Manoj Srivastava
Copyright (c) 2006 Manoj Srivastava
Revision History
Revision 1.0 25 Jul 2006
Obstacles to conformance with the new python policy. While the new Python
policy, specifically the [1]"Packaged Modules" chapter, contains the
elements that must be present in the debian/control filename, it is not
very explicit about how the values are to be substituted. The Debian Wiki
falls back on calling dh_python, which is not helpful in understanding the
actual logic to be followed. This article is an attempt to correct this
gap in documentation.
--------------------------------------------------------------------------
Table of Contents
1. [2]Introduction
1.1. [3]Categorization of Python software
2. [4]Requirements for packages (new policy)
2.1. [5]XS-Python-Version:
2.2. [6]XB-Python-Version:
2.3. [7]Depends:
2.4. [8]Provides
3. [9]Recipe for developers
3.1. [10]Based on type of python modules being packaged
3.1.1. [11]Script
3.1.2. [12]Private Pure Python Modules
3.1.3. [13]Private Extension
3.1.4. [14]Public pure Python Module
3.1.5. [15]Public Extension
1. Introduction
While trying to update SELinux packages, I ran across problems in trying
to determine if my packages were complying with the new python policy: any
practical tips for packaging generally devolved to the statement "Oh, just
run dh_python". This is my attempt to offer more concrete tips for
packaging, by reverse engineering dh_python for the specifications and
tips.
--------------------------------------------------------------------------
1.1. Categorization of Python software
Program/script
This consists of software directly called by an end user of
external program, and is independently interpreted by the Python
interpreter. Usually starts with the magic bytes #!, with the
interpreter being /usr/bin/python* or /usr/bin/env python*.
Modules
This is code included in python "programs/scripts", and not
invoked directly (serving as library modules in compiled
languages).
Modules can be categorized under two orthogonal criteria: firstly, based
on the whether or not they are implemented purely in python, like so:
Pure Python Module
These are python source code, to be interpreted by the Python
interpreter just like program/script code is, and may work across
many versions of Python.
Extension Module
Extensions are C code compiled and linked against a specific
version of the libpython library, and so can only be used by one
version of Python.
Another way of categorizing modules is based on whether or not they are
available for use by third party scripts/modules.
Public
Public modules are available for use in other Python scripts or
modules using the import directive. They are installed in one of
the directories
/usr/lib/pythonX.Y/site-packages
/usr/lib/pythonX.Y
/var/lib/python-support/pythonX.Y
/usr/share/pycentral
/usr/share/python-support
Private
Private modules are generally only accessible to a specific
program or suite of programs included in the same package. They
are installed in special directories, for example:
/usr/lib/<package>
/usr/share/<package>
/usr/lib/games/<package>
/usr/share/games/<package>
--------------------------------------------------------------------------
2. Requirements for packages (new policy)
The new python policy places certain requirements for packages that
contain Python bits.
--------------------------------------------------------------------------
2.1. XS-Python-Version:
The XS-Python-Version field in debian/control specifies the versions of
Python supported by the package. This is used to track packages during
Python transitions. This can be a single version, or one or more of a list
of non-overlapping ranges. The lowest range may optionally omit a low end,
and the highest range may optionally omit an upper end. In other words,
the overall range may be open ended. The ranges are often matched to the
set of all known Python version that have existed, and the supported set
is the intersection of the known versions of python and the range
specification.
--------------------------------------------------------------------------
2.2. XB-Python-Version:
This is substituted in the binary package using the values from the
XS-Python-Version field, filtered according to various rules detailed in
the next section.
--------------------------------------------------------------------------
2.3. Depends:
Packaged modules available for the default Python version (or many
versions including the default) must depend on python (>= X.Y). If they
require other modules to work, they must depend on the corresponding
python-foo. They must not depend on any pythonX.Y-foo. Packaged modules
available for only one particular version of Python (even if it is the
default), on the other hand, must depend on the corresponding pythonX.Y
package instead. If they need other modules, they must depend on the
corresponding pythonX.Y-foo packages, and must not depend on any
python-foo.
--------------------------------------------------------------------------
2.4. Provides
Packages with public modules and extensions should be named, or should
provide, python-foo, if the package contains an extension for more than
one python version. Also, for every version of python supported the
package should provide pythonX.Y-foo packages.
--------------------------------------------------------------------------
3. Recipe for developers
3.1. Based on type of python modules being packaged
This is essentially a restatement of policy and dh_python internals to
allow one to look at the type of python components being shipped in the
package, and get an idea of what the expectation of policy is, based on
both the policy and dh_python. My notes on the internal implementation of
dh_python are also [16]provided for reference.
There is certain amount of duplication in the sections below, but I have
elected to leave it in place, so each section on each type of module is
self contained.
--------------------------------------------------------------------------
3.1.1. Script
These are executable scripts which start with the magic string #!.
--------------------------------------------------------------------------
3.1.1.1. XS-Python-Version:
This is a list of python versions supported by the package. This field
can be a single version, or a set of ranges. This should be set to the
list of python versions that the script can support, or "all". If a script
invokes /usr/bin/pythonX.Y, then XS-Python-Version should be set to X.Y.
--------------------------------------------------------------------------
3.1.1.2. XB-Python-Version:
If there is no other value for the versions (set by any modules
packaged), then this field is the same as XS-Python-Version, if
XS-Python-Version is not empty. If it is empty, XB-Python-Version is set
to "all".
--------------------------------------------------------------------------
3.1.1.3. Depends:
If the script invokes a specific version of Python, the package must
depend on pythonX.Y. If there are separate scripts that invoke different
versions of Python, then all these versions must be in the Depends field
-- if you still want to continue packaging instead of just shooting the
upstream.
If the script just calls /usr/bin/python, and there is no other
restriction on the versions of Python supported, then the dependency is an
un-versioned dependency on "python". If there is a range of Python
versions supported, then the dependency has to be versioned, the process
is as follows.
1. If a lower bound to the range exists, then
a. If the lower bound is less than or equal to the current
version, then add a dependency on python (>= $min_version)
b. Or else depend on python (>= $min_version) |
python$min_version)
2. If the upper bound to the range exists, then the stop version is the
next minor version after the upper bound (found by incrementing the
minor version), whether or not it exists, and is form of Major.Minor.
a. If the stop version is strictly greater than the upper version,
then depend on python (<< $stop_version)
--------------------------------------------------------------------------
3.1.2. Private Pure Python Modules
These are python source code files that live in a private directory. If
a package contains a private pure Python module, only one Python version
may be supported, even if the private module can be recompiled for any new
version of Python.
--------------------------------------------------------------------------
3.1.2.1. Byte compilation
Any directory in which private pure Python modules are placed should be
remembered, and the modules byte-compiled on installation.
--------------------------------------------------------------------------
3.1.2.2. XS-Python-Version:
This should be set to the version of Python supported, or "current", if
there are no specific restrictions based on Python version.
--------------------------------------------------------------------------
3.1.2.3. XB-Python-Version:
If the package contains private extention modules, then the rules for
setting XB-Python-Version for the private extention modules take
precedence.
This should be set to "current" if there are no specific restrictions
based on Python version. Note that presence of private extension modules
in the same package would mean that there are restrictions on the
XB-Python-Version field.
If only a subset of Python versions are supported by the package, then
if the current version is supported, then XB-Python-Version should be set
to "current".
If the current version is not supported, and if the minimum version
supported is strictly higher than the current version, the
XB-Python-Version field should be set to the minimum version supported.
Failing that, if the current version is not supported, and the max
version is lower than the current version, then this package should be
deprecated, and is obsolete.
--------------------------------------------------------------------------
3.1.2.4. Depends:
If there is no restriction on the versions of Python supported, then the
dependency is an un-versioned dependency on "python". If a specific
version of python is supported, the following dependencies are created:
1. If the current version is supported, then add a dependency on python
2. Or else if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
If a upper bound of the supported versions exists, then the stop
version is the next minor version after the maximum supported version
(found by incrementing the minor version), whether or not it exists,
and is form of Major.Minor. If there is an upper bound on the
supported versions, depend on python (<< $stop_version)
3. If the current version is higher than any supported version, then
this package is obsolete. However, the following dependencies are
still valid, if the corresponding lower and upper bounds of supported
versions exist. python (>= $min_version) | python$min_version,
python (<< $stop_version)
--------------------------------------------------------------------------
3.1.3. Private Extension
These are compiled files linked to python libraries, and kept in a
private directory. Since these files are compiled with one specific
version of python, and do not live in versioned directories, only one
version of python is supported at any given time.
--------------------------------------------------------------------------
3.1.3.1. XS-Python-Version:
This should be set to the version of Python supported, or "current", if
there are no specific restrictions based on Python version.
--------------------------------------------------------------------------
3.1.3.2. XB-Python-Version:
If a single version of Python is supported, then this field is set to
that version (copied from XS-Python-Version). If the current version is
not supported, this field it set to the minimum version actually supported
by the module. If the current version is supported (or there are no
restrictions on the version of python supported), then this field is set
to the current version. Note that this is different from the case of the
private pure Python modules, where the explicit string "current" was used
in similar situations.
--------------------------------------------------------------------------
3.1.3.3. Depends:
Since only one version can be supported at any given time, and the
version supported is recorded in the XB-Python-Version, we create stop
version to be the next minor version up from the supported version (by
incrementing the minor version). The following process is followed.
1. If the supported version is less than or equal to the current
version, then depend on python (>= $version)
2. Or else, depend on python (>= $version) | python$version
Also, since only one version is supported, depend on: python (<<
$stop_version)
--------------------------------------------------------------------------
3.1.4. Public pure Python Module
Public modules should be packaged with a name of python-foo, where foo
is the name of the module. Such a package should support the current
Debian Python version, and more if possible.
There are two kinds of public pure Python modules, the most common being
the variety that live in unversioned public module directories, and, in
rare cases, pure python modules that live in versioned public module
directories. The latter is usually the case when the pure Python module
imports an public extension module from the same directory, and thus the
public extension and pure python modules must be in the same directory.
Otherwise, pure python modules should live in an unversioned public module
directory.
Official pure Python modules generally live in a different set of
directories than unofficial ones, but are otherwise treated exactly like
other public pure Python Module which live in unversioned directories as
detailed below.
--------------------------------------------------------------------------
3.1.4.1. XS-Python-Version:
This should be set to the version of Python supported, or "current", if
there are no specific restrictions based on Python version. In case of
pure Python modules in versioned public module directories, the set of
versions supported usually corresponds to the list of versioned
directories in which pure Python modules live.
--------------------------------------------------------------------------
3.1.4.2. Byte compiling
In the common case of pure Python modules in unversioned public module
directories, tools exist to help byte compile the pure Python modules for
all versions of Python installed on the target system. In case of pure
Python modules in versioned public module directories, byte compilation is
up to the package scripts.
--------------------------------------------------------------------------
3.1.4.3. XB-Python-Version:
The rules for this field setting depend on whether the pure python
module lives in a versioned directory, or not.
1. In the common case of pure Python modules in unversioned public
module directories, any private modules have preference when it comes
to determining the value for this field. If there are no private
modules packaged with this package, this is set to the value of
XB-Python-Version, if present. If XB-Python-Version is null, then this
field is set to "all"
2. In case of pure Python modules in versioned public module
directories, this field is set to a list of versions corresponding to
the list of versioned public module directories in which pure Python
modules live.
--------------------------------------------------------------------------
3.1.4.4. Depends:
The rules for this field setting depend on whether the pure python
module lives in a versioned directory, or not.
1. In the common case of pure Python modules in unversioned public
module directories, we look to XB-Python-Version to see how to
proceed.
a. If there is no restriction on the version of python supported
(there is no lower or upper bound to the range, and the current
version is supported), then add a dependency on python.
[17]Note If there is no upper or lower bound, but the current
version is specifically excluded, it is unclear what to
do. Consider the range -2.2,2.4-.
b. Or else if the minimum version supported is greater than the
current versions, then depend on python (>= $min_version) |
python$min_version
If a upper bound of the supported versions exists, then the
stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. If there is
an upper bound on the supported versions, depend on python (<<
$stop_version)
c. If the current version is higher than any supported version,
then this package is obsolete. However, the following
dependencies are still valid, if the corresponding lower and
upper bounds of supported versions exist. python (>=
$min_version) | python$min_version, python (<< $stop_version)
2. For the case where pure Python modules in versioned public module
directories, the set of versions corresponding to the list of
versioned public module directories in which pure Python modules live
is used in lieu of the value of XS-Python-Version.
a. if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
The stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. Depend on
python (<< $stop_version)
--------------------------------------------------------------------------
3.1.4.5. Provides:
Dependencies for the package also differ slightly based on whether the
pure Python module is version dependent or not.
1. In the common case of pure Python modules in unversioned public
module directories,
a. If there is a minimum supported version,
i. If the minimum version supported is less than or equal to
the current version, then depend on python (>=
$min_version)
ii. Or else, if the minimum version supported is greater than
the current version, then depend on python (>=
$min_version) | python$min_version
If a upper bound of the supported versions exists, then the
stop version is the next minor version after the maximum
supported version (found by incrementing the minor version),
whether or not it exists, and is form of Major.Minor. If there is
an upper bound on the supported versions, and the upper bround is
greater than or equal to the current version, then depend on
python (<< $stop_version)
b. If the current version is higher than any supported version,
then this package is obsolete. However, the following
dependencies are still valid, if the corresponding lower and
upper bounds of supported versions exist. python (>=
$min_version) | python$min_version, python (<< $stop_version)
2. In case of pure Python modules in versioned public module
directories, the set of versions corresponding to the list of
versioned public module directories in which pure Python modules live
is used in lieu of the value of XS-Python-Version.
a. If the lowest version in the set is less than or equal to the
current version, then depend on python (>= $min_version)
b. Or else depend on python (>= $min_version) |
python$min_version,
c. If the highest version in the set is greater than or equal to
the current version, then depend on python (<< $stop_version)
where the stop version is the next minor version greater than the
highest supported version.
d. For every version in the set, also depend on python$version
--------------------------------------------------------------------------
3.1.4.6. Provides
The rules for this field setting depend on whether the pure python
module lives in a versioned directory, or not.
1. For pure Python modules in unversioned directories, if the name of
the package is python-foo, provide a package named pythonX.Y-foo for
every officially supported version of Python (look at the contents of
the file /usr/share/python/debian_defaults).
2. For the case where pure Python modules in versioned public module
directories, for every version in the set corresponding to the list of
versioned public module directories in which pure Python modules live,
provide a package named pythonX.Y-foo (assuming that the package name
is python-foo).
--------------------------------------------------------------------------
3.1.5. Public Extension
Public extensions should be packaged with a name of python-foo, where
foo is the name of the module. Such a package should support the current
Debian Python version, and more if possible.
--------------------------------------------------------------------------
3.1.5.1. XS-Python-Version:
This should be set to the list of versions for which the compiled
extension modules are provided.
--------------------------------------------------------------------------
3.1.5.2. XB-Python-Version:
This should be set to the list of versions for which the compiled
extension modules are provided.
--------------------------------------------------------------------------
3.1.5.3. Depends:
The list of Python versions supported is used to determine the
dependency relationships as follows.
1. if the minimum version supported is greater than the current
versions, then depend on python (>= $min_version) |
python$min_version
The stop version is the next minor version after the maximum
supported version (found by incrementing the minor version), whether
or not it exists, and is form of Major.Minor. Depend on python (<<
$stop_version)
--------------------------------------------------------------------------
3.1.5.4. Provides
For every version of Python for which compiled extension modules are
provided, provide a package named pythonX.Y-foo (assuming that the package
name is python-foo).
References
Visible links
1. http://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html
2. file:///tmp/html-vM3345#AEN38
3. file:///tmp/html-vM3345#AEN44
4. file:///tmp/html-vM3345#AEN102
5. file:///tmp/html-vM3345#AEN105
6. file:///tmp/html-vM3345#AEN110
7. file:///tmp/html-vM3345#AEN114
8. file:///tmp/html-vM3345#AEN123
9. file:///tmp/html-vM3345#AEN128
10. file:///tmp/html-vM3345#AEN130
11. file:///tmp/html-vM3345#AEN137
12. file:///tmp/html-vM3345#AEN183
13. file:///tmp/html-vM3345#AEN229
14. file:///tmp/html-vM3345#AEN258
15. file:///tmp/html-vM3345#AEN385
16. file:///tmp/python_policy.txt
--
"A facility for quotation covers the absence of original thought."
Lord Peter Wimsey (Dorothy L. Sayers, "Gaudy Night")
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/%7Esrivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Reply to: