Re: Bug#373853: Please test updated dh_python
On Sat, 17 Jun 2006, Raphael Hertzog wrote:
> > Please find it attached:
> > - it's vastly refactored, it's easier to understand, and the difference
> > between old policy and new policy is easier to see
> > - it supports an -X option which can be used to not scan some files
> > - it fixes the bug 373853 (uses XS-Python-Version: value instead of "all"
> > when the modules are shared)
>
> I forgot:
> - it correctly puts a python dependency if the package has only shared
> python modules (via python-central / python-support) without shebang
And while doing NMU of various python packages I found another little
problem. A package providing modules which work only with python2.4 would
get a dependency "python (>= 2.4)" even if the current version is still
2.3, resulting in an uninstallable package.
An updated dh_python is attached and I also updated the debhelper package on my
site:
http://people.debian.org/~hertzog/python/debhelper_5.0.37.2_all.deb
BTW, since I worked quite a lot in dh_python recently, there's a bzr
branch available with my successive changes:
bzr branch http://ouaza.com/~rhertzog/dh-python/
(Not always very useful as I did rather big commits)
Cheers,
--
Raphaël Hertzog
Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/
#!/usr/bin/perl -w
=head1 NAME
dh_python - calculates python dependencies and adds postinst and prerm python scripts
=cut
use strict;
use File::Find;
use Debian::Debhelper::Dh_Lib;
=head1 SYNOPSIS
B<dh_python> [S<I<debhelper options>>] [B<-n>] [B<-X>I<item>] [B<-V> I<version>] [S<I<module dirs ...>>]
=head1 DESCRIPTION
dh_python is a debhelper program that is responsible for generating the
${python:Depends} substitutions and adding them to substvars files. It
will also add a postinst and a prerm script if required.
The program will look at python scripts and modules in your package, and
will use this information to generate adequate dependencies. There is two
scenarios: if the package uses the XS-Python-Version field then
the new policy will be applied, otherwise the old policy will be used.
=head2 New policy
The XS-Python-Version field (on the source package) defines which Python
versions are supported by the package. It can be "all", "current",
"current, >= X.Y", a single version ("2.3") or a list of versions with
optional comparison operators (ex: "2.3, 2.4" or ">= 2.2, << 2.5" or
">= 2.4").
The binary packages should have a "XB-Python-Version: ${python:Versions}"
field and dh_python will generate the right substvar for that. The
resulting value can be "all" for public modules which work with all python
versions, "current" for private modules which are always byte-compiled
with the current python version or a list of of all versions for which the
extensions have been compiled (ex: "2.3, 2.4"). The dependencies are
adjusted accordingly as well.
Packages with public extensions should also have a "Provides:
${python:Provides}" field. The corresponding substvar will indicate
"pythonX.Y-foo, pythonX.Z-foo" according to all the extensions
effectively available in the package.
=head2 Old policy
It looks at scripts and modules in your package and adds a dependency on python, with the
current major version, or on pythonX.Y if your scripts or modules need a
specific python version. The dependency will be substituted into your
package's control file wherever you place the token "${python:Depends}".
If some modules need to be byte-compiled at install time, appropriate
postinst and prerm scripts will be generated. If already byte-compiled
modules are found, they are removed.
If you use this program, your package should build-depend on python.
Note: in the old policy, /usr/lib/site-python is also scanned for modules.
=head1 OPTIONS
=over 4
=item I<module dirs>
If your package installs python modules in non-standard directories, you
can make dh_python check those directories by passing their names on the
command line. By default, it will check /usr/lib/$PACKAGE, /usr/share/$PACKAGE, /usr/lib/games/$PACKAGE,
/usr/share/games/$PACKAGE and /usr/lib/python?.?/site-packages.
Note: only /usr/lib/python?.?/site-packages and the
extra names on the command line are searched for binary (.so) modules.
=item B<-V> I<version>
If the .py files your package ships are meant to be used by a specific
pythonX.Y version, you can use this option to specify the desired version,
such as 2.3. Do not use if you ship modules in /usr/lib/site-python.
With the new policy, this option is mostly deprecated. Use the
XS-Python-Field to indicate that you're using a specific python version.
=item B<-n>, B<--noscripts>
Do not modify postinst/postrm scripts.
=item B<-X>I<item>, B<--exclude=>I<item>
Exclude files that contain "item" anywhere in their filename from being
taken into accout to generate the python dependency. You may use this
option multiple times to build up a list of things to exclude.
=back
=head1 CONFORMS TO
Debian policy, version 3.5.7
Python policy, version 0.3.7
=cut
init();
my $python = 'python';
# The current python major version
my $python_major;
my $python_version = `$python -V 2>&1`;
if (! defined $python_version || $python_version eq "") {
error("Python is not installed, aborting. (Probably forgot to Build-Depend on python.)");
}
elsif ($python_version =~ m/^Python\s+(\d+)\.(\d+)(\.\d+)*/) {
$python_version = "$1.$2" ;
$python_major = $1 ;
} else {
error("Unable to parse python version out of \"$python_version\".");
}
# The next python version
my $python_nextversion = next_minor_version($python_version);
my $python_nextmajor = $python_major + 1;
my @python_allversions = ('1.5','2.1','2.2','2.3','2.4','2.5');
foreach (@python_allversions) {
s/^/python/;
}
# Check for -V
my $usepython = "python$python_version";
if($dh{V_FLAG_SET}) {
$usepython = $dh{V_FLAG};
$usepython =~ s/^/python/;
if (! grep { $_ eq $usepython } @python_allversions) {
error("Unknown python version $dh{V_FLAG}");
}
}
# Cleaning the paths given on the command line
foreach (@ARGV) {
s#/$##;
s#^/##;
}
# Check if we have an XS-Python-Version and extract it from the control
# file, also check if we use python-central
my $python_header = "";
{
local $/ = ""; # Grab until empty line
open(CONTROL, "debian/control"); # Can't fail, dh_testdir has already been called
my $source = <CONTROL>;
$python_header = $1 if ($source =~ m/^XS-Python-Version: \s*(.*)$/m);
chomp($python_header);
}
# dependency types
use constant PROGRAM => 1;
use constant PY_PRIVATE_MODULE => 2;
use constant PY_PUBLIC_MODULE => 4;
use constant PY_OFFICIAL_MODULE => 8;
use constant PY_UNKNOWN_MODULE => 16;
use constant SO_PRIVATE_MODULE => 32;
use constant SO_PUBLIC_MODULE => 64;
use constant SO_OFFICIAL_MODULE => 128;
use constant SO_UNKNOWN_MODULE => 256;
foreach my $package (@{$dh{DOPACKAGES}}) {
my $tmp = tmpdir($package);
my @dirs = ("usr/lib/$package", "usr/share/$package", "usr/lib/games/$package", "usr/share/games/$package", @ARGV );
my @dirs_so = (@ARGV);
my $dep_on_python = 0;
my $strong_dep = 0;
# Fail early if the package use usr/lib/site-python
if (-d "$tmp/usr/lib/site-python") {
if ($python_header) {
# Error only on new policy
error("The package $package puts files in /usr/lib/site-python: forbidden by policy");
} else {
# Old policy allowed that directory, so scan it
push @dirs, "usr/lib/site-python";
push @dirs_so, "usr/lib/site-python";
}
}
@dirs = grep -d, map "$tmp/$_", @dirs;
@dirs_so = grep -d, map "$tmp/$_", @dirs_so;
my $deps = 0;
my %verdeps = ();
foreach (@python_allversions) {
$verdeps{$_} = 0;
}
# Global scan
my $private_pydirs_regex = join('|', map { "\Q$_\E" } @dirs);
my $private_sodirs_regex = join('|', map { "\Q$_\E" } @dirs_so);
my %private_dirs_list;
my %pyversions_found;
find sub {
return unless -f;
# See if we were asked to exclude this file.
# Note that we have to test on the full filename, including directory.
my $fn="$File::Find::dir/$_";
if (excludefile($fn)) {
verbose_print("Ignoring $fn");
return;
}
# Find scripts
if (-x or /\.py$/) {
local *F;
return unless open F, $_;
if (read F, local $_, 32 and m%^#!\s*/usr/bin/(env\s+)?(python(\d+\.\d+)?)\s%) {
if ( "python" eq $2 ) {
$deps |= PROGRAM;
} elsif(defined $verdeps{$2}) {
$verdeps{$2} |= PROGRAM;
}
}
close F;
}
# Continue only with .py or .so
return unless (/\.py$/ or /\.so$/);
# Remove any byte-compiled file
doit(("rm","-f",$_."c",$_."o")) if /\.py$/;
# Classify the file in the right category
if (/\.py$/ and $private_pydirs_regex and $fn =~ m/(?:$private_pydirs_regex)/) {
# Private python module
verbose_print("Found private module: $fn");
my $dir;
foreach $dir (@dirs) {
if ($fn =~ m/\Q$dir\E/) {
$dir =~ s/^$tmp//;
verbose_print("Memorizing dir to byte-compile: $dir");
$private_dirs_list{"$dir"} = 1;
}
}
if ($dh{V_FLAG_SET}) {
$verdeps{$usepython} |= PY_PRIVATE_MODULE;
} else {
$deps |= PY_PRIVATE_MODULE;
}
} elsif (/\.so$/ and $private_sodirs_regex and $fn =~ m/(?:$private_sodirs_regex)/) {
# Private python extension
verbose_print("Found private extension: $fn");
if ($dh{V_FLAG_SET}) {
$verdeps{$usepython} |= SO_PRIVATE_MODULE;
} else {
$deps |= SO_PRIVATE_MODULE;
}
} elsif ($fn =~ m|$tmp/usr/lib/python([\d\.]+)/site-packages/|) {
$pyversions_found{$1} = 1;
my $v = $1;
if (/\.py$/) {
verbose_print("Found public module: $fn");
$verdeps{"python$v"} |= PY_PUBLIC_MODULE;
} else {
verbose_print("Found public extension: $fn");
$verdeps{"python$v"} |= SO_PUBLIC_MODULE;
}
} elsif ($fn =~ m|$tmp/usr/lib/python([\d\.]+)/|) {
$pyversions_found{$1} = 1;
my $v = $1;
if (/\.py$/) {
verbose_print("Found official module: $fn");
$verdeps{"python$v"} |= PY_OFFICIAL_MODULE;
} else {
verbose_print("Found official extension: $fn");
$verdeps{"python$v"} |= SO_OFFICIAL_MODULE;
}
} elsif ($fn =~ m{$tmp(?:/usr/share/pycentral/|/usr/share/python-support/)}) {
if (/\.py$/) {
verbose_print("Found public module: $fn");
$deps |= PY_PUBLIC_MODULE;
} # No extensions here
} elsif ($fn =~ m|$tmp/usr/share/doc/|) {
# Ignore .py files in doc directory
} else {
# Unknown pyfiles
if (/\.py$/) {
verbose_print("Found unclassified module: $fn");
if ($dh{V_FLAG_SET}) {
$verdeps{$usepython} |= PY_UNKNOWN_MODULE;
} else {
$deps |= PY_UNKNOWN_MODULE;
}
} else {
verbose_print("Found unclassified extension: $fn");
if ($dh{V_FLAG_SET}) {
$verdeps{$usepython} |= SO_UNKNOWN_MODULE;
} else {
$deps |= SO_UNKNOWN_MODULE;
}
}
}
}, $tmp;
# Common dependency handling
foreach my $pyver (keys %verdeps) {
# Always add pythonX.Y dependency if a script uses that interpreter
if ($verdeps{$pyver} & PROGRAM) {
addsubstvar($package, "python:Depends", $pyver);
}
# Always add pythonX.Y dependency if some private modules are
# byte-compiled with it (or if extensions are
# byte-compiled with it)
if ($verdeps{$pyver} & (PY_PRIVATE_MODULE|SO_PRIVATE_MODULE)) {
addsubstvar($package, "python:Depends", $pyver);
}
}
if ($python_header) {
#
# NEW POLICY
#
# Generate the depends to accept all python
# versions that this package effectively provides
my $min_version = "";
my $stop_version = "";
my $versions_field = "";
# Reset again, analysis using new policy follows
$dep_on_python = 0;
# Extracting min version from XS-Python-version
if ($python_header =~ /^current(?:,\s*>=\s*([\d\.]+))?/) {
if (defined $1) {
$min_version = $1;
}
$versions_field = "current";
} elsif ($python_header =~ />=\s*([\d\.]+)/) {
$min_version = $1;
}
if ($python_header =~ /<<\s*([\d\.]+)/) {
$stop_version = $1;
}
# Private extensions, must be rebuilt for each python version
if ($deps & SO_PRIVATE_MODULE) {
$dep_on_python++;
$stop_version = next_minor_version($python_version);
# Packages using a private extension can only
# support one version and they indicate which one
# in XS-Python-Version
if ($versions_field eq "current") {
$versions_field = $python_version;
} else {
$versions_field = $python_header;
}
}
# Private modules
if ($deps & PY_PRIVATE_MODULE) {
# Package with private modules can only support one version at a time
# (even if the modules can be automatically byte-compiled for any new
# python version).
unless ($versions_field) {
# If XS-Python-Version indicated "current" we would already have a value,
# since we have not, XS-Python-Version must contain a hardcoded version.
if ($python_header =~ /^[\d\.]+$/) {
$versions_field = $python_header;
} else {
warning("The package contains private modules. It should indicate 'current'\n".
"or a harcoded python version (ex: '2.4') in the XS-Python-Version field.\n")
}
}
$versions_field = "current" unless $versions_field;
}
# Python scripts & public modules
if ($deps & (PROGRAM|PY_PUBLIC_MODULE)) {
$dep_on_python++;
}
# Public extensions
if (scalar keys %pyversions_found) {
# Extensions will always trigger this (as well as public
# modules not handled by python-support/python-central)
$dep_on_python++;
if (scalar grep { $python_version eq $_ } keys %pyversions_found) {
# Current versions is supported by the package
# It's safe to depends on the corresponding
# interval of python versions
$min_version = min(keys %pyversions_found);
unless ($stop_version) {
my $max_version = max(keys %pyversions_found);
$stop_version = next_minor_version($max_version);
}
}
# Generate the Python-Version field
foreach (keys %pyversions_found) {
addsubstvar($package, "python:Versions", $_);
}
# Generate provides for the python2.X-foo packages that we emulate
if ($package =~ /^python-/) {
foreach (keys %pyversions_found) {
my $virtual = $package;
$virtual =~ s/^python-/$python$_-/;
addsubstvar($package, "python:Provides", $virtual);
}
}
} else {
# Still try to describe the versions that the package support
$versions_field = $python_header unless $versions_field;
addsubstvar($package, "python:Versions", $versions_field);
}
if ($dep_on_python) {
# At least a script has been detected
if ($min_version) {
if (compare_version($min_version, $python_version) <= 0 ) {
# Min-version is less or equal to current version
addsubstvar($package, "python:Depends", $python, ">= $min_version");
} else {
# Min version is greater to current version
# Supposition: new stuff working only with the latest python,
# depend on that specific python version
addsubstvar($package, "python:Depends", "python$min_version");
}
}
# If a stronger dependency is needed
if ($stop_version) {
if (compare_version($stop_version, $python_version) > 0 ) {
# Stop version is strictly bigger than current version
addsubstvar($package, "python:Depends", $python, "<< $stop_version");
} else {
# Only works with old versions,
# package is mostly deprecated,
# no need for a python dependency
}
}
# Let's depend on python anyway
addsubstvar($package, "python:Depends", $python) unless ($min_version or $stop_version);
}
} else {
#
# OLD POLICY
#
# First, the case of python-foo and pythonX.Y-foo
if ($package =~ /^python-/) {
$dep_on_python = 1;
$strong_dep = 1;
# This adds a dependency to python<current>-foo in python-foo
my $pack = $package;
$pack =~ s/^python/python$python_version/;
if (grep { "$_" eq "$pack" } getpackages()) {
addsubstvar($package, "python:Depends", $pack);
}
}
# Dependencies on current python
$dep_on_python = 1 if ($deps &
(PROGRAM|PY_PUBLIC_MODULE|SO_PUBLIC_MODULE|PY_PRIVATE_MODULE|SO_PRIVATE_MODULE));
$strong_dep = 1 if ($deps & (PY_PUBLIC_MODULE|SO_PUBLIC_MODULE));
# Add python dependencies
if ($dep_on_python) {
addsubstvar($package, "python:Depends", $python, ">= $python_version");
if ($strong_dep) {
addsubstvar($package, "python:Depends", $python, "<< $python_nextversion");
} else {
addsubstvar($package, "python:Depends", $python, "<< $python_nextmajor");
}
}
# Add postinst/preinst
my $need_prerm = 0;
foreach my $pyver (@python_allversions) {
my $pydir="/usr/lib/$pyver/site-packages";
my @dirlist = (); # List of directories to byte-compile
# If we have public modules, byte-compile /usr/lib/pythonX.Y/site-packages/
if ($verdeps{$pyver} & PY_PUBLIC_MODULE) {
push @dirlist, $pydir;
}
# Byte-compile private modules with current python
# (or the one indicated by -V)
if (($pyver eq "$usepython") and scalar(keys %private_dirs_list)) {
push @dirlist, keys %private_dirs_list;
}
# If we have something to byte-compile, add the
# corresponding postinst snippet
if (scalar(@dirlist) && ! $dh{NOSCRIPTS}) {
autoscript($package,"postinst","postinst-python","s%#PYVER#%$pyver%;s%#DIRLIST#%@dirlist%");
$need_prerm = 1;
}
}
if ($need_prerm && ! $dh{NOSCRIPTS}) {
autoscript($package,"prerm","prerm-python","s%#PACKAGE#%$package%");
}
}
}
sub next_minor_version {
my $version = shift;
# Handles 2.10 -> 2.11 gracefully
my @items = split(/\./, $version);
$items[1] += 1;
$version = join(".", @items);
return $version;
}
sub compare_version {
my ($a, $b) = @_;
my @A = split(/\./, $a);
my @B = split(/\./, $b);
my $diff = 0;
for (my $i = 0; $i <= $#A; $i++) {
$diff = $A[$i] - $B[$i];
return $diff if $diff;
}
# They are the same
return 0;
}
sub max {
my $max = shift;
foreach (@_) {
$max = $_ if (compare_version($_, $max) > 0);
}
return $max;
}
sub min {
my $min = shift;
foreach (@_) {
$min = $_ if (compare_version($_, $min) < 0);
}
return $min;
}
=head1 SEE ALSO
L<debhelper(7)>
This program is a part of debhelper.
=head1 AUTHORS
Josselin Mouette <joss@debian.org>
Raphael Hertzog <hertzog@debian.org>
most ideas stolen from Brendan O'Dea <bod@debian.org>
=cut
Reply to: