Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
Hi,
Quoting Philipp Kern (2014-07-25 14:45:17)
> On Fri, Jul 25, 2014 at 02:19:38PM +0200, Johannes Schauer wrote:
> > Maybe. We'd be equally (if not more) happy if SRM would reconsider their
> > decision (expressed on #debian-release toward Helmut Grohne) that these patches
> > are too intrusive for a stable update.
>
> There are no decisions expressed on IRC, at most tendencies. Get them on the
> right list (debian-release@lists.d.o) and get them reviewed there. (Which never
> happened to far, aside Guillem's general question avoid feasibility, but
> without a patchset to consider.)
following this advice, let us propose the attached three minimal patches to
dpkg, apt and python-apt for a stable updated. Fixing bug#744246 would only
require apt and python-apt but maybe having dpkg not choke on it as well would
also be desirable.
The patches do not implement full profile support but just allow the new
Build-Depends syntax to be parsed without throwing an error. If the new syntax
is encountered, dependencies will be added as if no build profile was active.
The patches include no way to change the activated build profiles to keep them
as minimal as possible.
Do you think the attached patches are minimal/safe enough to include them as a
stable update?
If these patches were added to stable, then https://bugs.debian.org/744246
could be resolved before jessie release. This in turn would mean that we can
test build profiles before the jessie release. And this means that we can fix
not only bootstrapping bugs in packages before the release but also possibly
missing build profile related bugs in dpkg, apt, other build tools or even the
buildd machinery. That way, possible bugs could be weeded out before the jessie
release. This would also benefit our Google Summer of Code student for the
"Bootstrappable Debian" project whose patches could then immediately be applied
by package maintainers instead of his work only bearing fruit after the jessie
release.
Sorry at the deity and dpkg lists for posting those patches once again but
small changes were required to the debian/changelog versions, the Build-Depends
version of python-apt, the symbol version in apt and the newer version of dpkg
in stable required some refreshing of the patch offsets.
Thank you for your consideration.
cheers, josch
diff -Nru dpkg-1.16.15/debian/changelog dpkg-1.16.15+deb7u1/debian/changelog
--- dpkg-1.16.15/debian/changelog 2014-06-05 20:36:49.000000000 +0000
+++ dpkg-1.16.15+deb7u1/debian/changelog 2014-07-28 11:18:23.000000000 +0000
@@ -1,3 +1,19 @@
+dpkg (1.16.15+deb7u1) stable; urgency=low
+
+ * Non-maintainer upload.
+ * Minimal support for restriction list Build-Depends syntax extension
+ - the syntax is described at https://wiki.debian.org/BuildProfileSpec
+ - dpkg will correctly parse and interpret build profile annotations
+ in build dependencies of source packages
+ - it will not allow to set any build profiles and will always assume
+ that no build profile is set
+ - this allows dpkg to not throw an error when encountering source packages
+ that make use of build profiles in stable versions of Debian with dpkg
+ version 1.17.2 or later in which full support for build profiles
+ was implemented
+
+ -- Johannes Schauer <j.schauer@email.de> Mon, 28 Jul 2014 10:47:00 +0000
+
dpkg (1.16.15) wheezy-security; urgency=high
[ Guillem Jover ]
diff -Nru dpkg-1.16.15/man/deb-src-control.5 dpkg-1.16.15+deb7u1/man/deb-src-control.5
--- dpkg-1.16.15/man/deb-src-control.5 2014-06-05 20:01:05.000000000 +0000
+++ dpkg-1.16.15+deb7u1/man/deb-src-control.5 2014-07-28 10:30:33.000000000 +0000
@@ -177,8 +177,9 @@
of packages separated by vertical bar (or "pipe") symbols, "|". The
groups are separated by commas. Commas are to be read as "AND", and pipes
as "OR", with pipes binding more tightly. Each package name is
-optionally followed by a version number specification in parentheses and an
-architecture specification in square brackets.
+optionally followed by a version number specification in parentheses, an
+architecture specification in square brackets, and a profile specification
+in angle brackets.
The syntax of the
.BR Build\-Conflicts ,
@@ -188,7 +189,8 @@
fields is a list of comma-separated package names, where the comma is read
as an "AND". Specifying alternative packages using a "pipe" is not supported.
Each package name is optionally followed by a version number specification in
-parentheses and an architecture specification in square brackets.
+parentheses, an architecture specification in square brackets, and a profile
+specification in angle brackets.
A version number may start with a ">>", in which case any later version
will match, and may specify or omit the Debian packaging revision (separated
@@ -200,6 +202,10 @@
separated by whitespace. Exclamation marks may be prepended to each of the
names, meaning "NOT".
+A profile specification consists of one or more profile names, prefixed
+with the "\fBprofile.\fP" namespace, separated by whitespace. Exclamation
+marks may be prepended to each of the names, meaning "NOT".
+
Note that dependencies on packages in the
.B build\-essential
set can be omitted and that declaring build conflicts against them is
diff -Nru dpkg-1.16.15/scripts/Dpkg/Deps.pm dpkg-1.16.15+deb7u1/scripts/Dpkg/Deps.pm
--- dpkg-1.16.15/scripts/Dpkg/Deps.pm 2014-06-05 20:01:05.000000000 +0000
+++ dpkg-1.16.15+deb7u1/scripts/Dpkg/Deps.pm 2014-07-28 10:30:33.000000000 +0000
@@ -267,6 +267,26 @@
list so that the resulting dependencies are directly applicable to the
current architecture.
+=item use_profiles (defaults to 1)
+
+Take into account the profile restriction part of the dependencies. Set
+to 0 to completely ignore that information.
+
+=item reduce_profiles (defaults to 0)
+
+If set to 1, ignore dependencies that do not concern the current build
+profile. This implicitly strips off the profile restriction list so
+that the resulting dependencies are directly applicable to the current
+profiles.
+
+=item reduce_restrictions (defaults to 0)
+
+If set to 1, ignore dependencies that do not concern the current set of
+restrictions. This implicitly strips off any restriction list so that the
+resulting dependencies are directly applicable to the current restriction.
+This currently implies C<reduce_arch> and C<reduce_profiles>, and overrides
+them if set.
+
=item union (defaults to 0)
If set to 1, returns a Dpkg::Deps::Union instead of a Dpkg::Deps::AND. Use
@@ -287,9 +307,17 @@
$options{use_arch} = 1 if not exists $options{use_arch};
$options{reduce_arch} = 0 if not exists $options{reduce_arch};
$options{host_arch} = get_host_arch() if not exists $options{host_arch};
+ $options{use_profiles} = 1 if not exists $options{use_profiles};
+ $options{reduce_profiles} = 0 if not exists $options{reduce_profiles};
+ $options{reduce_restrictions} = 0 if not exists $options{reduce_restrictions};
$options{union} = 0 if not exists $options{union};
$options{build_dep} = 0 if not exists $options{build_dep};
+ if ($options{reduce_restrictions}) {
+ $options{reduce_arch} = 1;
+ $options{reduce_profiles} = 1;
+ }
+
# Strip trailing/leading spaces
$dep_line =~ s/^\s+//;
$dep_line =~ s/\s+$//;
@@ -311,6 +339,11 @@
$dep_simple->reduce_arch($options{host_arch});
next if not $dep_simple->arch_is_concerned($options{host_arch});
}
+ $dep_simple->{restrictions} = undef if not $options{use_profiles};
+ if ($options{reduce_profiles}) {
+ $dep_simple->reduce_profiles($options{build_profiles});
+ next if not $dep_simple->profile_is_concerned($options{build_profiles});
+ }
push @or_list, $dep_simple;
}
next if not @or_list;
@@ -562,6 +595,7 @@
$self->{'version'} = undef;
$self->{'arches'} = undef;
$self->{'archqual'} = undef;
+ $self->{'restrictions'} = undef;
}
sub parse {
@@ -591,6 +625,11 @@
\s* (.*?) # don't parse architectures now
\s* \] # closing bracket
)? # end of optional architecture
+ (?: # start of optional restriction
+ \s* < # open bracket for restriction
+ \s* (.*?) # don't parse restrictions now
+ \s* > # closing bracket
+ )? # end of optional restriction
\s*$ # trailing spaces at end
/x;
if (defined($2)) {
@@ -605,6 +644,9 @@
if (defined($5)) {
$self->{arches} = [ split(/\s+/, $5) ];
}
+ if (defined($6)) {
+ $self->{restrictions} = [ map { lc } split /\s+/, $6 ];
+ }
}
sub output {
@@ -619,6 +661,9 @@
if (defined($self->{'arches'})) {
$res .= " [" . join(" ", @{$self->{arches}}) . "]";
}
+ if (defined($self->{restrictions})) {
+ $res .= ' <' . join(' ', @{$self->{restrictions}}) . '>';
+ }
if (defined($fh)) {
print $fh $res;
}
@@ -748,6 +793,39 @@
}
}
+sub profile_is_concerned {
+ my ($self) = @_;
+
+ return 0 if not defined $self->{package}; # Empty dep
+ return 1 if not defined $self->{restrictions}; # Dep without restrictions
+
+ my $seen_profile = 0;
+ foreach my $restriction (@{$self->{restrictions}}) {
+ # Determine if this restriction is negated, and within the "profile"
+ # namespace, otherwise it does not concern this check.
+ next if $restriction !~ m/^(!)?profile\.(.*)/;
+
+ my $negated = defined $1 && $1 eq '!';
+
+ if ($negated) {
+ # "!profile.this" includes by default all other profiles
+ # unless they also appear in a "!profile.other".
+ $seen_profile = 1;
+ }
+ }
+ return $seen_profile;
+}
+
+sub reduce_profiles {
+ my ($self) = @_;
+
+ if (not $self->profile_is_concerned()) {
+ $self->reset();
+ } else {
+ $self->{restrictions} = undef;
+ }
+}
+
sub get_evaluation {
my ($self, $facts) = @_;
return undef if not defined $self->{package};
diff -Nru dpkg-1.16.15/scripts/dpkg-checkbuilddeps.pl dpkg-1.16.15+deb7u1/scripts/dpkg-checkbuilddeps.pl
--- dpkg-1.16.15/scripts/dpkg-checkbuilddeps.pl 2014-06-05 20:01:06.000000000 +0000
+++ dpkg-1.16.15+deb7u1/scripts/dpkg-checkbuilddeps.pl 2014-07-28 10:30:33.000000000 +0000
@@ -113,11 +113,13 @@
if ($bd_value) {
push @unmet, build_depends('Build-Depends/Build-Depends-Arch/Build-Depends-Indep',
deps_parse($bd_value, build_dep => 1, host_arch => $host_arch,
+ reduce_profiles => 1,
reduce_arch => 1), $facts);
}
if ($bc_value) {
push @conflicts, build_conflicts('Build-Conflicts/Build-Conflicts-Arch/Build-Conflicts-Indep',
deps_parse($bc_value, build_dep => 1, host_arch => $host_arch,
+ reduce_profiles => 1,
reduce_arch => 1, union => 1), $facts);
}
diff -Nru dpkg-1.16.15/scripts/dpkg-gencontrol.pl dpkg-1.16.15+deb7u1/scripts/dpkg-gencontrol.pl
--- dpkg-1.16.15/scripts/dpkg-gencontrol.pl 2014-06-05 20:01:06.000000000 +0000
+++ dpkg-1.16.15+deb7u1/scripts/dpkg-gencontrol.pl 2014-07-28 10:30:33.000000000 +0000
@@ -247,6 +247,7 @@
$fields->{'Architecture'}, $fields->{'Multi-Arch'});
if (exists $pkg->{"Provides"}) {
my $provides = deps_parse($substvars->substvars($pkg->{"Provides"}, no_warn => 1),
+ reduce_profiles => 1,
reduce_arch => 1, union => 1);
if (defined $provides) {
foreach my $subdep ($provides->get_deps()) {
@@ -269,6 +270,7 @@
msg_prefix => sprintf(_g("%s field of package %s: "), $field, $pkg->{Package}));
if (field_get_dep_type($field) eq 'normal') {
$dep = deps_parse($field_value, use_arch => 1,
+ reduce_profiles => 1,
reduce_arch => $reduce_arch);
error(_g("error occurred while parsing %s field: %s"), $field,
$field_value) unless defined $dep;
@@ -277,6 +279,7 @@
push @seen_deps, $dep;
} else {
$dep = deps_parse($field_value, use_arch => 1,
+ reduce_profiles => 1,
reduce_arch => $reduce_arch, union => 1);
error(_g("error occurred while parsing %s field: %s"), $field,
$field_value) unless defined $dep;
diff -Nru dpkg-1.16.15/scripts/dpkg-shlibdeps.pl dpkg-1.16.15+deb7u1/scripts/dpkg-shlibdeps.pl
--- dpkg-1.16.15/scripts/dpkg-shlibdeps.pl 2014-06-05 20:01:06.000000000 +0000
+++ dpkg-1.16.15+deb7u1/scripts/dpkg-shlibdeps.pl 2014-07-28 10:30:33.000000000 +0000
@@ -141,7 +141,7 @@
my $fields = $control->get_source();
my $build_depends = defined($fields->{"Build-Depends"}) ?
$fields->{"Build-Depends"} : "";
-my $build_deps = deps_parse($build_depends, build_dep => 1, reduce_arch => 1);
+my $build_deps = deps_parse($build_depends, build_dep => 1, reduce_restrictions => 1);
error(_g("error occurred while parsing %s"), "Build-Depends") unless defined $build_deps;
my %dependencies;
diff -Nru apt-0.9.7.9+deb7u2/apt-pkg/deb/deblistparser.cc apt-0.9.7.9+deb7u3/apt-pkg/deb/deblistparser.cc
--- apt-0.9.7.9+deb7u2/apt-pkg/deb/deblistparser.cc 2013-11-16 11:43:40.000000000 +0000
+++ apt-0.9.7.9+deb7u3/apt-pkg/deb/deblistparser.cc 2014-07-28 11:32:23.000000000 +0000
@@ -473,6 +473,12 @@
string &Package,string &Ver,
unsigned int &Op, bool const &ParseArchFlags,
bool const &StripMultiArch)
+ { return ParseDepends(Start, Stop, Package, Ver, Op, ParseArchFlags, StripMultiArch, false); }
+const char *debListParser::ParseDepends(const char *Start,const char *Stop,
+ string &Package,string &Ver,
+ unsigned int &Op, bool const &ParseArchFlags,
+ bool const &StripMultiArch,
+ bool const &ParseRestrictionsList)
{
// Strip off leading space
for (;Start != Stop && isspace(*Start) != 0; Start++);
@@ -480,7 +486,8 @@
// Parse off the package name
const char *I = Start;
for (;I != Stop && isspace(*I) == 0 && *I != '(' && *I != ')' &&
- *I != ',' && *I != '|' && *I != '[' && *I != ']'; I++);
+ *I != ',' && *I != '|' && *I != '[' && *I != ']' &&
+ *I != '<' && *I != '>'; ++I);
// Malformed, no '('
if (I != Stop && *I == ')')
@@ -594,6 +601,53 @@
}
// Skip whitespace
+ for (;I != Stop && isspace(*I) != 0; I++);
+ }
+
+ if (ParseRestrictionsList == true)
+ {
+ // Parse a restrictions list
+ if (I != Stop && *I == '<')
+ {
+ ++I;
+ // malformed
+ if (unlikely(I == Stop))
+ return 0;
+
+ const char *End = I;
+ bool Found = false;
+ bool NegRestriction = false;
+ while (I != Stop)
+ {
+ // look for whitespace or ending '>'
+ for (;End != Stop && !isspace(*End) && *End != '>'; ++End);
+
+ if (unlikely(End == Stop))
+ return 0;
+
+ if (*I == '!')
+ {
+ NegRestriction = true;
+ ++I;
+ }
+
+ if (*End++ == '>') {
+ I = End;
+ break;
+ }
+
+ I = End;
+ for (;I != Stop && isspace(*I) != 0; I++);
+ }
+
+ if (NegRestriction == true)
+ Found = !Found;
+
+ if (Found == false)
+ Package = ""; /* not for this restriction */
+ }
+
+ // Skip whitespace
for (;I != Stop && isspace(*I) != 0; I++);
}
diff -Nru apt-0.9.7.9+deb7u2/apt-pkg/deb/deblistparser.h apt-0.9.7.9+deb7u3/apt-pkg/deb/deblistparser.h
--- apt-0.9.7.9+deb7u2/apt-pkg/deb/deblistparser.h 2013-03-01 10:51:21.000000000 +0000
+++ apt-0.9.7.9+deb7u3/apt-pkg/deb/deblistparser.h 2014-07-28 11:32:23.000000000 +0000
@@ -72,11 +72,16 @@
bool LoadReleaseInfo(pkgCache::PkgFileIterator &FileI,FileFd &File,
std::string section);
-
+
static const char *ParseDepends(const char *Start,const char *Stop,
std::string &Package,std::string &Ver,unsigned int &Op,
bool const &ParseArchFlags = false,
bool const &StripMultiArch = true);
+ static const char *ParseDepends(const char *Start,const char *Stop,
+ std::string &Package,std::string &Ver,unsigned int &Op,
+ bool const &ParseArchFlags, bool const &StripMultiArch,
+ bool const &ParseRestrictionsList);
+
static const char *ConvertRelation(const char *I,unsigned int &Op);
debListParser(FileFd *File, std::string const &Arch = "");
diff -Nru apt-0.9.7.9+deb7u2/apt-pkg/deb/debsrcrecords.cc apt-0.9.7.9+deb7u3/apt-pkg/deb/debsrcrecords.cc
--- apt-0.9.7.9+deb7u2/apt-pkg/deb/debsrcrecords.cc 2013-03-01 10:51:21.000000000 +0000
+++ apt-0.9.7.9+deb7u3/apt-pkg/deb/debsrcrecords.cc 2014-07-28 11:32:23.000000000 +0000
@@ -90,7 +90,7 @@
while (1)
{
Start = debListParser::ParseDepends(Start, Stop,
- rec.Package,rec.Version,rec.Op,true, StripMultiArch);
+ rec.Package,rec.Version,rec.Op,true, StripMultiArch,true);
if (Start == 0)
return _error->Error("Problem parsing dependency: %s", fields[I]);
diff -Nru apt-0.9.7.9+deb7u2/debian/changelog apt-0.9.7.9+deb7u3/debian/changelog
--- apt-0.9.7.9+deb7u2/debian/changelog 2014-06-12 15:16:42.000000000 +0000
+++ apt-0.9.7.9+deb7u3/debian/changelog 2014-07-28 11:35:59.000000000 +0000
@@ -1,3 +1,19 @@
+apt (0.9.7.9+deb7u3) stable; urgency=low
+
+ * Non-maintainer upload.
+ * Minimal support for restriction list Build-Depends syntax extension
+ - the syntax is described at https://wiki.debian.org/BuildProfileSpec
+ - apt will correctly parse and interpret build profile annotations
+ in build dependencies of source packages
+ - it will not allow to set any build profiles and will always assume
+ that no build profile is set
+ - this allows apt to not throw an error when encountering source packages
+ that make use of build profiles in stable versions of Debian with apt
+ version 0.9.16.1 or later in which full support for build profiles
+ was implemented
+
+ -- Johannes Schauer <j.schauer@email.de> Mon, 28 Jul 2014 11:32:35 +0000
+
apt (0.9.7.9+deb7u2) wheezy-security; urgency=high
* SECURITY UPDATE: apt-get source validation (closes: #749795)
diff -Nru apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols
--- apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols 2013-03-01 10:51:21.000000000 +0000
+++ apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols 2014-07-28 11:32:23.000000000 +0000
@@ -369,6 +369,7 @@
(c++)"debListParser::VersionHash()@Base" 0.8.0
(c++)"debListParser::Architecture()@Base" 0.8.0
(c++)"debListParser::ParseDepends(char const*, char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int&, bool const&, bool const&)@Base" 0.8.0
+ (c++)"debListParser::ParseDepends(char const*, char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned int&, bool const&, bool const&, bool const&)@Base" 0.9.7.9+deb7u2
(c++)"debListParser::ParseDepends(pkgCache::VerIterator&, char const*, unsigned int)@Base" 0.8.0
(c++)"debListParser::ParseProvides(pkgCache::VerIterator&)@Base" 0.8.0
(c++)"debListParser::ArchitectureAll()@Base" 0.8.0
diff -Nru python-apt-0.8.8.2/debian/changelog python-apt-0.8.8.2+deb7u1/debian/changelog
--- python-apt-0.8.8.2/debian/changelog 2013-03-14 20:25:26.000000000 +0000
+++ python-apt-0.8.8.2+deb7u1/debian/changelog 2014-07-28 11:46:26.000000000 +0000
@@ -1,3 +1,19 @@
+python-apt (0.8.8.2+deb7u1) stable; urgency=low
+
+ * Non-maintainer upload.
+ * Minimal support for restriction list Build-Depends syntax extension
+ - the syntax is described at https://wiki.debian.org/BuildProfileSpec
+ - apt will correctly parse and interpret build profile annotations
+ in build dependencies of source packages
+ - it will not allow to set any build profiles and will always assume
+ that no build profile is set
+ - this allows apt to not throw an error when encountering source packages
+ that make use of build profiles in stable versions of Debian with apt
+ version 0.9.16.1 or later in which full support for build profiles
+ was implemented
+
+ -- Johannes Schauer <j.schauer@email.de> Mon, 28 Jul 2014 11:45:53 +0000
+
python-apt (0.8.8.2) unstable; urgency=low
[ David Prévot ]
diff -Nru python-apt-0.8.8.2/debian/control python-apt-0.8.8.2+deb7u1/debian/control
--- python-apt-0.8.8.2/debian/control 2013-03-13 22:25:59.000000000 +0000
+++ python-apt-0.8.8.2+deb7u1/debian/control 2014-07-28 11:46:59.000000000 +0000
@@ -10,7 +10,7 @@
apt-utils,
debhelper (>= 7.3.5),
fakeroot,
- libapt-pkg-dev (>= 0.8.11),
+ libapt-pkg-dev (= 0.9.7.9+deb7u3),
python-all-dev (>= 2.6.6-3~),
python-all-dbg,
python3-all-dev (>= 3.1.2-10~),
diff -Nru python-apt-0.8.8.2/python/apt_pkgmodule.cc python-apt-0.8.8.2+deb7u1/python/apt_pkgmodule.cc
--- python-apt-0.8.8.2/python/apt_pkgmodule.cc 2013-03-13 22:31:33.000000000 +0000
+++ python-apt-0.8.8.2+deb7u1/python/apt_pkgmodule.cc 2014-07-28 11:45:11.000000000 +0000
@@ -186,7 +186,7 @@
"only contains those dependencies for the architecture set in the\n"
"configuration variable APT::Architecture";
static PyObject *RealParseDepends(PyObject *Self,PyObject *Args,
- bool ParseArchFlags, std::string name,
+ bool ParseArchFlags, bool ParseRestrictionsList, std::string name,
bool debStyle=false)
{
std::string Package;
@@ -210,7 +210,7 @@
break;
Start = debListParser::ParseDepends(Start,Stop,Package,Version,Op,
- ParseArchFlags, StripMultiArch);
+ ParseArchFlags, StripMultiArch, ParseRestrictionsList);
if (Start == 0)
{
PyErr_SetString(PyExc_ValueError,"Problem Parsing Dependency");
@@ -243,11 +243,11 @@
}
static PyObject *ParseDepends(PyObject *Self,PyObject *Args)
{
- return RealParseDepends(Self, Args, false, "parse_depends");
+ return RealParseDepends(Self, Args, false, false, "parse_depends");
}
static PyObject *ParseSrcDepends(PyObject *Self,PyObject *Args)
{
- return RealParseDepends(Self, Args, true, "parse_src_depends");
+ return RealParseDepends(Self, Args, true, true, "parse_src_depends");
}
#ifdef COMPAT_0_7
static PyObject *ParseDepends_old(PyObject *Self,PyObject *Args)
Reply to: