Bug#785735: wheezy-pu: debmirror 2.16 for wheezy
On Wed, 27 May 2015, Adam D. Barratt wrote:
> Control: tags -1 + moreinfo
>
> On 2015-05-19 18:56, Santiago Vila wrote:
> > The debmirror package in wheezy is unable to mirror wheezy itself.
> > There is a request for a backport here:
> >
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749734
> >
> > but IMHO, it would even better to just have 2.16 in wheezy.
> >
> > With this bug I'm asking for permission to upload 2.16 for oldstable
> > (package does not need any change at all, just a rebuild with a proper
> > changelog entry).
>
> The interdiff doesn't look crazy, but please could we have a debdiff of the
> proposed - and tested :-) - package for final confirmation?
Sorry for the delay, here it is.
Debdiff from 2.14, which is what we have in wheezy right now.
I've tested the package and it works. On the other side, this is
essentially the same package we have in jessie, so the code has been
really tested by a lot of people since 2013.
Thanks.
diff -Nru debmirror-2.14/debian/changelog debmirror-2.16+deb7u1/debian/changelog
--- debmirror-2.14/debian/changelog 2012-06-27 01:29:34.000000000 +0200
+++ debmirror-2.16+deb7u1/debian/changelog 2015-05-27 18:00:12.000000000 +0200
@@ -1,3 +1,31 @@
+debmirror (1:2.16+deb7u1) oldstable; urgency=low
+
+ * Upload for wheezy. Closes: #749734.
+
+ -- Santiago Vila <sanvila@debian.org> Wed, 27 May 2015 17:59:12 +0200
+
+debmirror (1:2.16) unstable; urgency=low
+
+ * Fix confusing output with --precleanup.
+ Closes: #708355
+ * Support new Contents file location used for wheezy, while still
+ handling the old location.
+ Closes: #637442 Thanks, Christoph Pleger
+
+ -- Joey Hess <joeyh@debian.org> Mon, 26 Aug 2013 12:41:03 -0400
+
+debmirror (1:2.15) unstable; urgency=low
+
+ * Improved interface to gpgv. Thanks, Tom Jones.
+ * Add --keyring option. Thanks, Tom Jones.
+ * Add --exclude-field and --include-field options.
+ Closes: #695767. Thanks, Colin Watson
+ * Supports https. Closes: #697687 Thanks, Fernando Ike
+ * Treat "Origin: Canonical" like "Origin: Ubuntu"
+ Closes: #702319. Thanks, Tal Danzig
+
+ -- Joey Hess <joeyh@debian.org> Sat, 04 May 2013 23:44:27 -0400
+
debmirror (1:2.14) unstable; urgency=low
* Add --config-file option.
diff -Nru debmirror-2.14/debmirror debmirror-2.16+deb7u1/debmirror
--- debmirror-2.14/debmirror 2012-06-27 01:28:46.000000000 +0200
+++ debmirror-2.16+deb7u1/debmirror 2013-06-23 18:34:26.000000000 +0200
@@ -103,7 +103,7 @@
=item B<--method>=I<method>
Specify the method to download files. Currently, supported methods are
-B<ftp>, B<http>, and B<rsync>.
+B<ftp>, B<http>, B<https>, and B<rsync>.
=item B<--passive>
@@ -251,6 +251,19 @@
Controls whether gpg signatures from the F<Release.gpg> file should be
checked. The default is to check signatures.
+=item B<--keyring>=I<file>
+
+Use I<file> as an additional gpg-format keyring. May be given multiple times.
+
+Note that these will be used in addition to $GNUPGHOME/trustedkeys.gpg.
+The latter can be removed from the set of keyrings by setting
+$GNUPGHOME to something non-existent when using this option.
+
+On a typical Debian system, the Debian archive keyring can be used
+directly with this option:
+
+ debmirror --keyring /usr/share/keyrings/debian-archive-keyring.gpg ...
+
=item B<--ignore-release-gpg>
Don't fail if the F<Release.gpg> file is missing. If the file does exist, it
@@ -278,6 +291,20 @@
Limit download to files whose Debian Priority (required, extra,
optional, ...) match the regex. May be used multiple times.
+=item B<--exclude-field>=I<fieldname>=I<regex>
+
+Never download any binary packages where the contents of I<fieldname> match
+the regex. May be used multiple times. If this option is used and the mirror
+includes source packages, only those source packages corresponding to
+included binary packages will be downloaded.
+
+=item B<--include-field>=I<fieldname>=I<regex>
+
+Don't exclude any binary packages where the contents of I<fieldname> match
+the regex. May be used multiple times. If this option is used and the mirror
+includes source packages, only those source packages corresponding to
+included binary packages will be downloaded.
+
=item B<-t>, B<--timeout>=I<seconds>
Specifies the timeout to use for network operations (either FTP or rsync).
@@ -402,6 +429,13 @@
will automatically create a new symlink S<suite -E<gt> codename> (immediately
after moving meta files in place). This conversion should only be needed once.
+=item B<--disable-ssl-verification>
+
+When https is used, debmirror checks that the SSL certificate is value.
+
+If the server has a self-signed certificate, the check can be disabled
+with this option.
+
=item B<--debmarshal>
On each pull, keep the repository meta data from dists/* in a numbered
@@ -509,7 +543,8 @@
debmirror uses gpgv to verify Release and Release.gpg using the
default keying ~/.gnupg/trustedkeys.gpg. This can be changed by
exporting GNUPGHOME resulting in $GNUPGHOME/trustedkeys.gpg being
- used.
+ used. (Note that keyring files can also be specified directly
+ with debmirror's --keyring option -- see above).
To add the right key to this keyring you can import it from the
debian keyring (in case of the debian archive) using:
@@ -533,6 +568,8 @@
use Getopt::Long;
use File::Temp qw/ tempfile /;
use File::Path qw(make_path);
+use IO::Pipe;
+use IO::Select;
use LockFile::Simple;
use Compress::Zlib;
use Digest::MD5;
@@ -546,8 +583,9 @@
our @config_files;
our ($debug, $progress, $verbose, $passive, $skippackages, $getcontents, $i18n);
our ($ua, $proxy, $ftp);
-our (@dists, @sections, @arches, @ignores, @excludes, @includes);
+our (@dists, @sections, @arches, @ignores, @excludes, @includes, @keyrings);
our (@excludes_deb_section, @limit_priority);
+our (%excludes_field, %includes_field);
our (@di_dists, @di_arches, @rsync_extra);
our $state_cache_days = 0;
our $verify_checksums = 0;
@@ -580,6 +618,7 @@
our $omit_suite_symlinks=0;
our $allow_dist_rename=0;
our $debmarshal=0;
+our $disable_ssl_verification;
our $slow_cpu=0;
our $check_gpg=1;
our $new_mirror=0;
@@ -671,6 +710,8 @@
'exclude-deb-section=s' => \@excludes_deb_section,
'limit-priority=s' => \@limit_priority,
'include=s' => \@includes,
+ 'exclude-field=s' => \%excludes_field,
+ 'include-field=s' => \%includes_field,
'skippackages' => \$skippackages,
'i18n' => \$i18n,
'getcontents' => \$getcontents,
@@ -692,6 +733,8 @@
'allow-dist-rename' => \$allow_dist_rename,
'debmarshal' => \$debmarshal,
'slow-cpu' => \$slow_cpu,
+ 'disable-ssl-verification' => \$disable_ssl_verification,
+ 'keyring=s' => \@keyrings,
'help' => \$help,
) or usage;
usage if $help;
@@ -783,14 +826,14 @@
say("Download at most $rsync_batch files per rsync call.") if ($download_method eq "rsync");
if ($pre_cleanup) {
say("Will clean up before mirroring.");
-}
-if ($post_cleanup) {
+} elsif ($post_cleanup) {
say("Will clean up after mirroring.");
} else {
say("Will NOT clean up.");
}
say("Dry run.") if $dry_run;
say("Debmarshal snapshots kept.") if $debmarshal;
+say("Disable SSL verification.") if $disable_ssl_verification;
# Set up mirror directory and resolve $mirrordir to a full path for
# locking and rsync
@@ -843,6 +886,17 @@
$ua->show_progress($progress);
return;
};
+
+ /^https$/ && do {
+ $ua = LWP::UserAgent->new(keep_alive => 1, ssl_opts => {
+ verify_hostname => ! $disable_ssl_verification });
+ $ua->timeout($timeout);
+ $ua->proxy('https', $ENV{https_proxy}) if $ENV{https_proxy};
+ $ua->proxy('https', $proxy) if $proxy;
+ $ua->show_progress($progress);
+ return;
+ };
+
/^ftp$/ && do {
if ($proxy || $ENV{ftp_proxy}) {
@@ -1049,12 +1103,21 @@
next if $dist=~/experimental/;
next if $dist=~/.*-proposed-updates/;
next if $arch=~/source/;
- if ($diff_mode ne "none") {
- if (!update_contents("dists/$dist", "Contents-$arch")) {
- add_bytes("dists/$dist/Contents-$arch.gz");
- }
- } elsif (!check_lists("$tempdir/dists/$dist/Contents-$arch.gz")) {
- add_bytes("dists/$dist/Contents-$arch.gz");
+ # In Debian Wheezy, the Contents-*.gz moved to '/dists/$dist/$sect/'.
+ # This handles the new location, but also checks the old location
+ # for backwards compatibility.
+ push my @sects, @sections, "";
+ foreach my $sect (@sects) {
+ if ($sect ne "") {$sect = "/$sect";}
+ if (exists $file_lists{"$tempdir/dists/$dist${sect}Contents-$arch.gz"}) {
+ if ($diff_mode ne "none") {
+ if (!update_contents("dists/$dist$sect", "Contents-$arch")) {
+ add_bytes("dists/$dist$sect/Contents-$arch.gz");
+ }
+ } elsif (!check_lists("$tempdir/dists/$dist$sect/Contents-$arch.gz")) {
+ add_bytes("dists/$dist$sect/Contents-$arch.gz");
+ }
+ }
}
}
}
@@ -1082,6 +1145,9 @@
my $exclude_deb_section =
"(".join("|", @excludes_deb_section).")" if @excludes_deb_section;
my $limit_priority = "(".join("|", @limit_priority).")" if @limit_priority;
+ my $field_filters =
+ scalar(keys %includes_field) || scalar(keys %excludes_field);
+ my %binaries;
foreach my $file (@package_files) {
next if (!-f $file);
@@ -1104,6 +1170,9 @@
next if (defined($limit_priority) && defined($deb_priority)
&& ! ($deb_priority=~/$limit_priority/o));
}
+ next if $field_filters && !check_field_filters($_);
+ my ($package)=m/^Package:\s+(.*)/im;
+ $binaries{$package} = 1;
# File was listed in state cache, or file occurs multiple times
if (exists $files{$filename}) {
if ($files{$filename} >= 0) {
@@ -1131,9 +1200,10 @@
}
close(FILE);
}
-SOURCE: foreach my $file (@source_files) {
+ foreach my $file (@source_files) {
next if (!-f $file);
open(FILE, "<", $file) or die "$file: $!";
+SOURCE:
for (;;) {
my $stanza;
unless (defined( $stanza = <FILE> )) {
@@ -1169,6 +1239,19 @@
next SOURCE if (defined($limit_priority) && defined($deb_priority)
&& ! ($deb_priority=~/$limit_priority/o));
}
+ elsif ($line=~/^Binary:\s+(.*)/i) {
+ if ($field_filters) {
+ my @binary_names=split(/\s*,\s*/,$1);
+ my $fetching_binary=0;
+ for my $binary_name (@binary_names) {
+ if (exists $binaries{$binary_name}) {
+ $fetching_binary=1;
+ last;
+ }
+ }
+ next SOURCE unless $fetching_binary;
+ }
+ }
elsif ($line=~/^Files:/i) {
$parse_source_files->("MD5Sum");
}
@@ -1238,7 +1321,8 @@
batch_get();
sub batch_get {
- if ($download_method eq 'ftp' || $download_method eq 'http') {
+ if ($download_method eq 'ftp' || $download_method eq 'http' ||
+ $download_method eq 'https') {
my $dirname;
my $i=0;
foreach my $file (sort keys %files) {
@@ -1471,6 +1555,26 @@
}
}
+# Return true if a package stanza is permitted by
+# --include-field/--exclude-field.
+sub check_field_filters {
+ my $stanza = shift;
+ for my $name (keys %includes_field) {
+ if ($stanza=~/^\Q$name\E:\s+(.*)/im) {
+ my $value=$1;
+ return 1 if $value=~/$includes_field{$name}/;
+ }
+ }
+ return 0 if keys %includes_field;
+ for my $name (keys %excludes_field) {
+ if ($stanza=~/^\Q$name\E:\s+(.*)/im) {
+ my $value=$1;
+ return 0 if $value=~/$excludes_field{$name}/;
+ }
+ }
+ return 1;
+}
+
# Takes named parameters: filename, size.
#
# Optionally can also be passed parameters specifying expected checksums
@@ -1577,8 +1681,9 @@
$tdir=$tempdir unless $tdir;
chdir($tdir) or die "unable to chdir($tdir): $!\n";
- if ($download_method eq 'ftp' || $download_method eq 'http') {
- $res=$ftp ? ftp_get($file) : http_get($file);
+ if ($download_method eq 'ftp' || $download_method eq 'http' ||
+ $download_method eq 'https') {
+ $res=$ftp ? ftp_get($file) : http_get($file);
$res=$res && check_lists($file);
if (-f $file && !$res) {
say("$file failed checksum verification, removing");
@@ -1865,10 +1970,16 @@
sub link_contents_into_snapshot {
my ($dist,$mirrordir,$arch,$tempdir) = @_;
my $next = get_next_snapshot($dist);
- unlink("$mirrordir/dists/$dist/$next/Contents-$arch.gz");
- link("$tempdir/dists/$dist/Contents-$arch.gz",
- "$mirrordir/dists/$dist/$next/Contents-$arch.gz")
- or die "Error while linking $tempdir/dists/$dist/Contents-$arch.gz: $!\n";
+ push my @sects, @sections, "";
+ foreach my $sect (@sects) {
+ if ($sect ne "") {$sect = "/$sect";}
+ if (exists $file_lists{"$tempdir/dists/$dist$sect/Contents-$arch.gz"}) {
+ unlink("$mirrordir/dists/$dist/$next$sect/Contents-$arch.gz");
+ link("$tempdir/dists/$dist$sect/Contents-$arch.gz",
+ "$mirrordir/dists/$dist/$next$sect/Contents-$arch.gz")
+ or die "Error while linking $tempdir/dists/$dist$sect/Contents-$arch.gz: $!\n";
+ }
+ }
}
sub link_translation_into_snapshot {
@@ -1906,17 +2017,57 @@
} else {
# Verify Release signature
my $gpgv_res = 0;
- open GPGV, "gpgv 2>/dev/null --status-fd 1 $tdir/Release.gpg $tdir/Release|";
- while (<GPGV>) {
- $gpgv_res = 1 if /^\[GNUPG:\] VALIDSIG/;
+ my $outp = IO::Pipe->new;
+ my $errp = IO::Pipe->new;
+ my $gpgvout = "";
+ my $gpgverr = "";
+ if (my $child = fork) {
+ $outp->reader;
+ $errp->reader;
+ my $sel = IO::Select->new;
+ $sel->add($outp, $errp);
+ while (my @ready = $sel->can_read) {
+ for (@ready) {
+ my $buf = "";
+ my $bytesread = $_->read($buf, 1024);
+ if (!defined($bytesread)) {
+ die "read error: $!\n";
+ } elsif ($bytesread == 0) {
+ $sel->remove($_);
+ $_->close;
+ } else {
+ if ($_ == $outp) {
+ $gpgvout .= $buf;
+ }
+ if ($_ == $errp) {
+ $gpgverr .= $buf;
+ }
+ }
+ }
+ }
+
+ waitpid($child, 0) == -1
+ and die "was pid $child automatically reaped?\n";
+ $gpgv_res = not $?;
+ }
+ else {
+ $outp->writer;
+ $errp->writer;
+ STDOUT->fdopen(fileno($outp), "w") or die;
+ STDERR->fdopen(fileno($errp), "w") or die;
+ my @gpgv = qw(gpgv --status-fd 1);
+ push @gpgv, (map { ('--keyring' => $_) } @keyrings);
+ push @gpgv, "$tdir/Release.gpg", "$tdir/Release";
+ exec(@gpgv) or die "exec: $gpgv[0]: $!\n";
}
- close GPGV;
+
# In debug or verbose mode, display the gpg error message on stdout.
if (! $gpgv_res || $debug) {
- system("gpgv --status-fd 1 $tdir/Release.gpg $tdir/Release 2>&1");
+ print $gpgvout;
+ print $gpgverr;
}
if ($verbose && ! $debug) {
- system("gpgv --status-fd 1 $tdir/Release.gpg $tdir/Release 2>&1 >/dev/null");
+ print $gpgverr;
}
if (! $gpgv_res) {
say("Release gpg signature does not verify.");
@@ -1965,7 +2116,7 @@
if ($origin eq "none") {
$codename = $dist_raw;
- } elsif ($origin eq "Ubuntu") {
+ } elsif ($origin eq "Ubuntu" or $origin eq "Canonical") {
if ($suite) {
say("Ubuntu Release file: using Suite ($suite).");
$codename = $suite;
@@ -2176,17 +2327,23 @@
next if $dist=~/experimental/;
next if $dist=~/.*-proposed-updates/;
next if $arch=~/source/;
- if (!check_lists("$tempdir/dists/$dist/Contents-$arch.gz")) {
- if ($first) {
- say("Get Contents files.");
- $first = 0;
- }
- remote_get("dists/$dist/Contents-$arch.gz");
- }
- $files{"dists/$dist/Contents-$arch.gz"}=1;
- $files{$tempdir."/"."dists/$dist/Contents-$arch.gz"}=1;
- if ($debmarshal) {
- link_contents_into_snapshot($dist,$mirrordir,$arch,$tempdir);
+ push my @sects, @sections, "";
+ foreach my $sect (@sects) {
+ if ($sect ne "") {$sect = "/$sect";}
+ if (exists $file_lists{"$tempdir/dists/$dist$sect/Contents-$arch.gz"}) {
+ if (!check_lists("$tempdir/dists/$dist$sect/Contents-$arch.gz")) {
+ if ($first) {
+ say("Get Contents files.");
+ $first = 0;
+ }
+ remote_get("dists/$dist$sect/Contents-$arch.gz");
+ }
+ $files{"dists/$dist$sect/Contents-$arch.gz"}=1;
+ $files{$tempdir."/"."dists/$dist$sect/Contents-$arch.gz"}=1;
+ if ($debmarshal) {
+ link_contents_into_snapshot($dist,$mirrordir,$arch,$tempdir);
+ }
+ }
}
}
}
diff -Nru debmirror-2.14/doc/design.txt debmirror-2.16+deb7u1/doc/design.txt
--- debmirror-2.14/doc/design.txt 2010-09-07 00:36:10.000000000 +0200
+++ debmirror-2.16+deb7u1/doc/design.txt 2013-02-03 05:32:54.000000000 +0100
@@ -68,6 +68,7 @@
- ftp
- hftp (ftp via http://user:pass@proxy:port/)
- http
+ - https
- rsync
- wget (wget-ftp)
- print (output what should be done)
Reply to: