Checksum-Sha{256,1} vs Files missing in DSC / Sources
Hello,
While working on <https://bugs.debian.org/931729> in "apt-mirror" I
noticed a strange thing:
According to Debian Policy 4.5.0.3
<https://www.debian.org/doc/debian-policy/ch-controlfields.html#debian-source-control-files-dsc>
and <https://manpages.debian.org/testing/dpkg-dev/dsc.5.en.html>
"Files:" is still "mandatory, but reality looks different:
> $ curl -s http://ftp.de.debian.org/debian/dists/buster-updates/main/source/Sources.xz | xz -d | grep-dctrl -S tzdata
> Package: tzdata
> Binary: tzdata
> Version: 2019c-0+deb10u1
> Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
> Uploaders: Clint Adams <clint@debian.org>, Aurelien Jarno <aurel32@debian.org>, Adam Conrad <adconrad@0c3.net>
> Build-Depends: debhelper (>= 9)
> Build-Depends-Indep: po-debconf, rdfind, symlinks
> Architecture: all
> Standards-Version: 4.2.1
> Format: 3.0 (quilt)
> Vcs-Browser: https://salsa.debian.org/glibc-team/tzdata
> Vcs-Git: https://salsa.debian.org/glibc-team/tzdata.git
> Checksums-Sha256:
> 983c27d24d78c52d8f213b1b5800aaa90a171a4f805451b0845752f97c6f924b 2264 tzdata_2019c-0+deb10u1.dsc
> 79c7806dab09072308da0e3d22c37d3b245015a591891ea147d3b133b60ffc7c 392087 tzdata_2019c.orig.tar.gz
> cd31deaeee229d45e4f4b973441189e7619ef81679359e9c8b47b2a87aaf6a07 833 tzdata_2019c.orig.tar.gz.asc
> fa8071037767a7dfa054c26621c5079809ee038eddb32a58814faf3541d52d5a 104932 tzdata_2019c-0+deb10u1.debian.tar.xz
> Homepage: https://www.iana.org/time-zones
> Package-List:
> tzdata deb localization required arch=all
> Directory: pool/main/t/tzdata
> Priority: source
> Section: localization
(this is only one example; there are more entries where "Files:" is missing)
I remember a discussion to drop "Files:" in favor of "Checksum-Sha*",
but I'm unable to find the discussion and the conclusion again.
1. As "Files" is still mandatory can a mirror script assume the entry to
exist? (I already have a patch to try the Sha{1,256,512}, too)
2. If "Files" is still mandatory, why is it missing in the above entry?
Is there a bug in our build and/or mirror system?
Philipp
--
Philipp Hahn
Open Source Software Engineer
Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
📞 +49-421-22232-57
🖶 +49-421-22232-99
✉️ hahn@univention.de
🌐 https://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
From 2988f8d91dfbd97667be4224004deea85f888e66 Mon Sep 17 00:00:00 2001
Message-Id: <2988f8d91dfbd97667be4224004deea85f888e66.1601439012.git.hahn@univention.de>
In-Reply-To: <e4234896ee7051a8890e26b1736aff9041ed4547.1601439012.git.hahn@univention.de>
References: <e4234896ee7051a8890e26b1736aff9041ed4547.1601439012.git.hahn@univention.de>
From: Philipp Hahn <hahn@univention.de>
Date: Tue, 29 Sep 2020 12:40:43 +0200
Subject: Support SHA{1,256,512} for Sources too (closes #931729)
The real problem here is that process_index() assumes that "Files:" still used.
According to Debian Policy 4.5.0.3
<https://www.debian.org/doc/debian-policy/ch-controlfields.html#debian-source-control-files-dsc>
and <https://manpages.debian.org/testing/dpkg-dev/dsc.5.en.html> "Files:" is
still "mandatory, but reality looks different:
> $ curl -s http://ftp.de.debian.org/debian/dists/buster-updates/main/source/Sources.xz | xz -d | grep-dctrl -S tzdata
> Package: tzdata
...
> Version: 2019c-0+deb10u1
...
> Standards-Version: 4.2.1
...
> Checksums-Sha256:
> 983c27d24d78c52d8f213b1b5800aaa90a171a4f805451b0845752f97c6f924b 2264 tzdata_2019c-0+deb10u1.dsc
> 79c7806dab09072308da0e3d22c37d3b245015a591891ea147d3b133b60ffc7c 392087 tzdata_2019c.orig.tar.gz
> cd31deaeee229d45e4f4b973441189e7619ef81679359e9c8b47b2a87aaf6a07 833 tzdata_2019c.orig.tar.gz.asc
> fa8071037767a7dfa054c26621c5079809ee038eddb32a58814faf3541d52d5a 104932 tzdata_2019c-0+deb10u1.debian.tar.xz
...
As this example lacks the "Files:" entry, it is not mirrored by "apt-mirror".
The real fix is to try "Checksums-Sha256", "Checksum-Sha1" and "Files:" in that
order and use the first existing one.
---
apt-mirror | 76 ++++++++++++++++++++++++++++++++++--------------------
1 file changed, 48 insertions(+), 28 deletions(-)
diff --git a/apt-mirror b/apt-mirror
index 16c2118..5d81fcf 100755
--- a/apt-mirror
+++ b/apt-mirror
@@ -720,11 +720,21 @@ foreach ( keys %urls_to_download )
%urls_to_download = ();
-open FILES_ALL, ">" . get_variable("var_path") . "/ALL" or die("apt-mirror: can't write to intermediate file (ALL)");
-open FILES_NEW, ">" . get_variable("var_path") . "/NEW" or die("apt-mirror: can't write to intermediate file (NEW)");
-open FILES_MD5, ">" . get_variable("var_path") . "/MD5" or die("apt-mirror: can't write to intermediate file (MD5)");
-open FILES_SHA1, ">" . get_variable("var_path") . "/SHA1" or die("apt-mirror: can't write to intermediate file (SHA1)");
-open FILES_SHA256, ">" . get_variable("var_path") . "/SHA256" or die("apt-mirror: can't write to intermediate file (SHA256)");
+open my $FILES_ALL, ">" . get_variable("var_path") . "/ALL" or die("apt-mirror: can't write to intermediate file (ALL)");
+open my $FILES_NEW, ">" . get_variable("var_path") . "/NEW" or die("apt-mirror: can't write to intermediate file (NEW)");
+open my $FILES_MD5, ">" . get_variable("var_path") . "/MD5" or die("apt-mirror: can't write to intermediate file (MD5)");
+open my $FILES_SHA1, ">" . get_variable("var_path") . "/SHA1" or die("apt-mirror: can't write to intermediate file (SHA1)");
+open my $FILES_SHA256, ">" . get_variable("var_path") . "/SHA256" or die("apt-mirror: can't write to intermediate file (SHA256)");
+my %hashes_bin = (
+ "MD5sum:" => $FILES_MD5,
+ "SHA1:" => $FILES_SHA1,
+ "SHA256:" => $FILES_SHA256,
+);
+my %hashes_dsc = (
+ "Files:" => $FILES_MD5,
+ "Checksums-Sha1:" => $FILES_SHA1,
+ "Checksums-Sha256:" => $FILES_SHA256,
+);
my %stat_cache = ();
@@ -817,7 +827,6 @@ sub process_index
chomp $package;
my ( undef, %lines ) = split( /^([\w\-]+:)/m, $package );
- $lines{"Directory:"} = "" unless defined $lines{"Directory:"};
chomp(%lines);
remove_spaces( \%lines );
@@ -825,34 +834,45 @@ sub process_index
{ # Packages index
my $rel = remove_double_slashes( $path . "/" . $lines{"Filename:"} );
$skipclean{ $rel } = 1;
- print FILES_ALL $rel . "\n";
- print FILES_MD5 $lines{"MD5sum:"} . " " . $rel . "\n" if defined $lines{"MD5sum:"};
- print FILES_SHA1 $lines{"SHA1:"} . " " . $rel . "\n" if defined $lines{"SHA1:"};
- print FILES_SHA256 $lines{"SHA256:"} . " " . $rel . "\n" if defined $lines{"SHA256:"};
+ print $FILES_ALL $rel . "\n";
+ for my $alg ( keys %hashes_bin )
+ {
+ print { $hashes_bin{$alg} } $lines{$alg} . " " . $rel . "\n" if defined $lines{$alg};
+ }
if ( need_update( $mirror . "/" . $lines{"Filename:"}, $lines{"Size:"} ) )
{
- print FILES_NEW $rel. "\n";
+ print $FILES_NEW $rel. "\n";
add_url_to_download( $uri . "/" . $lines{"Filename:"}, $lines{"Size:"} );
}
}
- else
+ elsif ( exists $lines{"Directory:"} )
{ # Sources index
- foreach ( split( /\n/, $lines{"Files:"} ) )
+ foreach my $alg ( keys %hashes_dsc )
{
- next if $_ eq '';
- my @file = split;
- die("apt-mirror: invalid Sources format") if @file != 3;
- my $rel = remove_double_slashes( $path . "/" . $lines{"Directory:"} . "/" . $file[2] );
- $skipclean{ $rel } = 1;
- print FILES_ALL $rel . "\n";
- print FILES_MD5 $file[0] . " " . $rel . "\n";
- if ( need_update( $mirror . "/" . $lines{"Directory:"} . "/" . $file[2], $file[1] ) )
+ next unless exists $lines{$alg};
+ my $first = 0;
+ foreach ( split( /\n/, $lines{$alg} ) )
{
- print FILES_NEW $rel. "\n";
- add_url_to_download( $uri . "/" . $lines{"Directory:"} . "/" . $file[2], $file[1] );
+ next if $_ eq '';
+ my ( $hash, $size, $name ) = split;
+ my $rel = remove_double_slashes( $path . "/" . $lines{"Directory:"} . "/" . $name );
+ $skipclean{ $rel } = 1;
+ print { $hashes_dsc{$alg} } $hash . " " . $rel . "\n";
+ next unless $first;
+ $first = 1;
+ print $FILES_ALL $rel . "\n";
+ if ( need_update( $mirror . "/" . $lines{"Directory:"} . "/" . $name, $size ) )
+ {
+ print $FILES_NEW $rel. "\n";
+ add_url_to_download( $uri . "/" . $lines{"Directory:"} . "/" . $name, $size );
+ }
}
}
}
+ else
+ {
+ die("apt-mirror: Unknown section: $package");
+ }
}
close STREAM;
@@ -900,11 +920,11 @@ clear_stat_cache();
print "]\n\n";
-close FILES_ALL;
-close FILES_NEW;
-close FILES_MD5;
-close FILES_SHA1;
-close FILES_SHA256;
+close $FILES_ALL;
+close $FILES_NEW;
+close $FILES_MD5;
+close $FILES_SHA1;
+close $FILES_SHA256;
######################################################################################
## Main download
--
2.20.1
Reply to: