Your message dated Sun, 14 Jun 2015 19:19:07 +0000 with message-id <E1Z4DQy-0005Wt-02@franck.debian.org> and subject line Bug#787760: fixed in debmirror 1:2.18 has caused the Debian Bug report #787760, regarding Add the file download method to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) -- 787760: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787760 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
--- Begin Message ---
- To: submit@bugs.debian.org
- Subject: Add the file download method
- From: "Thaddeus H. Black" <thb@debian.org>
- Date: Thu, 4 Jun 2015 18:40:36 +0000
- Message-id: <[🔎] 20150604184035.GA2772@b-tk.org>
Package: debmirror Version: 1:2.17 Severity: normal Tags: patch Debmirror can download via ftp://, http://, https:// or rsync://, but not via file://. That is, a command like this does not work: debmirror --dist=stable --arch=amd64 --method=file\ --root=$SOURCE_DIRECTORY --no-check-gpg $TARGET_DIRECTORY Why do you care that this does not work? Answer: testing. It can be more convenient for users, and maybe even more convenient for developers, to test debmirror's behavior on localhost than over the network. Besides, most or all programs that support multiple network protocols should probably support a file:// method, for approximately the same reason your network should support a localhost and your system should support a /dev/null. Even if the file:// method is not often used, it should remain available as a null method, so to speak. (I believe that pbuilder has recently added a file:// method, for instance.) Please find attached a suggested patch to implement the change. I have successfully tested the patched binary by using it download jessie 8.0 stable, with source and binary-amd64, about 100 GiB, from one hard-drive partition of my laptop to another. Observe that though rsync is incidentally used in the normal debmirror manner, no rsync *daemon* is required to use the file:// method as the patch implements it. The severity is normal rather than wishlist because applying this patch may make it easier to fix some of debmirror's other, outstanding bugs of normal severity. At the time this bug is reported, debmirror has been orphaned (#768532) six months. Since no one else has maintained this package for the past six months, since I have now spent the time to learn a little about how debmirror internally works, and since I have used debmirror for years and prefer its bugs to be fixed, I had probably better adopt the package. I will file ITA after this report.diff -turN debmirror-2.17/debmirror debmirror-2.17.1/debmirror --- debmirror-2.17/debmirror 2014-07-02 20:54:08.000000000 +0000 +++ debmirror-2.17.1/debmirror 2015-06-03 22:41:55.517583458 +0000 @@ -103,7 +103,8 @@ =item B<--method>=I<method> Specify the method to download files. Currently, supported methods are -B<ftp>, B<http>, B<https>, and B<rsync>. +B<ftp>, B<http>, B<https>, and B<rsync>. The B<file> method is +experimentally supported. =item B<--passive> @@ -766,7 +767,7 @@ } # Backwards compatibility: remote root dir no longer needs prefix -$remoteroot =~ s%^[:/]%%; +$remoteroot =~ s%^[:/]%% unless downloads_via_file(); # Post-process arrays. Allow commas to separate values the user entered. # If the user entered nothing, provide defaults. @@ -802,7 +803,7 @@ # Display configuration. $|=1 if $debug; if ($passwd eq "anonymous@") { - if ($download_method eq "http") { + if (downloads_via_http()) { say("Mirroring to $mirrordir from $download_method://$host/$remoteroot/"); } else { say("Mirroring to $mirrordir from $download_method://$user\@$host/$remoteroot/"); @@ -823,7 +824,7 @@ say("Passive mode on.") if $passive; say("Proxy: $proxy") if $proxy; say("Download at most $max_batch files.") if ($max_batch > 0); -say("Download at most $rsync_batch files per rsync call.") if ($download_method eq "rsync"); +say("Download at most $rsync_batch files per rsync call.") if (downloads_via_rsync()); if ($pre_cleanup) { say("Will clean up before mirroring."); } elsif ($post_cleanup) { @@ -878,7 +879,7 @@ sub init_connection { $_ = $download_method; - /^http$/ && do { + downloads_via_http() && do { $ua = LWP::UserAgent->new(keep_alive => 1); $ua->timeout($timeout); $ua->proxy('http', $ENV{http_proxy}) if $ENV{http_proxy}; @@ -887,7 +888,7 @@ return; }; - /^https$/ && do { + downloads_via_https() && do { $ua = LWP::UserAgent->new(keep_alive => 1, ssl_opts => { verify_hostname => ! $disable_ssl_verification }); $ua->timeout($timeout); @@ -898,7 +899,7 @@ }; - /^ftp$/ && do { + downloads_via_ftp() && do { if ($proxy || $ENV{ftp_proxy}) { $ua = LWP::UserAgent->new; $ua->timeout($timeout); @@ -915,7 +916,15 @@ return; }; - /^rsync$/ && do { + downloads_via_file() && do { + $ua = LWP::UserAgent->new; + $ua->timeout($timeout); + $ua->show_progress($progress); + $host='localhost'; + return; + }; + + downloads_via_rsync() && do { return; }; @@ -926,13 +935,18 @@ # determine remote root for rsync transfers my $rsyncremote; if (length $remoteroot) { - $rsyncremote = "$host\:\:$remoteroot/"; - if ($user ne 'anonymous') { - $rsyncremote = "$user\@$rsyncremote"; + if (downloads_via_file()) { + $rsyncremote = "$remoteroot/"; + } + else { + $rsyncremote = "$host\:\:$remoteroot/"; + if ($user ne 'anonymous') { + $rsyncremote = "$user\@$rsyncremote"; + } } } else { - if ($download_method eq 'rsync') { + if (downloads_via_rsync()) { die "rsync cannot be used with a root of $remoteroot/\n"; } } @@ -1321,8 +1335,7 @@ batch_get(); sub batch_get { - if ($download_method eq 'ftp' || $download_method eq 'http' || - $download_method eq 'https') { + if (uses_LWP()) { my $dirname; my $i=0; foreach my $file (sort keys %files) { @@ -1681,9 +1694,8 @@ $tdir=$tempdir unless $tdir; chdir($tdir) or die "unable to chdir($tdir): $!\n"; - if ($download_method eq 'ftp' || $download_method eq 'http' || - $download_method eq 'https') { - $res=$ftp ? ftp_get($file) : http_get($file); + if (uses_LWP()) { + $res=$ftp ? ftp_get($file) : http_get($file); $res=$res && check_lists($file); if (-f $file && !$res) { say("$file failed checksum verification, removing"); @@ -2733,7 +2745,7 @@ if ($ftp) { $ftp->quit; } my $total_time = time - $start_time; - if ($download_method eq 'rsync' || $bytes_gotten == 0) { + if (downloads_via_rsync() || $bytes_gotten == 0) { say("Download completed in ".$total_time."s."); } else { my $avg_speed = 0; @@ -2839,6 +2851,44 @@ } } +sub downloads_via_http { + local $_ = shift; + defined or $_ = $download_method; + return $_ eq 'http'; +} + +sub downloads_via_https { + local $_ = shift; + defined or $_ = $download_method; + return $_ eq 'https'; +} + +sub downloads_via_http_or_https { + return downloads_via_http(@_) || downloads_via_https(@_); +} + +sub downloads_via_ftp { + local $_ = shift; + defined or $_ = $download_method; + return $_ eq 'ftp'; +} + +sub downloads_via_file { + local $_ = shift; + defined or $_ = $download_method; + return $_ eq 'file'; +} + +sub downloads_via_rsync { + local $_ = shift; + defined or $_ = $download_method; + return $_ eq 'rsync'; +} + +sub uses_LWP { + return !downloads_via_rsync(@_); +} + sub say { print join(' ', @_)."\n" if ($verbose or $progress); }Attachment: signature.asc
Description: Digital signature
--- End Message ---
--- Begin Message ---
- To: 787760-close@bugs.debian.org
- Subject: Bug#787760: fixed in debmirror 1:2.18
- From: thb@debian.org (Thaddeus H. Black)
- Date: Sun, 14 Jun 2015 19:19:07 +0000
- Message-id: <E1Z4DQy-0005Wt-02@franck.debian.org>
Source: debmirror Source-Version: 1:2.18 We believe that the bug you reported is fixed in the latest version of debmirror, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to 787760@bugs.debian.org, and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Thaddeus H. Black <thb@debian.org> (supplier of updated debmirror package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing ftpmaster@ftp-master.debian.org) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 1.8 Date: Sat, 13 Jun 2015 00:00:00 -0000 Source: debmirror Binary: debmirror Architecture: source all Version: 1:2.18 Distribution: unstable Urgency: low Maintainer: Thaddeus H. Black <thb@debian.org> Changed-By: Thaddeus H. Black <thb@debian.org> Description: debmirror - Debian partial mirror script, with ftp and package pool support Closes: 375381 576577 619363 628779 768532 787760 Changes: debmirror (1:2.18) unstable; urgency=low . * New maintainer, on a trial basis. We'll see how it goes. Thanks, Joey, for four years recently filling the gap; and earlier, for giving us the software in the first place. Joerg, Goswin, your work is appreciated (see? it continues to be used). Frans, we still miss you. Closes: #768532 * Somewhat refactored the code that distinguishes between download methods. * Implemented the option --method=file, which -- besides being useful in its own right -- may also make testing and debugging easier. * In the unpacked source, implemented "make -C test", which builds a small, mock Debian archive against which one can test and debug the program. Closes: #787760 * Applied Kees Cook's patch to add a retry cycle for rsync connection failures, implementing the cycle optionally via the new, experimental --retry-rsync-packages option. Closes: #576577 * Per Joey's advice, did not apply patch 619363 regarding the --ignore-missing-release option. Closes: #619363 * Appreciated patch 628779 to work around an out-of-sync parent mirror, but did not apply it. Joey explains why in the bug log. Closes: #628779 * Noted per Goswin's advice that the Debian archive ceased to ship uncompressed Packages files about 2006. Whatever trouble debmirror might once have had with the 2006 transition is ancient history now, so if a bug remains thereto, it probably affects no one and won't be fixed. Closes: #375381 * Did not yet fix bug #652138. However, refactored the code that downloads Contents files, with the intent to make this bug easier later to fix. * In the debmirror script, as a matter of the new maintainer's preference, expanded all leading tabs as spaces. Checksums-Sha1: c05ef8ace16928a7e5c23f68fd0db34ee9f8fbcf 1497 debmirror_2.18.dsc 20206aa6cf3da3e8f29ae5cbedb6545a366c8e78 59877 debmirror_2.18.tar.gz 077072e0774258b34a24b4dedac7c57dec8992f9 51754 debmirror_2.18_all.deb Checksums-Sha256: b3b080cd20d452d80a78110e98a747fe690388a8d547c43496f32d2285d73069 1497 debmirror_2.18.dsc 63b06e3d31832bcd1f08688c716e580805def7ef5b033b8d4b12105e165c79bd 59877 debmirror_2.18.tar.gz f75f67a5fde67e42c6210419625f523bb79be5584bf349b069697a67eebe442b 51754 debmirror_2.18_all.deb Files: a3bef0160e0a63af8c5b1cc7515b4ad6 1497 net extra debmirror_2.18.dsc 94f5e28d702622b96fb96d3ac683634a 59877 net extra debmirror_2.18.tar.gz fccac66efd67346c4740f9e8979d4fa7 51754 net extra debmirror_2.18_all.deb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJVfdFrAAoJEHBjJkMthSK8TqMQAOobJmqHQQ9Xt4D+XqPH0ysU qUbuZyy6n1NQT+rGJ+q4JrT8uC2Q3/eFMnejhNwUHIRq9SZZQdH6J/6CXCzwkA0R 79ATCrC4CRYZt6ijb0B6TLzgFCukb2tgogrhUbx8x4vVUnVvVmr5EE/DEB/k7/N9 9jFttTRnmYBOnIeZIPl5H+z2BcQbO8Uy37EUrDb8f0PZIcY30Zhe33/PFI8ZO8Ac 6Vab5hEWufh5PHo4ez+bFaAZ0o6sdnUkV9eWiqpnodBGK8To1AoAe3e9aER5qTYU flKNl2QpGGKiAL6C3WerigfbCq6cY2OSwuAJCTLiIYUPYFUWj5Td8XuAGs4PVNHQ Vju3+piYehbaqjGVdySFXaiTCT1pOaw7ZE2sy6rLKpEYa+qHEXNJhoh9jpCEIUy3 FR4NIxcTmw1NQ0yI1/KYn7cNkqot/dcNdUEvfcuomopQ8kob/wIsCqffVRmjow9d bq6qpn+GtpiGs+ZXNUC9H0TjlbYYa51LuOV5FXu8lJAHDQ1/aysV4Pik+vOPBWDo 60rAR+Y294nQFBoV/YhJXYUf29XvIAzOh6BhGD77mLkF9mpcMfki4Clr9dBCpRV/ xk2S2AAFZYWBTa+SjoFl6hxrGNuGeEBxmQNrvdi/Pu4+7xja+PVm7ofWpCX6sBt1 Yu4KfQ9S6C+FYGzLNFk9 =Ct2b -----END PGP SIGNATURE-----
--- End Message ---