[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#685787: devscripts: Enabling uscan to simply remove files from upstream source



Package: devscripts
Version: 2.10.69+squeeze2
Severity: wishlist
Tags: patch

Hi,

in a (bit longish) thread on debian-devel@l.d.o[1] there was some
discussion about enabling uscan to remove files from upstream archives
according to some information given in some control file.  There was no
real consensus about what control file to use.  The implementation below
is based on using debian/copyright but is easy to switch to other files
in case some other consensus might be reached.

The attached patch does the following:

 1. If (and only if) the debian/copyright file is

     Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

    and if it contains a non-empty field Files-Excluded containing a
    space separated list of globs (as used by find and for specifying
    file lists in machine readable debian/control files). The deletion
    process will loop over every expression and is using the find
    command to delete the according globs.

 2. If files matching are contained in the source tarball this will
    be repackaged except if the option --no-exclusion is given at
    uscan command line or if USCAN_NO_EXCLUSION is set in
    /etc/devscripts.conf or ~/.devscripts.  The removal is implemented
    for all tar compression methods as well as for zip archives (which
    are unpackaged using unzip).  This means if the conditions for
    file exclusion as given above are fullfilled the patch below
    works similar as --repack.

 3. If the tarball did not contained any of the globs in
    debian/copyright::Files-Excluded it will be left untouched.

 4. In case something was removed the version string will be appended by
    '+dfsg' to express the fact that the content of the original source
    was changed.  Note: There was no real consensus whether to use this
    suffix or rather '~dfsg'.  This could be solved by some additional
    configuration option that could be added later.  For some moment I
    also had the idea to obtain the suffix which is "wanted" by the
    maintainer either from debian/watch:dversionmangle or
    debian/changelog but I droped this idea because I did not found
    a reliable method to make a safe guess.

 5. Sometimes upstream tarballs are dirty and unpack a load of files
    into the current directory.  The patch tries to behave reasonable
    and checks whether it could move those files into a dir named
      $pkg-$newversion
    (in case no such file or directory just exists in such a dirty
    tarball).  Also some non-dirty but quite generically named
    directories (like "source") are renamed to "$pkg-$newversion".

I have tested the code with five different packages in Debian Med
repository which show different problematic directory structures and
also different compression methods.  The according copyright files
including Files-Excluded are commited to the following locations:

  Vcs-Svn: svn://svn.debian.org/debian-med/trunk/packages/ampliconnoise/trunk/
  Vcs-Svn: svn://svn.debian.org/debian-med/trunk/packages/conquest-dicom-server/trunk/
  Vcs-Svn: svn://svn.debian.org/debian-med/trunk/packages/imagej/trunk/
  Vcs-Svn: svn://svn.debian.org/debian-med/trunk/packages/saint/trunk/
  Vcs-Svn: svn://svn.debian.org/debian-med/trunk/packages/tm-align/trunk/

When applying the patch and using `uscan --verbose --force-download`
I get the actual resulting orig.tar.gz as I want it to be.

Remark:
Regarding the implementation there was some uncertainity about the
actual Perl module to use.  In the patch below script I decided to
stick to Dpkg::Control and left the code for Parse::DebControl as a
comment which could pretty easily could replace the other parser.

Please consider applying this patch after possibly further discussion on
debian-devel@l.d.o.

Kind regards

      Andreas.

PS: Some more skilled Perl programmer might see some room for enhancing
    the code - just be warned that Perl is not my native language.

[1] http://lists.debian.org/debian-devel/2012/08/msg00380.html

-- Package-specific info:

--- /etc/devscripts.conf ---

--- ~/.devscripts ---
Not present

-- System Information:
Debian Release: 6.0.5
Architecture: i386 (i686)

Kernel: Linux 2.6.36-xenU-4814-i386 (SMP w/1 CPU core)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages devscripts depends on:
ii  dpkg-dev               1.15.8.12         Debian package development tools
ii  libc6                  2.11.3-3          Embedded GNU C Library: Shared lib
ii  perl                   5.10.1-17squeeze3 Larry Wall's Practical Extraction 

Versions of packages devscripts recommends:
pn  at                 <none>                (no description available)
ii  bsd-mailx [mailx]  8.1.2-0.20100314cvs-1 simple mail user agent
ii  curl               7.21.0-2.1+squeeze2   Get a file from an HTTP, HTTPS or 
ii  dctrl-tools        2.14.5                Command-line tools to process Debi
pn  debian-keyring     <none>                (no description available)
pn  debian-maintainers <none>                (no description available)
ii  dput               0.9.6.1+squeeze1      Debian package upload tool
pn  equivs             <none>                (no description available)
ii  fakeroot           1.14.4-1              Gives a fake root environment
ii  git [git-core]     1:1.7.2.5-3           fast, scalable, distributed revisi
ii  git-core           1:1.7.2.5-3           fast, scalable, distributed revisi
ii  gnupg              1.4.10-4              GNU privacy guard - a free PGP rep
pn  libauthen-sasl-per <none>                (no description available)
pn  libcrypt-ssleay-pe <none>                (no description available)
pn  libjson-perl       <none>                (no description available)
pn  libparse-debcontro <none>                (no description available)
pn  libsoap-lite-perl  <none>                (no description available)
pn  libterm-size-perl  <none>                (no description available)
ii  libtimedate-perl   1.2000-1              collection of modules to manipulat
ii  liburi-perl        1.54-2                module to manipulate and access UR
ii  libwww-perl        5.836-1               Perl HTTP/WWW client/server librar
pn  libyaml-syck-perl  <none>                (no description available)
pn  lintian            <none>                (no description available)
ii  lsb-release        3.2-23.2squeeze1      Linux Standard Base version report
ii  lynx-cur [www-brow 2.8.8dev.5-1          Text-mode WWW Browser with NLS sup
ii  lzma               4.43-14               Compression method of 7z format in
ii  mailx              1:20071201-3          Transitional package for mailx ren
ii  man-db             2.5.7-8               on-line manual pager
ii  openssh-client [ss 1:5.5p1-6+squeeze2    secure shell (SSH) client, for sec
ii  patch              2.6-2                 Apply a diff file to an original
ii  patchutils         0.3.1-2               Utilities to work with patches
ii  sensible-utils     0.0.4                 Utilities for sensible alternative
pn  strace             <none>                (no description available)
ii  unzip              6.0-4                 De-archiver for .zip files
ii  w3m [www-browser]  0.5.2-9               WWW browsable pager with excellent
pn  wdiff              <none>                (no description available)
ii  wget               1.12-2.1              retrieves files from the web
ii  xz-utils           5.0.0-2               XZ-format compression utilities

Versions of packages devscripts suggests:
ii  build-essential        11.5              Informational list of build-essent
pn  cvs-buildpackage       <none>            (no description available)
pn  devscripts-el          <none>            (no description available)
pn  gnuplot                <none>            (no description available)
pn  libfile-desktopentry-p <none>            (no description available)
pn  libnet-smtp-ssl-perl   <none>            (no description available)
ii  mutt                   1.5.20-9+squeeze2 text-based mailreader supporting M
pn  svn-buildpackage       <none>            (no description available)
ii  w3m                    0.5.2-9           WWW browsable pager with excellent

-- no debconf information
diff --git a/scripts/uscan.1 b/scripts/uscan.1
index 86b3078..4f53df1 100644
--- a/scripts/uscan.1
+++ b/scripts/uscan.1
@@ -428,6 +428,10 @@ Give verbose output.
 .B \-\-no\-verbose
 Don't give verbose output.  (This is the default behaviour.)
 .TP
+.B \-\-no\-exclusion
+Do not automatically exclude files mentioned in
+\fIdebian/copyright\fR field \fBFiles-Excluded\fR
+.TP
 .B \-\-debug
 Dump the downloaded web pages to stdout for debugging your watch file.
 .TP
@@ -501,6 +505,10 @@ equivalent to the \fB\-\-destdir\fR option.
 If this is set to \fIyes\fR, then after having downloaded a bzip tar,
 lzma tar, xz tar, or zip archive, \fBuscan\fR will repack it to a gzip tar.
 This is equivalent to the \fB\-\-repack\fR option.
+.B USCAN_NO_EXCLUSION
+If this is set to \fIyes\fR, files mentioned in the field \fBFiles-Excluded\fR
+of \fIdebian/copyright\fR will be ignored and no exclusion of files will be
+tried.  This is equivalent to the \fB\-\-no-exclusion\fR option.
 .SH "EXIT STATUS"
 The exit status gives some indication of whether a newer version was
 found or not; one is advised to read the output to determine exactly
diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index 8723fb4..e118142 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -45,6 +45,11 @@ BEGIN {
 	}
     }
 }
+# Dpkg::Control::Hash prefered by James McCoy (who did the last three uscan.pl edits using a debian.org e-mail address)
+use Dpkg::Control::Hash;
+# Parse::DebControl suggested by Jonas Smedegaard
+# use Parse::DebControl;
+
 my $CURRENT_WATCHFILE_VERSION = 3;
 
 my $progname = basename($0);
@@ -70,6 +75,7 @@ sub dehs_die ($);
 sub dehs_output ();
 sub quoted_regex_replace ($);
 sub safe_replace ($$);
+sub get_main_source_dir($$$);
 
 sub usage {
     print <<"EOF";
@@ -136,6 +142,8 @@ Options:
     --no-conf, --noconf
                    Don\'t read devscripts config files;
                    must be the first option given
+    --no-exclusion no automatic exclusion of files mentioned in
+                   debian/copyright field Files-Excluded
     --help         Show this message
     --version      Show version information
 
@@ -178,6 +186,7 @@ my $dehs_start_output = 0;
 my $pkg_report_header = '';
 my $timeout = 20;
 my $user_agent_string = 'Debian uscan ###VERSION###';
+my $no_exclusion = 0;
 
 if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
     $modified_conf_msg = "  (no configuration files read)";
@@ -194,6 +203,7 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
 		       'USCAN_DEHS_OUTPUT' => 'no',
 		       'USCAN_USER_AGENT' => '',
 		       'USCAN_REPACK' => 'no',
+		       'USCAN_NO_EXCLUSION' => 'no',
 		       'DEVSCRIPTS_CHECK_DIRNAME_LEVEL' => 1,
 		       'DEVSCRIPTS_CHECK_DIRNAME_REGEX' => 'PACKAGE(-.+)?',
 		       );
@@ -231,6 +241,8 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
 	or $config_vars{'USCAN_DEHS_OUTPUT'}='no';
     $config_vars{'USCAN_REPACK'} =~ /^(yes|no)$/
 	or $config_vars{'USCAN_REPACK'}='no';
+    $config_vars{'USCAN_NO_EXCLUSION'} =~ /^(yes|no)$/
+	or $config_vars{'USCAN_NO_EXCLUSION'}='no';
     $config_vars{'DEVSCRIPTS_CHECK_DIRNAME_LEVEL'} =~ /^[012]$/
 	or $config_vars{'DEVSCRIPTS_CHECK_DIRNAME_LEVEL'}=1;
 
@@ -261,7 +273,7 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
 # Now read the command line arguments
 my $debug = 0;
 my ($opt_h, $opt_v, $opt_destdir, $opt_download, $opt_force_download,
-    $opt_report, $opt_passive, $opt_symlink, $opt_repack);
+    $opt_report, $opt_passive, $opt_symlink, $opt_repack, $opt_no_exclusion);
 my ($opt_verbose, $opt_level, $opt_regex, $opt_noconf);
 my ($opt_package, $opt_uversion, $opt_watchfile, $opt_dehs, $opt_timeout);
 my $opt_download_version;
@@ -293,6 +305,7 @@ GetOptions("help" => \$opt_h,
 	   "useragent=s" => \$opt_user_agent,
 	   "noconf" => \$opt_noconf,
 	   "no-conf" => \$opt_noconf,
+	   "no-exclusion" => \$opt_no_exclusion,
 	   "download-current-version" => \$opt_download_current_version,
 	   )
     or die "Usage: $progname [options] [directories]\nRun $progname --help for more details\n";
@@ -316,6 +329,7 @@ $timeout = 20 unless defined $timeout and $timeout > 0;
 $symlink = $opt_symlink if defined $opt_symlink;
 $verbose = $opt_verbose if defined $opt_verbose;
 $dehs = $opt_dehs if defined $opt_dehs;
+$no_exclusion = $opt_no_exclusion if defined $opt_no_exclusion;
 $user_agent_string = $opt_user_agent if defined $opt_user_agent;
 $download_version = $opt_download_version if defined $opt_download_version;
 if ($dehs) {
@@ -1429,6 +1443,56 @@ EOF
 	}
     }
 
+    if ( !$no_exclusion ) {
+        my $data = Dpkg::Control::Hash->new();
+        $data->load('debian/copyright');
+        # my $parser = new Parse::DebControl(1);
+        # my $data = $parser->parse_file('debian/copyright', {discardCase=>1,singleBlock=>1,});
+        my $okformat = qr'http://www.debian.org/doc/packaging-manuals/copyright-format/1.0';
+        if ($data->{'format'} =~ m{^$okformat/?$} and $data->{'files-excluded'} ) {
+            my $tempdir = tempdir ( "uscanXXXX", TMPDIR => 1, CLEANUP => 1 );
+            my $globpattern = "*";
+            my $hidden = ".[!.]*";
+            if (defined glob("$tempdir/$hidden")) {
+                $globpattern .= " $hidden";
+            }
+            my $absdestdir = abs_path($destdir);
+            unless ( system("cd $tempdir; tar -xaf \"$absdestdir/$newfile_base\" 2>/dev/null") == 0 ) {
+                print "-- $newfile_base is no tarball.  Try unzip.\n" if $verbose;
+                # try unzip if tar fails - we do want to do something sensible even if no --repack was specified
+                system('command -v unzip >/dev/null 2>&1') >> 8 == 0
+                   or die("unzip binary not found. This would serve as fallback because tar just failed.\n");
+                system('unzip', '-q', '-a', '-d', $tempdir, "$destdir/$newfile_base") == 0
+                   or die("Repacking from zip to tar.gz failed (could not unzip)\n");
+            }
+            my $main_source_dir = get_main_source_dir($tempdir, $pkg, $newversion);
+            unless ( -d $main_source_dir ) {
+                print STDERR "Error: $main_source_dir is no directory";
+            }
+            my $nfiles_before = `find $main_source_dir | wc -l`;
+            foreach (grep {/\//} split /\s+/, $data->{"files-excluded"}) {
+                # delete trailing '/' because otherwise find -path will fail
+                s?/+$?? ;
+                # use rm -rf to enable deleting non-empty directories
+                `find $main_source_dir -path "$main_source_dir/$_" | xargs rm -rf`;
+            };
+            foreach (grep {/^[^\/]+$/} split /\s+/, $data->{"files-excluded"}) {
+                `find $main_source_dir -type f -name $_ -delete`;
+            };
+            my $nfiles_after = `find $main_source_dir | wc -l`;
+            if ( $nfiles_before == $nfiles_after ) {
+                print "-- Source tree remains identical - no need for repacking.\n" if $verbose;
+            } else {
+                my $excludefuffix = '+dfsg' ;
+                my $suffix = 'gz' ;
+                my $newfile_base_dfsg = "${pkg}_${newversion}${excludefuffix}.orig.tar.$suffix" ;
+                system("cd $tempdir; GZIP='-n -9' tar --owner=root --group=root --mode=a+rX -czf \"$absdestdir/$newfile_base_dfsg\" $globpattern") == 0
+                   or die("Excluding files failed (could not create tarball)\n");
+                $symlink = 'no' # prevent symlinking or renaming
+            }
+        }
+    }
+
     my @renames = (
 	[qr/\.(tar\.gz|tgz)$/, 'gz'],
 	[qr/\.(tar\.bz2|tbz2?)$/, 'bz2'],
@@ -2005,3 +2069,50 @@ sub safe_replace($$) {
 	return 1;
     }
 }
+
+sub get_main_source_dir($$$) {
+    my ($tempdir, $pkg, $newversion) = @_;
+    my $fcount = 0;
+    my $main_source_dir = '' ;
+    my $any_dir = '' ;
+    opendir DIR, $tempdir or die "opendir $tempdir: $!";
+    my @files = readdir DIR ;
+    closedir DIR ;
+    foreach my $file (@files) {
+	unless ($file =~ /^\.\.?/) {
+            $fcount++;
+	    if ( -d $tempdir.'/'.$file ) {
+                $any_dir = $tempdir . '/' . $file ;
+                $main_source_dir = $any_dir if ( $file =~ /^$pkg\w*$newversion$/i ) ;
+            }
+        }
+    }
+    if ( $fcount == 1 and $main_source_dir ) {
+        return $main_source_dir ;
+    }
+    if ( $fcount == 1 and $any_dir ) {
+        # Unusual base dir in tarball - should be rather something like ${pkg}-${newversion}
+        $main_source_dir = $tempdir . '/' . $pkg . '-' . $newversion ;
+        move($any_dir, $main_source_dir) or die("Unable to move $any_dir directory $main_source_dir\n");
+        return $main_source_dir ;
+    }
+    print "-- Dirty tarball found.\n" if $verbose;
+    if ( $main_source_dir ) { # if tarball is dirty but does contain a $pkg-$newversion dir we will not undirty but leave it as is
+        print "-- No idea how to create proper tarball structure - leaving as is.\n" if $verbose;
+    	return $tempdir;
+    }
+    print "-- Move files to subdirectory $pkg-$newversion.\n" if $verbose;
+    $main_source_dir = $tempdir . '/' . $pkg . '-' . $newversion ;
+    mkdir($main_source_dir) or die("Unable to create temporary source directory $main_source_dir\n");
+    foreach my $file (@files) {
+	unless ($file =~ /^\.\.?/) {
+            # move("${tempdir}/$file", $main_source_dir) or die("Unable to move ${tempdir}/$file directory $main_source_dir\n");
+            unless ( move("${tempdir}/$file", $main_source_dir) ) {
+        	# HELP: why can't perl move not move directories????
+                print "Perl move seems to be not able to ` move(\"${tempdir}/$file\", $main_source_dir) ` ... trying system mv\n" if $debug;
+                system( "mv ${tempdir}/$file $main_source_dir" ) ;
+            }
+        }
+    }
+    return $main_source_dir;
+}

Reply to: