[PATCH] Adding coordination pages to the web site po-debconf l10n statistics

To: debian-www@lists.debian.org, debian-i18n@lists.debian.org, debian-l10n-devel@lists.alioth.debian.org
Subject: [PATCH] Adding coordination pages to the web site po-debconf l10n statistics
From: Nicolas François <nicolas.francois@centraliens.net>
Date: Wed, 13 Aug 2008 07:21:58 +0200
Message-id: <20080813052158.GA29099@nekral.nekral.homelinux.net>
Mail-followup-to: Nicolas François <nicolas.francois@centraliens.net>, debian-www@lists.debian.org, debian-i18n@lists.debian.org, debian-l10n-devel@lists.alioth.debian.org

Hello,

Here is a patch to the scripts (and data) used for the generation of the
l10n statistics on the web site.

It adds information about the progress of the translation done on the
mailing lists when this information is available.

You can have a example of the generated pages on:
http://i18n.debian.net/~nekral/robot_test/international/l10n/po-debconf/ro.en.html
http://i18n.debian.net/~nekral/robot_test/international/l10n/po-debconf/ru.en.html

This format will help the teams by only pointing to the files that really
need work (i.e. the work is not already done by another translator or just
waiting in the BTS)

Could anybody from -www review and apply it?


* english/international/l10n/dtc.def
  - new strings for translation
  - new table definitions
* english/international/l10n/po-debconf/todo.inc
  - Use a podebconf-table or a podebconf-table-status table depending on
    the language
* english/international/l10n/scripts/fix-files.sh
  - Support for table with or without status
  - Parse SECTIONS recursively
* english/international/l10n/scripts/gen-files.pl
  - Use the coordination status databases to provide information about
    ongoing work on the mailing lists
  - Move the PO files with errors on the top of the "DONE" list.

In addition to the patch, the status databases should be downloaded in
often from http://i18n.debian.net/debian-l10n/status/, and linked to
english/international/l10n/data/

Thanks in advance,
-- 
Nekral

Index: english/international/l10n/dtc.def
===================================================================
RCS file: /cvs/webwml/webwml/english/international/l10n/dtc.def,v
retrieving revision 1.47
diff -u -r1.47 dtc.def
--- english/international/l10n/dtc.def	17 Mar 2006 20:44:32 -0000	1.47
+++ english/international/l10n/dtc.def	13 Aug 2008 05:00:08 -0000
@@ -27,6 +27,22 @@
   <gettext domain="l10n">Team</gettext>
 </define-tag>
 
+<define-tag date whitespace=delete>
+  <gettext domain="l10n">Date</gettext>
+</define-tag>
+
+<define-tag status whitespace=delete>
+  <gettext domain="l10n">Status</gettext>
+</define-tag>
+
+<define-tag strings whitespace=delete>
+  <gettext domain="l10n">Strings</gettext>
+</define-tag>
+
+<define-tag bug whitespace=delete>
+  <gettext domain="l10n">Bug</gettext>
+</define-tag>
+
 
 <define-tag langcountryoutput whitespace=delete>
   <preserve lang country />
@@ -107,6 +123,21 @@
 </table>
 </define-tag>
 
+<define-tag podebconf-table-status endtag=required>
+<table border=1>
+<tr>
+  <th><package></th>
+  <th><score></th>
+  <th><file></th>
+  <th><translator></th>
+  <th><status></th>
+  <th><translator></th>
+  <th><date></th>
+  <th><bug></th>
+</tr>
+%body
+</table>
+</define-tag>
 
 #use wml::debian::language_names
 
Index: english/international/l10n/po-debconf/todo.inc
===================================================================
RCS file: /cvs/webwml/webwml/english/international/l10n/po-debconf/todo.inc,v
retrieving revision 1.1
diff -u -r1.1 todo.inc
--- english/international/l10n/po-debconf/todo.inc	8 Sep 2003 12:12:55 -0000	1.1
+++ english/international/l10n/po-debconf/todo.inc	13 Aug 2008 05:00:08 -0000
@@ -2,21 +2,51 @@
 
 <!-- BEGIN SECTION -->
 <h3><pkg-section main></h3>
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-main-nostatus -->
 <podebconf-table>
 <!-- DO NOT REMOVE THIS LINE: todo-main -->
 </podebconf-table>
 <!-- END SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-main-both -->
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-main-status -->
+<podebconf-table-status>
+<!-- DO NOT REMOVE THIS LINE: todo-main -->
+</podebconf-table-status>
+<!-- END SECTION -->
+<!-- END SECTION -->
 
 <!-- BEGIN SECTION -->
 <h3><pkg-section contrib></h3>
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-contrib-nostatus -->
 <podebconf-table>
 <!-- DO NOT REMOVE THIS LINE: todo-contrib -->
 </podebconf-table>
 <!-- END SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-contrib-both -->
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-contrib-status -->
+<podebconf-table-status>
+<!-- DO NOT REMOVE THIS LINE: todo-contrib -->
+</podebconf-table-status>
+<!-- END SECTION -->
+<!-- END SECTION -->
 
 <!-- BEGIN SECTION -->
 <h3><pkg-section non-free></h3>
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-non-free-nostatus -->
 <podebconf-table>
 <!-- DO NOT REMOVE THIS LINE: todo-non-free -->
 </podebconf-table>
 <!-- END SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-non-free-both -->
+<!-- BEGIN SECTION -->
+<!-- DO NOT REMOVE THIS LINE: todo-non-free-status -->
+<podebconf-table-status>
+<!-- DO NOT REMOVE THIS LINE: todo-non-free -->
+</podebconf-table-status>
+<!-- END SECTION -->
+<!-- END SECTION -->
Index: english/international/l10n/scripts/fix-files.sh
===================================================================
RCS file: /cvs/webwml/webwml/english/international/l10n/scripts/fix-files.sh,v
retrieving revision 1.6
diff -u -r1.6 fix-files.sh
--- english/international/l10n/scripts/fix-files.sh	8 Mar 2004 22:14:03 -0000	1.6
+++ english/international/l10n/scripts/fix-files.sh	13 Aug 2008 05:00:08 -0000
@@ -10,6 +10,20 @@
 do
     for dist in main contrib non-free
     do
+        echo 'DELETE-ME' > $dir/${dist}-$code.todo-nostatus
+        echo 'DELETE-ME' > $dir/${dist}-$code.todo-status
+        echo 'DELETE-ME' > $dir/${dist}-$code.todo-both
+        if [ -f $dir/${dist}-$code.todo ]
+        then
+            rm -f $dir/${dist}-$code.todo-both
+            if [ -f $dir/../../data/status.$code ]
+            then
+                rm -f $dir/${dist}-$code.todo-status
+            else
+                rm -f $dir/${dist}-$code.todo-nostatus
+            fi
+        fi
+
         for type in todo ok exc
         do
             file=$dir/${dist}-$code.$type
@@ -22,9 +36,18 @@
     sed -e "s/@tmpl_lang@/$code/" \
         -e "s/@tmpl_lang_stats@/$stat/" \
         -e "s/href=\"tmpl\\./href=\"$code./" \
-        -e "/LINE: todo-main/r     $dir/main-$code.todo" \
-        -e "/LINE: todo-contrib/r  $dir/contrib-$code.todo" \
-        -e "/LINE: todo-non-free/r $dir/non-free-$code.todo" \
+        -e "/LINE: todo-main /r     $dir/main-$code.todo" \
+        -e "/LINE: todo-main-nostatus/r $dir/main-$code.todo-nostatus" \
+        -e "/LINE: todo-main-status/r   $dir/main-$code.todo-status" \
+        -e "/LINE: todo-main-both/r     $dir/main-$code.todo-both" \
+        -e "/LINE: todo-contrib /r  $dir/contrib-$code.todo" \
+        -e "/LINE: todo-contrib-nostatus/r  $dir/contrib-$code.todo-nostatus" \
+        -e "/LINE: todo-contrib-status/r    $dir/contrib-$code.todo-status" \
+        -e "/LINE: todo-contrib-both/r      $dir/contrib-$code.todo-both" \
+        -e "/LINE: todo-non-free /r $dir/non-free-$code.todo" \
+        -e "/LINE: todo-non-free-nostatus/r $dir/non-free-$code.todo-nostatus" \
+        -e "/LINE: todo-non-free-status/r   $dir/non-free-$code.todo-status" \
+        -e "/LINE: todo-non-free-both/r     $dir/non-free-$code.todo-both" \
         -e "/LINE: ok-main/r     $dir/main-$code.ok" \
         -e "/LINE: ok-contrib/r  $dir/contrib-$code.ok" \
         -e "/LINE: ok-non-free/r $dir/non-free-$code.ok" \
@@ -33,7 +56,32 @@
         -e "/LINE: exc-non-free/r $dir/non-free-$code.exc" \
             tmpl.$lang.tmpl |\
         sed -e '/<!-- DO NOT REMOVE THIS LINE/d' |\
-        perl -e 'while(<>) {if (m/BEGIN SECTION/) {$body=$_; $found=0;do {$_ = <>; $found=1 if m/DELETE-ME/; $body.=$_} until m/END SECTION/; print $body unless $found} else {print}}' \
+        perl -e '
+my @print = ();
+my @block = ();
+my $print = 1;
+my $block = "";
+while (<>) {
+	if (m/BEGIN SECTION/){
+		$block .= $_;
+		push @print, $print;
+		push @block, $block;
+		$block = "";
+		$print = 1;
+	} elsif (m/END SECTION/) {
+		my $tmp = "";
+		$tmp = $block if $print;
+		$print = pop @print;
+		$block = pop @block;
+		$block .= $tmp;
+		$block .= $_;
+	} elsif (m/DELETE-ME/) {
+		$print = 0;
+	} else {
+		$block .= $_;
+	}
+}
+print $block;' \
                 > $code.$lang.html
 done
 
Index: english/international/l10n/scripts/gen-files.pl
===================================================================
RCS file: /cvs/webwml/webwml/english/international/l10n/scripts/gen-files.pl,v
retrieving revision 1.62
diff -u -r1.62 gen-files.pl
--- english/international/l10n/scripts/gen-files.pl	22 Oct 2006 17:09:10 -0000	1.62
+++ english/international/l10n/scripts/gen-files.pl	13 Aug 2008 05:00:08 -0000
@@ -10,6 +10,34 @@
 
 use vars qw($opt_h $opt_d $opt_l $opt_s $opt_D $opt_P $opt_T $opt_L $opt_M);
 
+my %LanguageList = (
+	AR    => 'arabic',
+	CA    => 'catalan',
+	DE    => 'german',
+# Used by the Smith project, not for translations
+#	EN    => 'english',
+	ES    => 'spanish',
+	FR    => 'french',
+# Not supported yet by the robot. Not all messages are sent ot the list
+#	NL    => 'dutch',
+	PT_BR => 'portuguese',
+	RO    => 'romanian',
+# Has not used pseudo-urls recently
+#	TR    => 'turkish',
+);
+my %Status = (
+	todo => 0,
+	maj  => 1,
+	itt  => 2,
+	rfr  => 3,
+	itr  => 4,
+	lcfc => 5,
+	bts  => 6,
+	fix  => 7,
+	done => 8,
+	hold => 9,
+	);
+
 sub usage {
         print "Usage:  gen-files.pl [--dist=DIST] [--l10ndir=DIR] [--sort=FILE] [--po] [--templates] [--podebconf] [--langs] [--po4a]\n";
         exit($_[0]);
@@ -418,6 +446,7 @@
 
         my %done  = ();
         my %todo  = ();
+        my %error  = ();
         my %excl = ();
         my $none = '';
         my $tmpl_errors = {};
@@ -487,8 +516,13 @@
                                 $score{$lang} += $1;
                         }
 		    	if (percent_stat($stat) eq "100%") {
+			  if (defined $tmpl_errors->{$pkg}) {
+                           $error{$lang}  = '' unless defined($error{$lang});
+			   $error{$lang} .= $str;
+			  } else {
                            $done{$lang}  = '' unless defined($done{$lang});
 			   $done{$lang} .= $str;
+			  }
  			} else {
                            $todo{$lang}  = '' unless defined($todo{$lang});
 			   $todo{$lang} .= $str;
@@ -520,6 +554,14 @@
                 close (GEN);
         }
         foreach $lang (@td_langs) {
+                next unless defined $error{uc $lang};
+                open (GEN, ">> $opt_l/templates/gen/$section-$lang.ok")
+                        || die "Unable to write into $opt_l/templates/gen/$section-$lang.ok";
+                print GEN "<tr><td colspan=3>PO files with errors</td></tr>\n";
+                print GEN $error{uc $lang};
+                close (GEN);
+        }
+        foreach $lang (@td_langs) {
                 next unless defined $todo{uc $lang};
                 open (GEN, "> $opt_l/templates/gen/$section-$lang.todo")
                         || die "Unable to write into $opt_l/templates/gen/$section-$lang.todo";
@@ -635,11 +677,21 @@
 
         $total{$section} = 0;
         my %done  = ();
+        my %error = ();
         my %todo  = ();
+        my %excl_pending  = ();
         my %excl  = ();
         my $orig  = '';
 	my $podebconf_errors = {};
 	my $podebconf_errors_by_language = {};
+	# Load the coordination status databases
+        my %status_db    = ();
+        for my $l (qw/ar ca de es fr nl pt_BR ro tr/) {
+                if (-r "$opt_l/data/status.$l") {
+                        $status_db{uc $l} = Debian::L10n::Db->new();
+                        $status_db{uc $l}->read("$opt_l/data/status.$l", 0);
+                }
+        }
         foreach $pkg (sort pkgsort @{$packages}) {
                 next unless $data->has_podebconf($pkg);
 
@@ -722,10 +774,9 @@
                         $link =~ s/:/\%3a/g;
                         $lang = uc($lang) || 'UNKNOWN';
                         $list{$lang} = 1;
+                        my $color = get_color(percent_stat($stat));
 		        my $str = '';
-                        $str .= "<tr style=\"background-color: ".
-                              get_color(percent_stat($stat)).
-		              "\"><td>";
+                        $str .= "<tr style=\"background-color: $color\"><td>";
                         if ($stat =~ m/^(\d+)t(\d+)f(\d+)u$/) {
                                 $score{$lang} += $1;
                         }
@@ -740,12 +791,41 @@
 		        $str .= "</td><td>".show_stat($stat)."</td><td><a href=\"";
                         $str .= ($data->section($pkg) =~ m/non-US/ ? $rootnonus : $root) . "po/$opt_d/";
                         $str .= $data->pooldir($pkg)."/$link.gz\">$pofile</a></td>";
-		        $str .= "<td>$translator</td>".
-                              "</tr>\n";
+		        $str .= "<td>$translator</td>";
 		    	if (percent_stat($stat) eq "100%") {
+			  $str .= "</tr>\n";
+			  if (   defined $podebconf_errors_by_language->{$pkg}->{global}
+			      or defined $podebconf_errors_by_language->{$pkg}->{$lang}) {
+			   $error{$lang}  = '' unless defined($error{$lang});
+			   $error{$lang} .= $str;
+			  } else {
                            $done{$lang}  = '' unless defined($done{$lang});
 			   $done{$lang} .= $str;
+			  }
  			} else {
+			   if (defined $status_db{$lang}) {
+			      my $add = "";
+			      if (    $status_db{$lang}->has_package($pkg)
+			          and $status_db{$lang}->has_status($pkg)) {
+				 foreach my $statusline (@{$status_db{$lang}->status($pkg)}) {
+				    my ($type, $file, $date, $status, $translator, $list, $url, $bug_nb) = @{$statusline};
+				    my $bug_link = (defined $bug_nb) ? "<a href=\"http://bugs.debian.org/$bug_nb\";>$bug_nb</a>" : "";
+				    if ($type eq "podebconf") {
+				       # Only keep the last status (most recent)
+				       # Assume there is only one podebconf file
+				       $date =~ s/\s*\+0000$//;
+				       $list =~ /^(\d\d\d\d)-(\d\d)-(\d\d\d\d\d)$/;
+				       $add = "<a href=\"http://lists.debian.org/debian-l10n-$LanguageList{$lang}/$1/debian-l10n-$LanguageList{$lang}-$1$2/msg$3.html\";>$status</a>";
+				       $add = "<td>$add</td><td>$translator</td><td>$date</td><td>$bug_link</td>";
+				    }
+				 }
+			      }
+			      unless (length $add) {
+				   $add .= "<td></td><td></td><td></td><td></td>";
+			      }
+			      $str .= $add;
+			   }
+			   $str .= "</tr>\n";
                            $todo{$lang}  = '' unless defined($todo{$lang});
 			   $todo{$lang} .= $str;
 			}
@@ -756,26 +836,64 @@
                 foreach $lang (@pd_langs) {
                         my $l = uc($lang) || 'UNKNOWN';
                         next if $list{$l};
-                        $excl{$l}  = '' unless defined($excl{$l});
+			my $str;
 			if (defined $podebconf_errors_by_language->{$pkg}->{global}) {
-			    $excl{$l} .= " (<a href=\"errors-by-pkg#P$pkg\">!</a>)";
+			    $str .= " (<a href=\"errors-by-pkg#P$pkg\">!</a>)";
 			} elsif (defined $podebconf_errors_by_language->{$pkg}->{$lang}) {
-			    $excl{$l} .= " (<a href=\"errors-by-pkg#P$pkg\">!</a>)";
+			    $str .= " (<a href=\"errors-by-pkg#P$pkg\">!</a>)";
 			}
-                        $excl{$l} .= "<a href=\"pot#$pkg\">$pkg</a>, ";
-                }
+                        $str .= "<a href=\"pot#$pkg\">$pkg</a>";
+			my $add = "";
+			if (defined $status_db{$l}) {
+			   if (    $status_db{$l}->has_package($pkg)
+			       and $status_db{$l}->has_status($pkg)) {
+			      foreach my $statusline (@{$status_db{$l}->status($pkg)}) {
+				 my ($type, $file, $date, $status, $translator, $list, $url, $bug_nb) = @{$statusline};
+				 my $bug_link = (defined $bug_nb) ? "<a href=\"http://bugs.debian.org/$bug_nb\";>$bug_nb</a>" : "";
+				 if ($type eq "podebconf") {
+				    # Only keep the last status (most recent)
+				    # Assume there is only one file with a
+				    # podebconf type
+				    $date =~ s/\s*\+0000$//;
+				    $list =~ /^(\d\d\d\d)-(\d\d)-(\d\d\d\d\d)$/;
+				    $add = "<a href=\"http://lists.debian.org/debian-l10n-$LanguageList{$l}/$1/debian-l10n-$LanguageList{$l}-$1$2/msg$3.html\";>$status</a>";
+				    $add = "<td>$add</td><td>$translator</td><td>$date</td><td>$bug_link</td>";
+				 }
+			      }
+			   }
+			   if (length $add) {
+			      $str = "<td>$str</td><td>0\% (0t0f$curtotal)</td><td></td><td></td>".$add;
+			   }
+			}
+			if (length $add) {
+				$excl_pending{$l}  = '' unless defined($excl_pending{$l});
+				$excl_pending{$l} .= "<tr>$str</tr>\n";
+			} else {
+				$excl{$l}  = '' unless defined($excl{$l});
+				$excl{$l} .= $str."&nbsp;($curtotal)";
+				$excl{$l} .= ", ";
+			}
+		}
         }
         foreach $lang (@pd_langs) {
-                next unless defined $todo{uc $lang};
+                next unless defined $todo{uc $lang} or defined $excl_pending{uc $lang};
                 open (GEN, "> $opt_l/po-debconf/gen/$section-$lang.todo")
                         || die "Unable to write into $opt_l/po-debconf/gen/$section-$lang.todo";
-                print GEN $todo{uc $lang};
+                print GEN $todo{uc $lang} if defined $todo{uc $lang};
+                print GEN $excl_pending{uc $lang} if defined $excl_pending{uc $lang};
                 close (GEN);
         }
         foreach $lang (@pd_langs) {
-                next unless defined $done{uc $lang};
+                next unless defined $error{uc $lang};
                 open (GEN, "> $opt_l/po-debconf/gen/$section-$lang.ok")
                         || die "Unable to write into $opt_l/po-debconf/gen/$section-$lang.ok";
+                print GEN $error{uc $lang};
+                close (GEN);
+        }
+        foreach $lang (@pd_langs) {
+                next unless defined $done{uc $lang};
+                open (GEN, ">> $opt_l/po-debconf/gen/$section-$lang.ok")
+                        || die "Unable to write into $opt_l/po-debconf/gen/$section-$lang.ok";
                 print GEN $done{uc $lang};
                 close (GEN);
         }

Reply to:

Follow-Ups:
- Re: [PATCH] Adding coordination pages to the web site po-debconf l10n statistics
  - From: "Leandro Regueiro" <leandro.regueiro@gmail.com>
- Re: [PATCH] Adding coordination pages to the web site po-debconf l10n statistics
  - From: Simon Paillard <simon.paillard@resel.enst-bretagne.fr>

Prev by Date: aptitude: users must enter "Yes" message in each localized words and more, when they are warned to use unofficial repositories
Next by Date: [PATCH] Translation statistics do not include packages with a debian/patches directory
Previous by thread: aptitude: users must enter "Yes" message in each localized words and more, when they are warned to use unofficial repositories
Next by thread: Re: [PATCH] Adding coordination pages to the web site po-debconf l10n statistics
Index(es):
- Date
- Thread