Jouet de statistiques

To: dlf <debian-l10n-french@lists.debian.org>
Subject: Jouet de statistiques
From: Martin Quinson <mquinson@ens-lyon.fr>
Date: Mon, 15 Mar 2004 02:40:32 -0800
Message-id: <[🔎] 20040315104032.GB26958@galadriel.cri2000.ens-lyon.fr>

Hello,

Denis, est ce que tu peux commiter le script en attachement sous 
webwml/english/international/l10n/scripts/ s'il te plait?
C'est la qu'il va.

Exemples d'usage:
1) Les podebconf restant a faire vis a vis de ce qu'on trouve dans les
   paquets, et dont on a pas parle sur la liste (faut un status.fr comme
   genere par mes scripts d'il y a quelques jours):
./display_txt --status --show=podebconf --todo fr |grep -v 'podebconf('
                    ______________________
 __________________|_____po-debconf______|
|______name________|__%__|____details____|
|atlas             |96%  |    27/1/0     |
|atokx             |66%  |    14/7/0     |
|bugreporter-udeb  |50%  |    4/0/4      |
|dist              |     |    0/0/4      |
|htdig             |87%  |    7/1/0      |
|ilisp             |     |    0/0/5      |
|ilohamail         |85%  |    6/0/1      |
|libnet-perl       |95%  |    39/2/0     |
|libpaper          |66%  |    2/0/1      |
|mailagent         |     |    0/0/4      |
|mldonkey          |98%  |    70/1/0     |
|nagios            |96%  |    25/1/0     |
|nautilus-media    |     |    0/0/3      |
|openwebmail       |98%  |    98/0/1     |
|phalanx           |80%  |    4/1/0      |
|phpwiki           |94%  |    17/1/0     |
|pptpd             |69%  |    16/6/1     |
|silo-installer    |     |    0/0/3      |
|sl-modem          |     |    0/0/3      |
|spip              |86%  |    40/6/0     |
|vdr               |     |    0/0/9      |
|wwwoffle          |97%  |    48/0/1     |
|xcdroast          |92%  |    13/0/1     |
|__________________|_____|_______________|
|TOTAL (fr)        |90%  | 6131/86/521   |
|__________________|_____|_______________|

2) Les po restant a faire, en restreignant aux paquets debian et en classant
   par ordre popcon (by_inst se trouve sur popcon.debian.org, je crois)
   .
sed -e 's/#.*//' -e 's/^[0-9]* *//g' -e 's/ .*//' by_inst|egrep -v '^$' > popcon

./display_txt --status --show=po --debian --todo --list=popcon fr
                    ______________________
 __________________|_________po__________|
|______name________|__%__|____details____|
|dpkg              |99%  |   1002/4/1    |po(rfr)
|debconf           |96%  |    61/0/2     |
|whois             |97%  |    33/0/1     |
|grep-dctrl        |     |     ---       |po(itt rfr)
|apt-listchanges   |     |    0/0/13     |
|debootstrap       |     |     ---       |
|euro-support      |     |     ---       |
|apt-listbugs      |     |     ---       |
|linda             |     |   0/0/541     |
|gkdebconf         |     |    0/0/34     |
|gnome-find        |     |   0/0/303     |
|pdmenu            |     |    0/0/24     |
|pointerize        |     |     ---       |
|rpncalc           |     |    0/0/43     |
|xmorph            |     |   0/0/203     |
|quintuple-agent   | 3%  |    3/0/84     |
|fpm               |     |    0/0/66     |
|krecord           |     |    0/0/63     |
|textchk           |     |     ---       |
|dedit             |     |    0/0/98     |
|tlpr              |     |    0/0/18     |
|alml              |     |     ---       |
|turqstat          |     |     ---       |
|xcall             |78%  |   86/1/23     |
|xlog              |99%  |   267/0/1     |
|xconvers          |     |   0/0/101     |
|__________________|_____|_______________|
|TOTAL (fr)        |68%  | 3563/5/1619   |
|__________________|_____|_______________|

3) Etat des pages de manuel dans debian (ordre popcon)
./display_txt --show=man --debian --list=popcon fr
                    _______
 __________________| # man |
|______name________|_______|
|base-passwd       | 0/1   |
|debianutils       | 2/9   |
|dpkg              |14/24  |
|adduser           | 1/4   |
|apt               | 0/13  |
|debconf           |15/16  |
|defoma            | 0/14  |
|modconf           | 0/1   |
|debhelper         | 0/46  |
|base-config       | 1/5   |
|reportbug         | 0/2   |
|kernel-package    | 0/5   |
|po-debconf        | 5/5   |
|fakeroot          | 1/2   |
|initrd-tools      | 0/2   |
|pppconfig         | 0/1   |
|devscripts        | 0/19  |
|debsums           | 1/2   |
|apt-listchanges   | 0/1   |
|apt-show-versions | 0/1   |
|euro-support      | 0/1   |
|auto-apt          | 0/1   |
|debiandoc-sgml    | 0/1   |
|language-env      | 0/3   |
|cvs-buildpackage  | 0/5   |
|apt-show-source   | 1/1   |
|apt-move          | 0/1   |
|apt-src           | 0/1   |
|apt-proxy         | 1/3   |
|apt-build         | 1/1   |
|debget            | 0/1   |
|debarchiver       | 0/1   |
|__________________|_______|
|TOTAL (fr)        |43/193 |
|__________________|_______|
Significance of the 'man' column: [# french pages]/[# english pages]
WARNING: 'french' is hardcoded in that script for now.
WARNING: do not trust the stats about man for now.

(Tiens, des clients pour po4a...)

4) Comparatif entre des langues a la fois en po et en podebconf

./display_txt --total --show=po,podebconf de fr es it sv nl ja ru pl cs
de: po(69%;157993/5960/64695)  podebconf(47%;3174/283/3262)  
fr: po(68%;156112/6733/66164)  podebconf(90%;6131/86/521)  
es: po(57%;132568/6501/90461)  podebconf(38%;2561/127/4041)  
it: po(47%;116221/8964/119717)  podebconf(21%;1468/93/5142)  
sv: po(48%;108956/6404/111468)  podebconf(16%;1101/128/5505)  
nl: po(46%;104489/8674/113518)  podebconf(46%;3105/87/3542)  
ja: po(43%;99274/8428/120238)  podebconf(41%;2714/38/3859)  
ru: po(40%;95869/9276/129687)  podebconf(27%;1836/148/4744)  
pl: po(41%;94033/8223/125339)  podebconf(20%;1359/64/5311)  
cs: po(39%;90386/4392/134061)  podebconf(24%;1629/5/5100)  

5) Idem, en restreignant aux paquets debian
./display_txt --debian --total --show=po,podebconf de fr es it sv nl ja ru pl cs
de: po(78%;4270/53/1133)  podebconf(86%;1382/46/173)
fr: po(72%;4326/15/1624)  podebconf(97%;1556/1/44)
es: po(72%;4229/45/1552)  podebconf(77%;1236/36/329)
it: po(63%;3776/72/2069)  podebconf(69%;1109/17/475)
sv: po(68%;3963/18/1834)  podebconf(57%;913/56/632)
nl: po(72%;3479/44/1296)  podebconf(81%;1301/4/296)
ja: po(60%;2854/50/1787)  podebconf(78%;1254/13/334)
ru: po(52%;2857/231/2323)  podebconf(63%;1021/74/506)
pl: po(61%;3344/32/2054)  podebconf(66%;1064/34/503)
cs: po(55%;2997/33/2336)  podebconf(73%;1169/0/432)

(tiens, les trads sont en general meilleures dans debian qu'en dehors)

Tiens, dans ce cas, les 'de' ont un meilleur pourcentage de po avec moins de
chaines... Ca va pas etre simple a corriger...



Bonne nuit, Mt.

-- 
The tragedy of modern man is not that he knows less and less about the
meaning of his own life, but that it bothers him less and less.
          --- Vaclav Havel

#! /usr/bin/perl -w

use strict;
use Getopt::Long; #to parse the args

my $progname= $0; $progname= $& if $progname =~ m,[^/]+$,;

my $TRANSMONITOR_VERSION = "0.1"; #External Version Number
my $BANNER = "Translation Monitor -- text output programm v$TRANSMONITOR_VERSION"; # Version Banner - text form
my $DB_FILE="../data/unstable.gluck";
my $STATUS_FILE='../data/status.$lang';

my $list_file=undef;
my $take_debian=0;
my $fmt = "po,podebconf";
my $mask_done = 0;
my $show_status = 0;
my $show_total = 0;

use lib ($0 =~ m|(.*)/|, $1 or ".") ."/../../../../Perl";
use Webwml::L10n::Db;

sub syntax_msg {
    my $msg = shift;
    if (defined $msg) {
	print "$progname: $msg\n";
    } else {
	print "$BANNER\n";
    }
    print 
"Syntax: $0 [options] [lang]+
General options:
    -h, --help                display short help text
    -V, --version             display version and exit
  
Package selection:
    --debian                  Only take debian specific packages
    --list=file               Only handle the packages listed in the provided file
    -t,--todo                 Display only when the translation is NOT completed

Informations to display:
    --total                   Show only summary for the lang, not each package details
    -s,--status               Show status (hard to read when there is more than one format)
    --show=fmt                Show only selected format (instead of $fmt)

Database to use:
    --db=DB_FILE              use DB_FILE as database file 
                                (instead of $DB_FILE)
    --sdb=STATUS_FILE         use STATUS_FILE as status file 
                                (instead of $STATUS_FILE)
";
    if (defined $msg) {
	exit 1;
    } else {
	exit 0;
    }
}

# Display Version Banner
# Options: -V|--version, --print-version
sub banner {
  if ($_[0] eq 'print-version') {
    print "$TRANSMONITOR_VERSION\n";
  } else {
    print "$BANNER\n";
  }
  exit 0;
}

# Hash used to process commandline options
my %opthash = (# ------------------ general options
    "help|h" => \&syntax_msg,
    "version|V" => \&banner,
    "print-version" => \&banner,
    
    # ------------------ configuration options
    "todo|t"    => \$mask_done,
    "status|s"    => \$show_status,
    "total"  => \$show_total,
    
    "debian" => \$take_debian,
    "show=s"  => \$fmt,
    "db=s" => \$DB_FILE,
    "sdb=s" => \$STATUS_FILE,
    "list=s" => \$list_file,
);

# init commandline parser
Getopt::Long::config('bundling', 'no_getopt_compat', 'no_auto_abbrev');

# process commandline options
GetOptions(%opthash)
    or syntax_msg("error parsing options");



#-----------------------------------------------------------------------------
#                        The main program                                     
#-----------------------------------------------------------------------------
###
### initialisation
###

(@ARGV > 0) or &syntax_msg("Nothing to do !");
my $arg;
my @todo_lang;
while ($arg = shift @ARGV) {
    push @todo_lang,$arg;
}

my $data = Webwml::L10n::Db->new();
$data->read($DB_FILE);

my @todo_pkg;
if (defined($list_file)) {
    open LIST, "$list_file" || die "Impossible to read the list file $list_file\n";
    while (<LIST>) {
	chomp;
	next unless $_;
	if ($data->has_package($_)) {
	    push @todo_pkg, $_;
#	    print "['$_' added]\n";
	} # else {
#	    print "['$_' is not in the DB, skipped]\n";
#	}
    }
    close LIST;
} else {
    @todo_pkg = sort $data->list_packages();
}
my ($pkg,%p,%d,$man,$status);
my (%total);
my ($man_en_total,$man_fr_total);

my %parts;
map {$parts{$_} = 1} split (/,/, $fmt);
my @poparts=qw(po templates podebconf);
    
my $format_top = "format STDOUT_TOP = \n".
  '                    '.($parts{'po'}?'______________________':'').($parts{'templates'}?'______________________':'').($parts{'podebconf'}?'______________________':'').($parts{'man'}?'_______':'')."\n".
  ' __________________|'.($parts{'po'}?'_________po__________|':'').($parts{'templates'}?'______templates______|':'').($parts{'podebconf'}?'_____po-debconf______|':'').($parts{'man'}?' # man |':'')."\n".
  '|______name________|'.($parts{'po'}?'__%__|____details____|':'').($parts{'templates'}?'__%__|____details____|':'').($parts{'podebconf'}?'__%__|____details____|':'').($parts{'man'}?'_______|':'')."\n".
  ".\n";
# print $format_top;
eval $format_top;
die $@ if $@;

my $format = "format STDOUT = \n";
$format .= '|@<<<<<<<<<<<<<<<< |';
foreach my $part (@poparts) {
    $format .= '@||| |@||||||||||||| |' if ($parts{$part});
}
$format .= '@||||| |' if $parts{'man'};
$format .= "@<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<\n".'$pkg,               ';
foreach my $part (@poparts) {
    $format .= '$p{\''.$part.'\'},$d{\''.$part.'\'}, ' if ($parts{$part});
}
$format .= ' $man, ' if $parts{'man'};
$format .= ' $status'." ;\n.\n";
# print $format;
eval $format;
die $@ if $@;

map {
    my $lang=$_;
    my $statusDBname = "$STATUS_FILE";
#    print STDERR "Handle $lang\n" if (scalar @todo_lang > 1);
    $statusDBname =~ s/\$lang/$lang/g;
    my $statusDB = Webwml::L10n::Db->new();
    $statusDB->read($statusDBname,0) if $show_status;

    print "Status of the ".($take_debian?"debian ":"")."packages ".($mask_done?"to do ":"")."in $lang\n\n" unless $show_total;
    
    my %total;
    foreach $pkg (@todo_pkg) {
	# Take only packages having material
	next unless $data->has_po($pkg) || $data->has_templates($pkg) || $data->has_podebconf($pkg) || $data->has_man($pkg);

	# Take only debian packages
	if ($take_debian) {
	    next if ($data->has_upstream($pkg) && $data->upstream($pkg) ne "debian");
	    next if ($data->version($pkg) =~ m/-/);
	}

	# Take only packages having material in this language
	my $found=0;
	$man=' ';
	my (%score,%ori);
	$status='';
	foreach my $part (@poparts) {
	    my $has_part="has_$part";
	    $p{$part}='';
	    $d{$part}='';
	    if ($parts{$part} && $data->$has_part($pkg)) {
		$status .= "$part(";
		$score{$part} = '---';
		$found = 1;	
		foreach my $line (@{$data->$part($pkg)}){
		    my ($pofile, $langfound, $stat) = @{$line};
		    if ($langfound eq $lang) {
			$score{$part} = $stat;
		    
			$found = ! ($mask_done && output_percent($stat) eq '100%');
		    } elsif ($langfound eq '_') {
			$ori{$part} = add_stat($stat, $ori{$part});
		    }
		}
		$score{$part} = $ori{$part} if $score{$part} eq '---' && defined($ori{$part});
	    
		if (defined $score{$part} && $score{$part} ne '---') {
		    $total{$part} = add_stat($score{$part},$total{$part});
		}
		$p{$part}=output_percent($score{$part});
		$d{$part}=output_details($score{$part});
		
		if ($show_status && $statusDB->has_package($pkg) && $statusDB->has_status($pkg)) {
		    foreach my $statusline (@{$statusDB->status($pkg)}) {
			my ($kind,$translator,$status_from_db,$statusdate) = @{$statusline};
			if ($kind eq $part) {
			    $status .= "$status_from_db ";
			    $status  =~ s/<[^>]*>//g;
			}
		    }
		}
		$status .= ') ';
		$status  =~ s/ \)/)/;
		$status  =~ s/$part\(\)//;
	    }
	}
	  
	# Search for mans
	if ($parts{'man'} && $data->has_man($pkg)) {
	    my $en=0;
	    my $fr=0;
	    foreach my $line (@{$data->man($pkg)}){
		my ($name, $langfound) = @{$line};
		$en++ if ($langfound eq 'english');
		$fr++ if ($langfound eq 'french');
	    } 
	    $man="$fr/$en";
	    $man_fr_total += $fr;
	    $man_en_total += $en;
	    $found = 1 unless $mask_done && $fr == $en ;
	}
	
	write if ($found && !$show_total);
    }
    
    if ($show_total) {
	print "$lang: ";
	foreach my $part (@poparts) {
	    print "$part("
	      .output_percent($total{$part}).";"
	      .output_details($total{$part}).")  "
	      if ($parts{$part});
	}
	print "\n";
	
    } else {
	print "|__________________|";
	foreach my $part (@poparts) { 
	    print "_____|_______________|" if $parts{$part};
	}
	print "_______|" if $parts{'man'};
	print "\n";
    
	$pkg = "TOTAL ($lang)";
	foreach my $part (@poparts) {
	    $p{$part}=output_percent($total{$part});
	    $d{$part}=output_details($total{$part});
	}
	$man="$man_fr_total/$man_en_total" if $parts{'man'};
	write;
	
	print "|__________________|";
	foreach my $part (@poparts) { 
	    print "_____|_______________|" if $parts{$part};
	}
	print "_______|" if $parts{'man'};
	print "\n\n\n";
    
	print "When there is some ---, that means that the material exists, but is not \n".
	  "translated to this language and that some issue (in pot file or DB) prevent to find the amount of string.\n\n";
	print "Significance of the 'details' columns:\n".
	  "   [# translated strings]/[# fuzzy translation]/[# untranslated strings]\n\n";
	if ($parts{'man'}) {
	    print "Significance of the 'man' column: [# french pages]/[# english pages]\n";
	    print "WARNING: 'french' is hardcoded in that script for now.\n";
	    print "WARNING: do not trust the stats about man for now.\n";
	}
    }
} @todo_lang;


sub add_stat {
    my $new=shift;
    my $old=shift;
         
    return $new unless defined($old);
    $new =~ /([0-9]*)t([0-9]*)f([0-9]*)u/;
    my ($nt,$nf,$nu) = ($1||0, $2||0, $3||0);
    $old =~ /([0-9]*)t([0-9]*)f([0-9]*)u/;
    my ($ot,$of,$ou) = ($1||0, $2||0, $3||0);
    my $res= ($nt+$ot)."t".($nf+$of)."f".($nu+$ou)."u";
    #    print STDERR "$new + $old = $res\n";
    return $res;
}

sub output_percent {
    my $stats=shift||"";
    my $t = "0";
    my $u = "0";
    my $f = "0";
    my $percent;

    if ($stats =~ /([0-9]*)t/) {  $t=$1;  }
    if ($stats =~ /([0-9]*)u/) {  $u=$1;  }
    if ($stats =~ /([0-9]*)f/) {  $f=$1;  }
    $percent = calc_percent($t,$t+$u+$f);
    if ($percent eq "NaN" || $percent == 0) {
	return '';
    }
    return "$percent\%";
}
sub output_details {
    my $stats = shift||"";
    my $t = "0";
    my $u = "0";
    my $f = "0";
    my $percent;

    if ($stats =~ /([0-9]*)t/) {  $t=$1;  }
    if ($stats =~ /([0-9]*)u/) {  $u=$1;  }
    if ($stats =~ /([0-9]*)f/) {  $f=$1;  }
    return ($t+$f+$u == 0 ? $stats : "$t/$f/$u");
}

sub calc_percent{
    my $up=shift;
    my $down=shift;
    my $res;

    if ($down==0) {
	return "NaN";
    }
    $res = $up/$down*100;
    $res =~ s/^([0-9]*)\..*/$1/;
    return $res;
}

Attachment: signature.asc
Description: Digital signature

Reply to:

Follow-Ups:
- Re: Jouet de statistiques
  - From: barbier@linuxfr.org (Denis Barbier)
- Re: Jouet de statistiques
  - From: Patrice Karatchentzeff <p.karatchentzeff@free.fr>

Prev by Date: Re : [RFR] po-debconf://sl-modem/fr.po
Next by Date: Re: [RFR] po-debconf://openwebmail/fr.po
Previous by thread: [DONE] po-debconf://partman/fr.po
Next by thread: Re: Jouet de statistiques
Index(es):
- Date
- Thread