Re: upstream-metadata.debian.net restarted.
(dropping blends list for the moment)
Le Mon, Jun 09, 2014 at 11:32:44PM +0800, Paul Wise a écrit :
> On Mon, Jun 9, 2014 at 11:00 PM, Charles Plessy wrote:
>
> > While as a website upstream-metadata.debian.net was never off, in practice the
> > system has been broken for a couple of months.
>
> As a result we had to change (see r3061 in the qa SVN) the PTS to link
> to the collab-qa SVN repo instead of yaml-all on the upstream-metadata
> website. As far as I can tell this isn't yet working, will you be
> fixing that too?
>
> http://upstream-metadata.debian.net/umegaya/chromium-bsu/yaml-all
> https://anonscm.debian.org/viewvc/collab-qa/packages-metadata/c/chromium-bsu.upstream?view=co
Ouch…
So in the end there is a bug that I can not understand.
Let me first remind how things should work.
The ‘umegaya’ web service is a Perl program that ties a hash to a Berkeley DB.
This database contains results from previous queries, and a timestamp for the
last access, so that it is used as a cache if the last access was recent. If
not, querying the webservice trigger a refreshing of the information. Then,
the result of the query is returned.
To refresh the information about a package, the web service searches in the
Berkeley DB via the tied hash for a VCS URL where to download a metadata file.
If that URL is not available, it is inferred by calling ‘debcheckout -d’. The
metadata file is then parsed and its contents are inserted in the Berkeley DB.
A copy is saved on the local filesystem, which is synchronised daily with the
collab-qa Subversion repository on Alioth.
On blends.debian.net, which hosts http://upstream-metadata.debian.net, attempts
to refresh information on chromium-bsu (and probably other packages) are
silently ignored. In the following example, I query the system for the ‘test’
key of the ‘chromium-bsu’ package. I added it in the Berkeley DB via
the command-line tool ‘umegaya-adm’:
plessy@debian-med:~$ umegaya-adm -p chromium-bsu -k test -v asfsad
plessy@debian-med:~$ umegaya-adm -p chromium-bsu -k test
asfsad
plessy@debian-med:~$ umegaya-adm -p chromium-bsu
test asfsad
YAML-URL svn://anonscm.debian.org/pkg-games/packages/trunk/chromium-bsu/
At this point, the Berkeley DB contains two entries for chromium-bsu: test and
YAML-URL. YAML-REFRESH-DATE is not set to querying the web service
will automatically trigger a refresh. Here is what it gives in debug mode.
http://upstream-metadata.debian.net/cgi-bin/umegaya?package=chromium-bsu&key=yaml-all&debug=1
Content-Type: text/plain; charset=utf-8
Debug mode on; somebody is probably working on this site…
chromium-bsu is not yet in the database, trying to pull its metadata…
Refreshing data for chromium-bsu
Running “svn cat svn://anonscm.debian.org/pkg-games/packages/trunk/chromium-bsu/debian/upstream/metadata”)
Writing /var/lib/umegaya/packages-metadata/c/chromium-bsu.upstream ... done.
Running “svn cat svn://anonscm.debian.org/pkg-games/packages/trunk/chromium-bsu/debian/control”)
Writing /var/lib/umegaya/packages-metadata/c/chromium-bsu.control ... done.
Running “svn cat svn://anonscm.debian.org/pkg-games/packages/trunk/chromium-bsu/debian/copyright”)
Writing /var/lib/umegaya/packages-metadata/c/chromium-bsu.copyright ... done.
Deleting chromium-bsu:test
Key type is .
Setting chromium-bsu:Contact to chromium-bsu-devel@lists.sourceforge.net
Key type is .
Setting chromium-bsu:Bug-Submit to https://sourceforge.net/p/chromium-bsu/bugs/new/
Key type is .
Setting chromium-bsu:Repository to git://git.code.sf.net/p/chromium-bsu/code
Key type is .
Setting chromium-bsu:FAQ to http://chromium-bsu.sourceforge.net/faq.htm
Key type is .
Setting chromium-bsu:Homepage to http://chromium-bsu.sourceforge.net/
Key type is .
Setting chromium-bsu:Screenshots to http://sourceforge.net/projects/chromium-bsu/
Key type is .
Setting chromium-bsu:Repository-Browse to http://sourceforge.net/p/chromium-bsu/code/
Key type is .
Setting chromium-bsu:Changelog to http://sourceforge.net/p/chromium-bsu/code/ci/master/log/
Key type is .
Setting chromium-bsu:Name to Chromium B.S.U.
Key type is .
Setting chromium-bsu:Watch to http://sf.net/chromium-bsu/chromium-bsu-(\d[\d\.]+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
Key type is .
Setting chromium-bsu:Bug-Database to http://sourceforge.net/p/chromium-bsu/bugs/
Looking for key yaml-all of package chromium-bsu.
No yaml-all key for package chromium-bsu.
So the metadata file for chromium-bsu is found and parsed, but not inserted
in the Berkeley DB. Here is the Perl code that produces the output above:
# Drops all previous keys.
foreach (keys (%stored)) {
next unless /^$package:/;
next if /:YAML-URL$/;
say "Deleting $_" if $debug;
undef $stored{$_};
}
# Loads the new keys.
foreach (keys (%$package_metadata)) {
my $key = $package_metadata->{$_};
my $keytype = ref($key);
say "Key type is $keytype." if $debug;
# Compound keywords are equivalent to nested mappings:
#
# Foo-Bar: baz
#
# is the same as:
#
# Foo:
# Bar: baz
#
if ( $keytype eq "HASH" ) {
my $left = $_;
foreach ( keys (%$key)) {
say "Setting ${package}:${left}-$_ to $key->{$_}" if $debug;
$stored{"${package}:${left}-$_"} = $key->{$_};
}
} elsif ( $keytype eq "ARRAY" ) {
say "Setting ${package}:$_ to $key" if $debug;
$stored{"${package}:$_"} = $key;
} else {
say "Setting ${package}:$_ to $key" if $debug;
$stored{"${package}:$_"} = $key;
}
}
# Keeps a backup of the 'upstream' file.
$stored{"${package}:YAML-ALL"} = $package_metadata_yaml;
This works for other packages and I do not understand why the changes to the
hash fail silently for this package while they work for others. Also, on other
computers, I have no problem with chromium-bsu. Lastly, I have seen a very
similar problem for the ‘emboss’ package earlier, that I solved by restarting
the Berkeley DB from scratch. Therefore, I am not able to reduce this bug
to a simple test case and I am blocked…
(and I exhausted my free-time-in-front-of-computer for this week-end…)
Have a nice Sunday,
--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan
Reply to: