Bug#1006326: issues connecting to https://bugs.debian.org/cgi-bin/soap.cgi (generation of wnpp related pages)
Package: www.debian.org
User: www.debian.org@packages.debian.org
Usertag: scripts
Severity: normal
Dear all
Since several days, the script wnpp.pl fails intermittently when trying to connect to bugs.debian.org, and when it fails, we receive the following messages in the webwml build log:
[...]
make -C wnpp install
make[3]: Entering directory '/srv/www.debian.org/webwml/english/devel/wnpp'
wml -q -D CUR_YEAR=2022 -o UNDEFuEN:../../../english/devel/wnpp/wnpp.data@g+w -W7,-Squotes ../../../english/devel/wnpp/wnpp.pl
ePerl:Error: Perl runtime error (interpreter rc=2)
---- Contents of STDERR channel: ---------
500 Can't connect to bugs.debian.org:443 (Connection refused) at /tmp/wml.12011.tmp1 line 147.
------------------------------------------
** WML:Break: Error in Pass 3 (rc=1).
Died at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 403.
TheWML::Frontends::Wml::Runner::_run_pass(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458), 3, SCALAR(0x55f984f718d8), SCALAR(0x55f984f718a8), SCALAR(0x55f984f718c0)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 441
TheWML::Frontends::Wml::Runner::_passes_loop(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 726
TheWML::Frontends::Wml::Runner::_output_and_cleanup(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 930
TheWML::Frontends::Wml::Runner::run_with_ARGV(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458), HASH(0x55f984b252b8)) called at /usr/bin/wml line 47
make[3]: *** [Makefile:34: ../../../english/devel/wnpp/wnpp.data] Error 2
make[3]: Leaving directory '/srv/www.debian.org/webwml/english/devel/wnpp'
make[2]: [../../Makefile.common:84: wnpp-install] Error 2 (ignored)
make[2]: Leaving directory '/srv/www.debian.org/webwml/english/devel'
[...]
Since the line refers to a temp file, I'm not 100% sure about which is the command that causes the failure, but the code in wnpp.pl that uses bugs.debian.org I think is restricted to these lines (initial connection, and retrieval of bug info to build the wnpp.data local database):
https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/devel/wnpp/wnpp.pl#L41 (L41 to 48)
my $soap = SOAP::Lite->uri('Debbugs/SOAP')->proxy('https://bugs.debian.org/cgi-bin/soap.cgi')
or die "Couldn't make connection to SOAP interface: $@";
my $bugs = $soap->get_bugs(package=>'wnpp')->result;
my $status = {};
while (my @slice = splice(@$bugs, 0, 500)) {
my $tmp = $soap->get_status(@slice)->result() or die;
%$status = (%$status, %$tmp);
}
Some people commented the issue in the #debian-admin IRC channel, for the case there was some issue in the bugs.d.o servers or some configuration change, but it seems the failures are related to the connections being blocked by the http rate-limiting. It's not clear, however, why these issues are appearing now (nothing in the script changed in the recent times, and the amount of bugs is not drastically bigger than weeks before). In addition to this, some other days the script is tolerated well (for example, we had failures on 2022/02/18 and 2022/02/22, but today 2022/02/23 the builds are finishing successfully).
Looking at the code, I see we're processing bugs in groups of 500 bugs. I'm not sure if we could try to reduce the number of the bugs in the group and introduce some "sleep" time between queries, or it's better to do the opposite, increase the number of bugs in a group to reduce the total number of queries.
Other approach could be to rewrite the wnpp.pl script so it uses queries against UDD instead of the SOAP CGI interface of bugs.debian.org
I don't know which approach is better, and also my Perl skills are not enough to propose a patch myself for none of this two options. If anybody can and wants to work on this, it's very appreciated.
I'll point the Debbugs team to this bug one it is created for the case they changed any configuration in their deployment that caused the connection failures, or for the case they have any advice about the better way to connect or retrieve the information we need to create the WNPP-related web pages.
Kind regards,
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
Reply to: