Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)
On Wed, Jan 24, 2018 at 05:32:45PM +0100, Cyril Brulebois wrote:
> > mod_fcgid: stderr: Can't call method "get_document" on an undefined value at ../lib/Packages/Search.pm line 264.
> > End of script output before headers: dispatcher.fcgi
>
> Looking at the code, that's a method call on a xapian query result.
>
> Looking at the xapian directory on picconi:
> > pkg_user@picconi:/srv/packages.debian.org$ ls -l files/db/xapian/
> > total 302384
> > -rw-r--r-- 1 pkg_user pkg_maint 1974272 Jan 24 15:56 docdata.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 15:53 flintlock
> > -rw-r--r-- 1 pkg_user pkg_maint 138 Jan 24 15:56 iamglass
> > -rw-r--r-- 1 pkg_user pkg_maint 181952512 Jan 24 15:56 position.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 80699392 Jan 24 15:56 postlist.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 44998656 Jan 24 15:56 termlist.glass
>
> Looking at the xapian directory on the mirror:
> > pkg_user@pkgmirror-csail:/srv/packages.debian.org$ ls -l files/db/xapian/
> > total 537584
> > -rw-r--r-- 1 pkg_user pkg_maint 1974272 Jan 24 09:48 docdata.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 09:45 flintlock
> > -rw-r--r-- 1 pkg_user pkg_maint 28 Nov 8 09:26 iamchert
> > -rw-r--r-- 1 pkg_user pkg_maint 138 Jan 24 09:48 iamglass
> > -rw-r--r-- 1 pkg_user pkg_maint 1699 Nov 8 09:30 position.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint 1725 Nov 8 09:30 position.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint 111788032 Nov 8 09:30 position.DB
> > -rw-r--r-- 1 pkg_user pkg_maint 181944320 Jan 24 09:48 position.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 1254 Nov 8 09:30 postlist.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint 1263 Nov 8 09:30 postlist.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint 81616896 Nov 8 09:30 postlist.DB
> > -rw-r--r-- 1 pkg_user pkg_maint 80707584 Jan 24 09:48 postlist.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 54 Nov 8 09:30 record.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint 56 Nov 8 09:30 record.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint 2523136 Nov 8 09:30 record.DB
> > -rw-r--r-- 1 pkg_user pkg_maint 691 Nov 8 09:30 termlist.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint 703 Nov 8 09:30 termlist.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint 44883968 Nov 8 09:30 termlist.DB
> > -rw-r--r-- 1 pkg_user pkg_maint 45006848 Jan 24 09:48 termlist.glass
>
> For files dated Jan 24, the timestamps don't match, but that's probably
> just a sync waiting to happen, and that doesn't explain the issues which
> have been happening for so long. I suspected the stray files instead,
> dating back to November. I've created an “old-files” directory and moved
> them there, and the mirror seems to be behaving properly now.
>
> I'm tagging this bug report with “pending” for the time being, to give
> people a chance to comment. But I suppose it should be safe to remove
> those old files entirely?
Yes. The directory on pkgmirror-csail actually had two different
versions of the database using different database backends, so the
files which aren't in the directory on picconi are essentially an
old version of the same database.
That alone shouldn't cause the search to fail though - it should just
open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
open the old one in this rather odd situation).
Xapian's replication should cleanly handle the database being replicated
switching backends so I'm guessing the replication here is using rsync
(without --delete-after) or similar? That might also explain why the
old database is broken as the databases aren't safe to rsync if an
update is in progress - if the last rsync of the old database happened
while an update was in progress, it could have been left broken, which
would typically result in particular searches failing.
Updating with rsync can also break searches on the mirror while the
rsync is in progress even if the master isn't updating, so perhaps we
should switch to using Xapian's replication? That's already being
used for the main website search (I set it up with weasel at Debconf
15).
Happy to assist with that, though I'll probably need ssh access to
pkgmirror-csail (seems I currently only have it for picconi). My
Debian username is "olly".
Cheers,
Olly
Reply to: