Hi, Snapshot frontends are served by two servers. One of which was generating frequent 500 under higher than usual traffic. On the 6th, DSA removed a configuration from the problematic server that was generating most of the 500s. The problem should be gone now. FYI, here is a count of 500 served by apache around that time: Date # of 500s 2023-01-05 286574 2023-01-06 173184 2023-01-07 24 2023-01-08 28 2023-01-09 7 2023-01-10 7 2023-01-11 87 2023-01-12 14 While I'm still investigating the rest of the generated 500s, the server should be more stable now. On Fri, 2023-01-06 at 10:54 -0500, Yaroslav Halchenko wrote: > confirming seeing those while talking to snapshot service from > appveyor > in > https://github.com/datalad/datalad/pull/7243 > https://ci.appveyor.com/project/mih/datalad/builds/45842271 > e.g. > > 2023-01-04T22:24:41+0000 [INFO ] datalad_installer > Downloading > http://snapshot.debian.org/archive/debian/20210906T204127Z/pool/main/g/git-annex/git-annex_8.20210903-1_amd64.deb > Traceback (most recent call last): > File "/home/appveyor/dlvenv/bin/datalad-installer", line 8, > in <module> > sys.exit(main()) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 1788, in main > return manager.main(argv) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 657, in main > self.addcomponent(name=cr.name, **cr.kwargs) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 690, in addcomponent > component(self).provide(**kwargs) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 1032, in provide > bins = self.get_installer(method).install(self.NAME, > **kwargs) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 1118, in install > bindir = self.install_package(package, **kwargs) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 1386, in install_package > download_file(url, debpath) > File "/home/appveyor/dlvenv/lib/python3.8/site- > packages/datalad_installer.py", line 1722, in download_file > with urlopen(req) as r: > File "/usr/lib/python3.8/urllib/request.py", line 222, in > urlopen > return opener.open(url, data, timeout) > File "/usr/lib/python3.8/urllib/request.py", line 531, in > open > response = meth(req, response) > File "/usr/lib/python3.8/urllib/request.py", line 640, in > http_response > response = self.parent.error( > File "/usr/lib/python3.8/urllib/request.py", line 569, in > error > return self._call_chain(*args) > File "/usr/lib/python3.8/urllib/request.py", line 502, in > _call_chain > result = func(*args) > File "/usr/lib/python3.8/urllib/request.py", line 649, in > http_error_default > raise HTTPError(req.full_url, code, msg, hdrs, fp) > urllib.error.HTTPError: HTTP Error 500: Internal Server Error > Command exited with code 1 > Running "on_finish" scripts > > > > On Wed, 04 Jan 2023, Andrew Pollock wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA512 > > > Hi, > > > We've been seeing intermittent 500's from snapshot.debian.org (in > > particular the machine-readable API URIs). > > > I haven't been able to pin it down to a particular IP address. Are > > there > > load balancers involved? > > > Also, the 500 page mentions emailing team@snapshot.debian.org, > > which > > doesn't appear to actually work... > > > regards > > > Andrew -- Baptiste Beauplat
Attachment:
signature.asc
Description: This is a digitally signed message part