Re: Is Access to Salsa restricted to a certain number of queries per time and host?
On Wed, Jul 11, 2018 at 01:34:29PM +0200, Julien Cristau wrote:
> On 07/11/2018 10:18 AM, Andreas Tille wrote:
> > Hi,
> >
> > I'm running a daily cron job on host blends.debian.net to gather machine
> > readable data from all blends packages. The cron job fetches only the
> > following files
> >
> > debian/changelog
> > debian/control
> > debian/copyright
> > debian/README.Debian
> > debian/upstream/edam
> > debian/upstream/metadata
> >
> > (if the latter two exist) from about 2000 repositories. These data are
> > consumed in UDD from where they are used in the Blends web sentinel. The
> > script which is running can be found in Git[1].
> >
> > Unfortunately the cron job seems to stop with
> >
> > Traceback (most recent call last):
> > File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 251, in wrapped_f
> > return f(*args, **kwargs)
> > File "/usr/lib/python3/dist-packages/gitlab/mixins.py", line 48, in get
> > server_data = self.gitlab.http_get(path, **kwargs)
> > File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 728, in http_get
> > streamed=streamed, **kwargs)
> > File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 706, in http_request
> > response_body=result.content)
> > gitlab.exceptions.GitlabHttpError: 429: b'Retry later\n'
> >
> > During handling of the above exception, another exception occurred:
> >
> > Traceback (most recent call last):
> > File "/srv/blends.debian.org/misc/machine_readable/fetch-machine-readable_salsa.py", line 106, in <module>
> > project = gl.projects.get(pr.attributes['id']) # without this extra get repository_tree() fails
> > File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 253, in wrapped_f
> > raise error(e.error_message, e.response_code, e.response_body)
> > gitlab.exceptions.GitlabGetError: 429: b'Retry later\n'
> >
> You could probably save yourself some trouble by not polling repos that
> have had no activity since you last looked at them.
Or by setting up a webhook[0] so the relevant repos can notify you when
there's a commit that changes the files you care about.
[0]: https://salsa.debian.org/help/user/project/integrations/webhooks.md
Cheers,
--
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB
Reply to: