Re: rewrite of UDD importers runner
Hi Lucas,
On Sun, Aug 16, 2015 at 11:50:38PM +0200, Lucas Nussbaum wrote:
> I changed the way UDD importers are executed.
>
> Previously, they were executed via cron.
>
> Now, there's a small cron-like daemon taking care of executing them.
> - the importers are grouped into 'pools'. only one importer of each pool
> can run at a given time, which should help reduce the load on the server
> at some times.
Sounds sensible. Is there any way to define a sequence? In
scripts I defined
cron_ftpnew_blends.sh
which was calling three importers in the correct sequence. The
rationale is that later importers might be override citations that are
more up-to-date than by the importer before. Can your pool method
implement such a determined sequence?
> - the exit status and the output of the execution is stored in a db.
> - there's a status page showing the status of each importer:
> https://udd.debian.org/udd-status.cgi
> - since the importers no longer run from cron, no mails are sent when
> they fail. Instead, there's a separate script providing a list of
> current errors via email, ran 4 times a day via cron. Please don't
> ignore this email, but help fix the errors instead. (I'm working on the
> ones for ubuntu-bugs and archive-ubuntu.)
I'd love to work on the importers I have written but might not
understand others.
> - to add an importer, simply add it to rudd.conf.
Is this somewhere outside of this mail documented?
> - all importers have been migrated. There's still a few other things
> running via cron (database dumps, for example).
>
> Please let me know if you have questions / want more info.
Thanks for your work on this
Andreas.
--
http://fam-tille.de
Reply to: