Re: Niced cron jobs
On Fri, Jun 25, 1999 at 09:53:10PM -0700, Joey Hess wrote:
> Craig Sanders wrote:
> > niced cron jobs be a disastrous on heavily loaded servers that
> > never have any idle time to service the niced jobs. you can end up
> > with several cron jobs running simultaneously, some of them sucking
> > up massive amounts of memory (e.g. calamaris running on a squid
> > box or analog on a web server) and slowing the system down even
> > further. you can end up with the cron jobs from several days running
> > simultaneously, which complicates things even more (most cron jobs
> > are written with the assumption that only one instance will be
> > running at a time, which can cause locking or other contention
> > problems)
> >
> > this is not just speculation, it's experience. i've seen it happen
> > because i've tried it myself. i thought that nicing the cron jobs
> > would result in a performance boost, smooth out the load. what
> > actually happens is that the load goes through the roof and can
> > bring down the machine due to resource starvation.
>
> That's very interesting. Did you try tuning the nice levels so it
> wasn't quite *so* nice?
it doesn't matter. if there is no CPU available for nice -10 tasks
because the machine is too busy then there is no CPU available for nice
-9 or -8 or -1 either. because they hardly ever get any cpu cycles the
cron jobs stay around for ages, using memory, file handles, and other
resources.
at first glance, using nice seems like a good way of smoothing out the
load caused by cron but it is nowhere near that simple in practice.
on busy machines it creates a situation which gets progressively
worse until the machine runs out of resources and dies....while on
lightly loaded machines it makes hardly any difference at all because
the thrashing is disk activity, not CPU load (certainly not enough
difference to justify endangering server boxes).
the effect is like going out of your way to mis-tune a machine.
simplistic solutions are dangerous.
forcing everyone to use nice for the cron.{daily,weekly,monthly} jobs
would cause harm for no noticeable benefit.
> But consider this: Heavily loaded servers are rare.
they are not rare at all. nearly all of my debian machines are servers,
many of them under moderate to heavy load. And these are the machines
that matter - they have to be up and running 24x7 even though they
aren't monitored by a human 24x7. OTOH, workstations generally *are*
monitored by a human whenever they are in use.
> Most linux boxes (that arn't running a niced client like
> distributed-net) are idle 99% of the time.
> Heavily loaded servers that don't have an admin who can deal with a
> problem caused by niced cron jobs are even rarer.
why break something that is already working fine? why make people do
extra work that they don't have to? what good does it serve to make yet
another special case, yet another file that has to be edited after every
upgrade?
theoretical benefits may be nice, but i'll take practical experience
over untested theory any day.
so far we have a few people who think it's a great idea no matter what
(even thought they haven't actually tried it), one person (me) who HAS
tried it and reported that it causes problems in some situations, and a
bunch of other people who thought it was a good idea at first but are
willing to learn from the experience of others. if you think it's such
a great idea why dont you test it and report back with what benefits it
gained you?
> On the other hand, lightly loaded systems whose admin has no idea why
> their disk starts thrashing and everything gets so slow are quite
> common.
>
> Seems to me we should address the common case.
the common case is that people DON'T notice the cron jobs thrashing
their system because they run at approx 6:45am. The common case is that
people are not using their workstations at 6.45am. the only people who
notice are those running who run anacron, which runs the jobs when they
boot the machine. as i noted in a previous message, using nice for
anacron would probably be a good idea.
(i'll ignore the cases of pathological weirdoes who wake up at obscene
hours, and for those who are still awake from the night before the cron
jobs are a good alert that getting some sleep would be a good idea :)
at the most there should be an /etc/cron.nice.conf (or an extra field
in /etc/default/rcS) file which allows people to set once and for all
what nice level their cron jobs run at - this could be read in by
/etc/init.d/cron when cron starts up, or alternatively by run-parts.
if you're going to advocate doing weird and unusual things then you
should at least advocate doing it in a typically sensible debian fashion
- don't force it on everyone, make it an easily tweaked configuration
option.
craig
--
craig sanders
Reply to: