[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

On Mon, 2010-06-21 at 11:11 +0100, Ian Jackson wrote:
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> > We really need to see the kernel messages reporting soft-lockup.
> There aren't any.  Or, if there are, it isn't printing them to the
> serial console.  Perhaps it is trying to send them only to syslogd
> which obviously can't write to the disk either.
> I know that the serial console is working after the crash as it echoes
> my first carriage return (although then login or the shell wedges
> before being able to print a prompt).  I could have asked for a magic
> sysrq process dump but given that the process table is probably full I
> expect this would take many hours at 9600bps.

Even if you can't get a process dump, you can get some useful
information with:
'd' - show locks held
'l' - show backtrace for active CPUs
'w' - show uninterruptible tasks

> > Note that your disk configuration is unusual and probably not
> > well-tested by others.  It is unlikely that anyone in the kernel team
> > will be able to debug this as I don't think any of us have particular
> > expertise in this area.
> Search the web suggests that symptoms very similar to mine are not
> uncommon, including instances without soft lockup messages, and none
> of the other users seem to have a similar disk layout.
> I can't easily test this theory but I think the unusual disk layout is
> probably simply making a race easier to trigger.

Thinking of some kind of lock-dependency bug?  Blocking on a mutex for a
long period should still trigger a soft-lockup message.  Since there are
no messages from the kernel it's something of a mystery what's going on.

> >  If you have the opportunity, it would be
> > helpful if you could test a newer kernel version.  This would give us a
> > clue as to whether the bug has subsequently been fixed upstream, and if
> > not then it would be the basis for an upstream bug report.
> Unfortunately testing this bug on the live system involves (a) risking
> a crash and then if the test fails (b) an extremely vulnerable system
> without backups for the following four days.

That's what I suspected.

> I'll see if I can borrow a spare R210 from Jump, in which case I may
> be able to reproduce the problem in controlled conditions on my coffee
> table at home (and with access to the VGA console).  Which kernel
> should I test in that case ?

Please try 2.6.34 from experimental.


Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

Attachment: signature.asc
Description: This is a digitally signed message part

Reply to: