Re: [Nbd] Async writes and NBD's deadlock

[Wouter Verhelst <wouter@...3...>]

On Fri, Mar 03, 2006 at 09:35:50AM -0800, Daniel Phillips wrote:
> Bogdan Harjoc wrote:
> >Hi,
> >
> >It may well be the case that I misunderstood the description
> >in the README for the uniprocessor deadlock problem, but
> >from what I can tell, appending NBD_CMD_WRITE requests
> >to a queue and returning with err=0 immediately instead of
> >eventually blocking while calling write(2) should avoid the
> >deadlock (would having to touch the swap device change
> >anything ?).

That might work, haven't tried it.

Note that the deadlock described in the README has nothing to do with
swap, but everything with filesystems.

> >Although probably not that useful in general, I would be
> >grateful for any comments on whether it should work or
> >not.
> 
> I am not sure that the deadlock described in the README even exists,

I'm quite sure of it, actually.

> this looks to me like just another instance of the memory inversion
> deadlock problem which manifests in at least one form in NBD:
> 
>   1. The daemon may use a syscall that allocates memory (e.g., write)
> 
>   2. But the kernel is completely out of memory

No, this is absolutely not the problem. I've seen the deadlock at points
where there were still huge amounts of free memory, both in RAM and in
swapspace.

It goes like this:

nbd-server 1234 /some/file
nbd-client localhost 1234 /dev/nb0 # on the same host
mount /dev/nb0 /some/path

Now write huge gobs of data to /some/path, and wait a few seconds. If
you compile nbd with CFLAGS='-DDODBG -DNOFORK', you'll even see that it
does write, but suddenly stops while in uninterruptible sleep. At that
point, you cannot write /anything/ to your filesystem anymore, and every
process that tries to will go to the same uninterruptible sleep.

>   3. So the daemon blocks
> 
>   4. But the kernel needs to write out dirty memory over the blocked
>     daemon
> 
>   5. Oops.
> 
> The solution to this is not pretty.  It starts with memlock_all and
> continues with hackery to give the daemon access to some of the
> kernel memory normally reserved for the vm system.  I will be coding
> up an in-kernel NBD-like daemon pretty soon which attempts to
> address this and other problems.  As for fixing it in user space...
> good luck.  It's possible but you won't get it done this afternoon.

Nor is it necessary. The issue only occurs when you try to do NBD to
localhost, which is nice for testing but otherwise not very useful.

-- 
Fun will now commence
  -- Seven Of Nine, "Ashes to Ashes", stardate 53679.4