Re: [Nbd] Starting over?

[Alex Bligh <alex@...872...>]

Wouter,

>> Yeah I rewrote negotiation as a state machine (the idea being to publish
>> it under something more liberal than GPL) and it was truly fugly.
> 
> My dabbling came to the same conclusion.
> 
> But I think that's okay. We can keep negotiation as a blocking thing --
> we need do it only once, anyway -- and implement a state machine for the
> data pushing phase. The "negotiation" would be a single state in the
> whole state machine, then.

Sure.

>> I agree using the stack for state (as you put it) is not nice. But
>> if the idea is merely to handle simultaneous requests, threading
>> might be an alternative route. IE have one thread handling the
>> socket, and a bunch of workers handling the requests.
> 
> My dabbling was going down the "start one thread per request" route. It
> became complicated because of my misunderstanding that a FLUSH needed to
> ensure that all outstanding requests have flushed; but since it's clear
> now that that was a misunderstanding, I suppose we can go down that
> road easily.

So I got this working (at least to the extent it passed the torture tests
for nbd I wrote - I think they are in tree now) and carefully wrote it
to conform to the kernel's definition of flush / FUA. I wrote some
clarifying text for the kernel's documentation now. I can't quite remember
the flush semantics.

This was for an investigatory / research project to provide an nbd server
endpoint with a variety of back ends - there's a simple file back end,
an S3 back end (IIRC) and a number of other bits and bobs. It's a bit
more complex than that as it handled snapshots and so forth. In the end
Ceph appeared, and that pretty much made it pointless.

It's far from speed optimised. I'm wondering if I can just open source
it (probably dual Apache & (L)GPL). I'm not sure you we would want to use
any of the code but it might (or might not) be interesting to look at.
My concern is whether it uses any libraries that are proprietary. If
I square that circle, is this interesting to you?

>> This would be trivial in golang.
> 
> Sure, but golang is portability--; x86* and arm* only, AFAICS, whereas C
> works everywhere.

Indeed, though I think that's changing.

> There's also a GThreadPool API in glib, which
> simplifies things a bit. We'd just have to do this:
> 
> main thread:
> - read request off socket
> - if request.length > 0, read data
> - g_thread_pool_push(pool, buffers, &error)

I'm not familiar with glib threads (I just wrote to the POSIX API).
In practice with the clients I tried (Linux kernel and Qemu) there
were very few overlapping requests. What I did was have a fixed
size pool of worker threads, and allocated requests to them. It's
only when Flush comes that you have to start being careful.

The value of this approach really appears when your connection and
ability to serve data is fast but carries latency.

> in the thread
> - get buffers (we have a pointer to those buffers as a function
>  argument)
> - handle request (when we need to read or write, use pread() or
>  pwrite() so multiple threads can safely deal with the same file at
>  different offsets)
> - lock a mutex (for the socket)
> - send the reply
> - clean up (free buffers, unlock mutex, ...)
> - exit function

I actually sent the packets back from another thread I think.
I can't immediately remember why. Your approach seems easier.

Alex

> 
> That's not *too* hard, right?
> 
> -- 
> It is easy to love a country that is famous for chocolate and beer
> 
>  -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26
> 
> 

-- 
Alex Bligh