Re: [Nbd] Starting over?

[Wouter Verhelst <w@...112...>]

Hi Alex,

On Fri, Jun 05, 2015 at 09:31:53AM +0100, Alex Bligh wrote:
> Wouter,
> 
> >> Yeah I rewrote negotiation as a state machine (the idea being to publish
> >> it under something more liberal than GPL) and it was truly fugly.
> > 
> > My dabbling came to the same conclusion.
> > 
> > But I think that's okay. We can keep negotiation as a blocking thing --
> > we need do it only once, anyway -- and implement a state machine for the
> > data pushing phase. The "negotiation" would be a single state in the
> > whole state machine, then.
> 
> Sure.
> 
> >> I agree using the stack for state (as you put it) is not nice. But
> >> if the idea is merely to handle simultaneous requests, threading
> >> might be an alternative route. IE have one thread handling the
> >> socket, and a bunch of workers handling the requests.
> > 
> > My dabbling was going down the "start one thread per request" route. It
> > became complicated because of my misunderstanding that a FLUSH needed to
> > ensure that all outstanding requests have flushed; but since it's clear
> > now that that was a misunderstanding, I suppose we can go down that
> > road easily.
> 
> So I got this working (at least to the extent it passed the torture tests
> for nbd I wrote - I think they are in tree now) and carefully wrote it
> to conform to the kernel's definition of flush / FUA. I wrote some
> clarifying text for the kernel's documentation now. I can't quite remember
> the flush semantics.
> 
> This was for an investigatory / research project to provide an nbd server
> endpoint with a variety of back ends - there's a simple file back end,
> an S3 back end (IIRC) and a number of other bits and bobs. It's a bit
> more complex than that as it handled snapshots and so forth. In the end
> Ceph appeared, and that pretty much made it pointless.

Right.

> It's far from speed optimised. I'm wondering if I can just open source
> it (probably dual Apache & (L)GPL). I'm not sure you we would want to use
> any of the code but it might (or might not) be interesting to look at.
> My concern is whether it uses any libraries that are proprietary. If
> I square that circle, is this interesting to you?

I suppose it could be useful as 'something to look at while
reimplementing the same thing', I guess.

> >> This would be trivial in golang.
> > 
> > Sure, but golang is portability--; x86* and arm* only, AFAICS, whereas C
> > works everywhere.
> 
> Indeed, though I think that's changing.

It would also require a reimplementation, something you *just* convinced
me not to do ;-)

> > There's also a GThreadPool API in glib, which
> > simplifies things a bit. We'd just have to do this:
> > 
> > main thread:
> > - read request off socket
> > - if request.length > 0, read data
> > - g_thread_pool_push(pool, buffers, &error)
> 
> I'm not familiar with glib threads (I just wrote to the POSIX API).

The glib GThread API is pretty much modeled on the POSIX one.

But even so, that's irrelevant; I'm not talking about GThread, I'm
talking about GThreadPool. With the latter, you create a GThreadPool
object with a function to run when data appears, tell it a maximum nuber
of threads, and then call g_thread_pool_push() every so often. That last
function will check the number of idle threads and the number of
available thread slots (or whatever), and will then, depending on the
situation, call your function with your data in an already-running
thread, fire up a new thread with your data, or queue your new data for
the first available thread.

The system will also occasionally clean up idle threads.

In other words, you end up not dealing with threads directly. I suppose
I could implement a thread pool myself, but why should I? We're already
using glib.

> In practice with the clients I tried (Linux kernel and Qemu) there
> were very few overlapping requests. What I did was have a fixed
> size pool of worker threads, and allocated requests to them. It's
> only when Flush comes that you have to start being careful.

Since I was told (by Paolo) that TRIM isn't supposed to care about
outstanding-but-not-finished requests, that isn't very relevant. TRIM
can just call fsync() on all open files for that socket and return
success (or failure, if fsync() fails).

> The value of this approach really appears when your connection and
> ability to serve data is fast but carries latency.

Right.

> > in the thread
> > - get buffers (we have a pointer to those buffers as a function
> >  argument)
> > - handle request (when we need to read or write, use pread() or
> >  pwrite() so multiple threads can safely deal with the same file at
> >  different offsets)
> > - lock a mutex (for the socket)
> > - send the reply
> > - clean up (free buffers, unlock mutex, ...)
> > - exit function
> 
> I actually sent the packets back from another thread I think.
> I can't immediately remember why. Your approach seems easier.

It would also be slightly faster, especially if the thread from which
you sent the reply packets was the same thread on which you read
incoming requests.

-- 
It is easy to love a country that is famous for chocolate and beer

  -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26