Re: [Nbd] Yet another NBD server out there

[Wouter Verhelst <w@...112...>]

On Sun, Mar 10, 2013 at 10:20:46PM +0100, Michal Belczyk wrote:
> On Sun, Mar 10, 2013 at 04:02:30PM +0100, Wouter Verhelst wrote:
> > On Sat, Mar 09, 2013 at 11:36:22PM +0100, Michal Belczyk wrote:
> > > On Sat, Mar 09, 2013 at 02:17:19PM +0100, folkert wrote:
> > > > It also would just disappear without any messages in syslog.
> > > 
> > > Now this looks like a bug to me -- could you please file an issue for it
> > > on the project's bitbucket site and provide more details?
> > > Much appreciated!
> > > I think we should discuss this in private as this mailing list is
> > > definitely not the right place to solve any bnbd-server issues...
> > 
> > why not?
> 
> Oh come on, bnbd project's bugtracker is the right place to submit bugs.

this mailinglist isn't limited to the reference implementation. If it's
a bug, sure. Other than that, as long as it's nbd-related, it's welcome
here.

> > [...]
> > > > > It is a true network _block_ device, not a network _memory_ device as it
> > > > > does not take any advantage of the buffer cache on the data origin
> > > > > server.
> > > > 
> > > > What's the advantage of that?
> > > 
> > > There are both advantages and disadvantages -- it all depends on the use
> > > case and it all depends on what you expect from a block device...
> > > I do not expect more than the underlying physical device's raw
> > > performance,
> > 
> > I haven't looked at the code, but can you explain how this differs from
> > the "sync" option in nbd-server?
> 
> It differs, a lot.
> What the well known nbd-server does is a loop of:
> 
>   1) read a request from the socket
>   2) issue the request to the "disk"

2a) if the "sync" option is on, call a sync function (fsync(),
fdatasync(), or sync(), not sure which one is used, doesn't really
matter for the purpose of this discussion)

>   3) send it back to the client, goto 1)
> 
> The only thing that makes it work fast is that the "disk" is in fact a
> buffer cache.

By default, yes. Since nbd-server is most often used as a _read-only_
device for netbooted clients, that means doing what's best,
performance-wise, is obviously the best choice. I understand that
sometimes it's not the best choice, that's why the sync option is there.
I realize that performance could be better, but implementing it so that
it still has all the features it has today is something I've never had
enough time to work on.

> I do not like the idea of a block device server being in fact a
> write-back cache for the underlying physical device, ignore my reasons
> for it, perhaps I wanted to try something new and do some research on
> the protocol itself, perhaps I was bored, it does not matter.

Yes, well.

Every time someone reimplements nbd-server, I die a little inside.
You're not the first one to do that, and you'll probably not be the last
one, either; I'm aware of at least 7 other implementations, besides the
reference implementation. That's just a waste of effort.

No, nbd-server isn't the best code ever. I know that. And no, the nbd
protocol isn't very difficult either; it's probably possible to
implement a basic nbd server in half an afternoon. That doesn't mean you
need to do so, though. Patches are (and have always been) welcome.

[...]
> What bnbd-server does is also "a loop" of:
> 
>   1) read as many requests from the socket as possible
>   2) submit as many AIO requests as possible to the disk via io_submit()
>   3) collect as many AIO replies as possible via io_getevents()
>   4) sent as many replies as possible via writev()

This is pretty close to what I've been wanting to do for nbd-server,
too. "Pretty close", as in, AIO isn't necessarily as
portable as plain read() and write(); but multithreading and writev()
and friends is certainly something I've looked into in the past. It
hasn't happened yet, mainly due to "lack of time" etc, but that doesn't
mean I'm not interested in it.

> > > I do expect that buffer cache flushes on the client side
> > > actually hit at least the underlying device's write cache (nearly sync
> > > writes on the server side),
> > 
> > With the FUA patch (which finally has been accepted into Linux proper),
> > this will happen; when the client side does an explicit buffer cache
> > flush, this will be finished with an FUA call, which will cause an
> > fsync() on the server side.
> > 
> > The effect will be the same, except that the system will be better,
> > performance wise.
> 
> We shall see...
> Still, what about e.g. AIO on top of NBD devices?
> I thought the FUA was dropped, FLUSH was introduced, is that what you
> meant?

Yes, sorry for the confusion.

[...]

-- 
Copyshops should do vouchers. So that next time some bureaucracy requires you
to mail a form in triplicate, you can mail it just once, add a voucher, and
save on postage.