Re: [Nbd] [PATCH 4/4] nbd: add a nbd-control interface

[Alex Gartrell <agartrell@...2204...>]

On 1/25/17, 6:23 AM, "arndbergmann@...17... on behalf of Arnd Bergmann" <arndbergmann@...17... on behalf of arnd@...2332...> wrote:
> On Wed, Jan 25, 2017 at 2:47 PM, Josef Bacik <jbacik@...2204...> wrote:
> > On Tue, Jan 24, 2017 at 2:11 AM, Greg KH <gregkh@...1299...> wrote:
> >> On Mon, Jan 23, 2017 at 10:52:42AM -0500, Josef Bacik wrote:
> >>>  On Mon, Jan 23, 2017 at 10:03 AM, Greg KH <gregkh@...1299...>
> >>>  Loop accomplishes this with the /dev/loop-control
> >>>  and an ioctl.  Then we also need a way to figure out what is the first
> >>>  /dev/nbd# device that isn't currently in use in order to pick the next
> >>> one
> >>>  to configure.  Keep in mind that all of the configuration for /dev/nbd#
> >>>  devices is done through ioctls to those devices, so having a ioctl
> >>> interface
> >>>  for the control device is consistent with the rest of how NBD works.
> >>
> >>
> >> But adding a random char device node and using ioctls on it is different
> >> than using an ioctl on an existing block device node that is already
> >> there.
> >>
> >> So what _exactly_ do you need to do with this interface?  What data do
> >> you need sent/received to/from the kernel?  Specifics please.
> >
> >
> > I need to say "Hey kernel, can you setup some new nbd devices for me?" and
> > "Hey kernel, what is the first free nbd device I can use for this new
> > connection and (optionally) create a new device for me if one does not
> > exist?"  Loop accomplished this (in 2011 btw, not some far off date) with
> > /dev/loop-control.  Btrfs has it's scanning stuff controlled by
> > /dev/btrfs-control.  Device mapper has /dev/mapper/control.  This is a well
> > established and widely used way for control block based device drivers.
> 
> We have multiple established ways to deal with this kind of problem, the most
> common ones being a separate chardev as you propose, a sysfs interface and
> netlink.
> 
> They all have their own set of problems, but the needs of nbd as a network
> storage interface seem most closely resemble what we have for other network
> related interfaces, where we typically use netlink to do the setup, see e.g.
> macvtap as an example for a network chardev interface.
> 
> Can you have a look if that would solve your problem more nicely?

FWIW, I'm the one consuming this nbd stuff at Facebook and bringing
netlink into this would be a huge pain for me.  It's very weird to
need to pull sockets in and then drop back to ioctls from a design
perspective, and future consumers of this API would be sad as well.
This is compounded, I think, by the fact that the breadth of
functionality here doesn't really merit a shared library for everyone
to use -- could you imagine libnbdcontrol, which exposes a single
"get_next_device" function?

If nbd were *all* netlink I think that that'd be fine, but you'd have
problems implementing the NBD_DOIT function in that fashion.  So I'd
rather stick to the char device ioctl thing because it's more
consistent with the old NBD stuff as well as the loop device stuff.

-- 
Alex Gartrell <agartrell@...2204...>