Re: One new nbd-runner project to support the Gluster/Ceph/Azure, etc
On Fri, Apr 12, 2019 at 01:54:16PM +0800, Xiubo Li wrote:
> On 2019/4/12 3:50, Wouter Verhelst wrote:
> > Hi,
> >
> > Sorry about the late reply; I recently moved continents, and am still
> > getting used to my new life...
>
> Hi Wouter
>
> Interesting and no worry :-)
>
> Thanks very much for your feedback and great ideas.
>
>
> > On Mon, Mar 25, 2019 at 09:01:30PM +0800, Xiubo Li wrote:
> > > Hi ALL,
> > >
> > > The NBD is one great project :-)
> > >
> > > Currently there are many Distributed Storages projects such as the Gluster and
> > > Ceph are widely use, but they couldn't support NBD directly. For example, for
> > > the Gluster, if we want to export the volume file as one NBD block device we
> > > must mount the volume first and then use the nbd utils to export it again,
> > > which a little ugly.
> > >
> > > To make it more efficient, we also need one common and generic utils, something
> > > like [1], to support them directly. And good news is that I have a working code
> > > with most basic things @nbd-runner project[2]. It is hosted under the gluster/
> > > currently and I'd like to request a repository under [3] to host this.
> > I'm not necessarily opposed to this, but it would be nice to understand
> > how your nbd-runner works, and is different from, say, nbdkit? It seems
> > to me that they have similar goals.
>
> Yeah, they have similar goals, but there also have many differences and
> there has server limitation for use to use the great nbdkit in some cases.
>
> For the nbd-runner we just want it works like a C/S utils and there will be
> one systemd nbd-runner service to handle the backstore stuff, and the
> nbd-cli will support the create/delete/map/unmap/list commands. Then there
> is no need for users to care about the Backstore storage stuff, once the
> nbd-runner is fired what they need to focus on is in the Client side. And
> the nbd-cli in will parse all the options.
Cool.
What protocols do the two talk? Do they use the NBD negotiation as
defined in the proto.md document, or do they speak some custom thing?
If a client that isn't nbd-cli (e.g., qemu) tried to connect to
nbd-runner, would it be able to, or would that not be possible?
> The 'create/delete' commands will help to create/delete the devices/files in
> the backstore pools/volumes. And the 'map/unmap' commands will do the
> backstore devices/files <--> /dev/nbdX mapping/unmapping stuff. And the
> 'list' command here will do like the linux 'ls' command.
>
> For the Gluster/Ceph/Azure handlers we need to create/map hundreds of blocks
> in several storage pools/volumes in any time when needed, such as in the
> Openshift when any new pods are being created.
>
> We also need one easy utils, like the iscsi utils, to help us remap the
> /dev/nbdXX <---> backstore device/file when the nbd-runner daemon or the
> node running the nbd-runner is restarted.
nbd-client actually has that ability:
- nbdtab allows you to predefine device nodes
- the -persist option is meant to reconnect a device when the connection
drops (however, this ability does not work very well, because the
kernel API is not well defined).
The same is currently not true for the server. It is one of my long-term
goals to be able to change server configuration at run time, without a
restart. Currently you can only add devices, not remove them -- it's
something I want to work on, but I'm not paid to do this, so don't have
time to do the refactoring that nbd-server urgently needs...
> That means we need to save the backstore device infos into one config
> file then reload them when restarting. This is very similar to what
> the LIO/targetci does. So we need one client daemon to do the ping and
> remap stuff, just like the iscsid, it is very easy to support this in
> nbd-runner project, and this is to be done.
>
> Above all we want the great NBD could work as easy/simple as the current
> iscsi utils does in the Openshift.
Sure. iSCSI is kindof complicated when compared to NBD though ;-)
--
To the thief who stole my anti-depressants: I hope you're happy
-- seen somewhere on the Internet on a photo of a billboard
Reply to: