Re: Any tips or documentation for writing a new Curl_handler (protocol)?
On Wed, Jul 11, 2018 at 02:09:02PM +0200, Wouter Verhelst wrote:
> Hi all,
>
> On Wed, Jul 11, 2018 at 12:18:39PM +0100, Richard W.M. Jones wrote:
> > [Adding NBD mailing list to CC]
> >
> > On Wed, Jul 11, 2018 at 08:43:24AM +0200, Daniel Stenberg wrote:
> > > On Tue, 10 Jul 2018, Richard W.M. Jones wrote:
> > >
> > > >I was wondering if you have any documentation, or general tips,
> > > >for writing a new handler? I was thinking of implementing an NBD
> > > >(Network Block Device) client.
> > >
> > > I'm afraid we don't have much docs for that! :-( I suppose mostly
> > > because it doesn't happen very often.
> > >
> > > I think the best way to do it is to take a look at how other
> > > handlers are written, like perhaps the smb support would be similar
> > > in style and spirit?
> >
> > After sending this I worked it out by looking at previous commits of
> > backends and the code (for SMTP and others). My ongoing difficulty is
> > understanding the non-blocking model.
> >
> > > Does NBD actually have a URL syntax defined?
> >
> > Not officially. Whatever URL syntax defined needs to be able
> > to specify:
> >
> > - hostname + optional port
> > - OR Unix domain socket
> > - export name
> >
> > In libguestfs we defined our own URL syntax:
> >
> > http://libguestfs.org/guestfish.1.html#a-nbd:-example.com-:port
> >
> > but it's not completely satisfactory since the export name is a free
> > text string and does not necessarily begin with a '/' character.
>
> I've been pondering for a while now that it might make sense to have an
> official URL syntax defined, too. Qemu has a URL syntax, libguestfs has
> one, and I believe dracut does too; but TTBOMK, none of these are in
> sync.
>
> This is at the very least annoying for people who use all three, and we
> should look at fixing that.
>
> > > What's the use case for an application to use this?
> >
> > This is a very good question. My need is for a C client library for
> > accessing NBD servers. I don't believe that one exists.
>
> None exist that I know of, no.
>
> I've been thinking of writing one, but for a long time the protocol was
> simple enough that it didn't seem worth it (and honestly, nbd-client
> wouldn't use it, since I'm not adding a library to the kernel ;-).
>
> We've recently extended the protocol by quite a while, so it may be
> worthwhile to do so now.
>
> > There is plenty of C code for accessing NBD servers (eg. in qemu) but
> > it's encapsulated in other programs and not easy to separate.
> >
> > My idea was to use libcurl instead of writing a new library. The
> > advantage is that curl gives us ‘for free’ a command line tool,
> > multiple language bindings, and widespread availability.
> >
> > This is all relatively uncontroversial I hope.
> >
> > However the complicated bit is that we will eventually need to do more
> > than just simple read/write of blocks of data. The NBD protocol[1]
> > supports other features such as:
> >
> > - discard blocks
> > - efficiently write zeroes
> > - negotiate block size
> > - list exports
> > - query sparseness
> > - TLS with certificates or PSK
> >
> > At some point we'd want to support as many of those features as
> > possible. Curl supports non-orthogonal features for other protocols
> > (eg. FTP active port), so I suppose it's not impossible to extend Curl
> > for NBD, but you may feel that you don't want to do that.
> >
> > Anyway I think even a basic read/write backend is a worthwhile
> > addition at the moment.
>
> Personally, I'm not entirely sure that libcurl really maps well to NBD,
> tbh, since NBD is not a "one request per URL" protocol like most other
> libcurl-supported protocols currently are (at least for the most part).
>
> But I have to admit that I'm not too familiar with the libcurl internals
I believe we're OK here. As long as I write the NBD Curl handler so
that it holds open the NBD connection between READFUNCTION/
WRITEFUNCTION callbacks.
In other words, it acts somewhat like HTTP keepalive.
(Of course most people will be more familiar with the curl command
line tool, and in that case a single use of the curl command line tool
will correspond to a single NBD connection, roughly speaking).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
Reply to: