Re: Status flags definition for NBD_REPLY_TYPE_BLOCK_STATUS
> On Sat, Jul 27, 2024 at 08:02:35PM GMT, Connor Kuehl wrote:
> > > 32 bits, length of the extent to which the status below
> > > applies (unsigned, MUST be nonzero)
> > > 32 bits, status flags
> >
> > I can't seem to find the definition for these status flags. It's likely
> > I may have just missed it. Any pointers will be much appreciated.
>
> The status flags are defined per-metacontext. The spec defines the
> "base:allocation" metacontext:
>
> https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md#baseallocation-metadata-context
>
> which in turn defines flag 0x01 for NBD_STATE_HOLE (clear by default
> or if the extent is known to be allocated, set only if the extent is
> known to be unallocated), and flag 0x02 for NBD_STATE_ZERO (clear by
> default or if the extent is known to contain non-zero data, set only
> if the extent is known to read as zero).
>
> If you are implementing NBD_BLOCK_STATUS on top of files,
> lseek(SEEK_HOLE) can be used to set NBD_STATE_HOLE|NBD_STATE_ZERO in
> parallel (either both bits are clear or both bits are set). But there
> are some other storage mechanisms where the distinction between bits
> is meaningful (for example, iSCSI has the notion of some hardware that
> supports the notion of allocation but does not promise whether
> unallocated regions read as zero; conversely, qcow2 files have a
> notion of regions of a disk image that read as zero but which are
> already allocated so that writing to those regions won't expand the
> qcow2 file size).
>
> There are other defined metacontexts linked from the NBD spec; in
> particular, QEMU defines two metacontexts at
> https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/nbd.txt
>
> where "qemu:dirty-bitmap:XXX" defines 0x01 to be set if an extent is
> known to be dirty in the bitmap named XXX, and where "qemu:allocation"
> is an integer representing the depth of where the allocation comes
> from (0 for unallocated, 1 for the current image, 2 for the first
> backing layer, 3 for the second backing layer, and so forth).
Aha! Thank you very much for the doc reference and the detailed
explanation. My main goal was to provide a higher level abstraction for
interpreting and using the values returned by the BLOCK_STATUS command.
In any case, I can still define these as constants to make life easier
for the library's clients, especially if they are primarily interested
in "base:allocation" and "qemu:dirty-bitmap", as I am.
> There may be other NBD metacontexts out there, but so far, none of
> them have submitted a patch to be included in the mention from the NBD
> spec. One proposal that I have considered (but have not actually
> coded up) would be a series of NBD metacontexts allowing the ability
> to expose Zoned Block Devices (ZBD,
> https://zonedstorage.io/docs/linux/overview) across NBD; in
> particular, this would include metacontexts that reveal information
> such as the current 64-bit append location within a given zone
> (getting 64-bit block status would require the use of extended
> headers, currently mentioned as experimental in spec but currently
> implemented by qemu and libnbd).
Very interesting! As mentioned above, "base:allocation" and
"qemu:dirty-bitmap" are top of mind for me today, but I'll check up on
ZBD just out of curiosity. For now, I am targeting the base NBD
protocol, but I may end up implementing the documented extensions.
> For what it's worth, the nbdkit project can be used for testing
> interoperability with your client implementation (among other things,
> nbdkit provides it easy to write a shell script server that advertises
> various block status combinations for "base:allocation" for testing
> how your client reacts). To date, no one has contributed patches to
> nbdkit to support the emulation of any NBD metacontext other than
> "base:allocation", but if it would help integration testing, that may
> be a worthwhile feature to add to nbdkit.
> https://libguestfs.org/nbdkit.1.html
Yes! I was planning on using nbdkit as the basis of my integration
testing.
> I can also point you to the libnbd project that may serve as a way to
> incorporate an existing client implementation into your project rather
> than writing a client from scratch.
> https://libguestfs.org/libnbd.3.html
On the one hand I'd love to use an off-the-shelf, battle-tested, and
mature implementation, but on the other hand I wouldn't learn nearly as
much! lol
I did seriously consider it, but I am writing my client implementation
in the Go programming language, and I would like to avoid a CGO/FFI
dependency since those tend to complicate multi-platform builds for the
usual suspects, adds a system dependency to the deployment target, and
Go's runtime behavior when interacting with the FFI is somewhat strange
and slow.
There are some other Go projects in this space, but the ones I have
found all seem to be some combination of:
* Abandoned and implements only one or two parts of the client protocol,
mainly just opening a TCP connection and one or two NBD option verbs
* Focused exlusively on creating NBD servers, or /dev/nbdX devices
* Depend on CGO/FFI
All that said, I have basically implemented the whole thing already, and
I am just working on polishing it up and adding automated unit testing.
I suspect the last 5% of the project may be the longest though, since I
need to go through it carefully and make sure it behaves the way the
protocol says it should behave (for example, client MUST initiate a hard
disconnect when server does X.)
Anyways, thanks again! I really appreciate your insights.
Sincerely,
Connor
Reply to: