Re: [Nbd] NBD_CMD_FLAG_FUA?

[Alex Bligh <alex@...872...>]

Folkert,

> > You can't "perform" an NBD_CMD_FLAG_FUA, it's saying that the
> > command to which it is attached (which, as per above, is NBD_CMD_WRITE
> > only at the moment, though we'd also want to allow it on the
> > recent NBD_CMD_DISCARD), you treat the command as having "force
> > unit access" set, which means you should not acknowledge the
> > command as complete until the data is written to the physical disk. IE
> > it has the same semantics as the Linux block layer flag.
>
> But isn't it the same as sending a NBD_CMD_FLUSH after a write/discard?
>
> (thanks for the reply by the way!)

No. NBD_CMD_FLUSH ensures ALL previous commands are sent to disk. That
could be quite a lot. It thus has ordering implications.

The semantics are taken from the Linux kernel. See
 Documentation/block/writeback_cache_control.txt
in the kernel sources for more examples.

The text below is a file I wrote for the kernel documentation (patch
not accepted), but might be helpful.

--
Alex Bligh


Ordering requirements for request-based drivers
===============================================

Request based drivers may handle requests in any order. This means:

a) the requests may be handled (i.e. read/written from the device
  in any order)

b) the completions may be done in any order (which may be different from
  the order in which the requests were submitted and/or from
  the order the requests were handled).

This is subject ONLY to the restriction that a REQ_FLUSH command
may not be completed until all writes that were completed prior
to the REQ_FLUSH being received have been written to disk. It is
not strictly necessary to ensure that writes received after
the REQ_FLUSH but completed prior to it are also written to
disk, but this is advisable. Note that request based drivers
will only receive a REQ_FLUSH on an empty request (i.e. with
no associated data).

REQ_FUA has no ordering implications.

This has the following results:

1. If a write is submitted to a driver covering block X, and
  a second write is submitted also covering block X prior to
  completion of the first write, the driver  may disorder the
  writes, meaning that the value of block X on disk
  cannot be determined.

2. If a read is submitted to a driver covering block X, and
  a write is submitted also covering block X prior to the
  completion of the read, the value of block X in the read
  cannot be determined.

3. If a write is submitted to a driver covering block X, and
  a read is submitted also covering block X prior to the
  completion of the write, the value of block X in the read
  cannot be determined.

Neither REQ_FLUSH nor REQ_FUA represent a barrier. Thus note the
following:

1. A write issued after a REQ_FLUSH may be written to disk
  before the REQ_FLUSH is completed, irrespective of whether
  the completion is received prior to or after the
  completion of the REQ_FLUSH.

2. A write issued before a REQ_FLUSH may not be written to
  disk if it is not completed before the REQ_FLUSH is
  completed (though drivers should avoid this behaviour).

3. A write with REQ_FUA issued subsequent to a REQ_FLUSH
  may be performed prior to the flush going to disk.

4. Writes with REQ_FUA set may be disordered just as any
  other writes can be disordered.

You should not rely on the kernel sending your driver REQ_FLUSH.
Specifically:

1. You will not receive a REQ_FLUSH during filing system operation
  unless the filing system itself supports it.

2. There is no guarantee that you will receive a REQ_FLUSH on a umount()
  of a device.

3. There is no guarantee you will receive a REQ_FLUSH when your device
  ceases to be used, is unplugged, etc. - it is up to you to
  ensure data is appropriately flushed.