Re: Allowing > 32 bit lengths for NBD_CMD_TRIM, NBD_CMD_WRITE_ZEROES

[Wouter Verhelst <w@uter.be>]

On Thu, Sep 06, 2018 at 12:13:23PM +0100, Richard W.M. Jones wrote:
> On Thu, Sep 06, 2018 at 12:55:49PM +0200, Wouter Verhelst wrote:
> > > (4) Allow a different request message format, differentiated by using
> > > a different magic (! NBD_REQUEST_MAGIC).  We could use the opportunity
> > > to widen a few fields and reserve space for future expansion.
> > > Similar to (1).
> > 
> > I was originally opposed to this, but have since come to realize that
> > that (being opposed) may have been a mistake. We probably should have
> > used the opportunity to change the request format when we changed the
> > response format, too.
> > 
> > It's not too late to introduce a request format change, though. In that
> > case, I would suggest:
> > 
> > - magic (32 bit value): different magic number, indeed.
> > - request length (32 bit value): length of header and data (I don't
> >   think it makes sense to have requests that are >4GiB, but could be
> >   persuaded otherwise).
> 
> I think the way to think about this is in terms of time rather than
> request size.  How long does it take to transmit a 4GB or larger
> request, given current & expected future improvements in network
> performance?
> 
> I would say if it would always be longer than [number picked at
> random] 20 ms then limiting it to 4GB is fine, because otherwise the
> TCP connection is blocked for too long while one request is being
> transmitted causing interactivity problems.
> 
> This actually ties in nicely with another conversation I was having
> recently.  I was promoting NBD to an iSCSI user and they said
> something to the effect of “but but but with iSER [iSCSI Extensions
> for RDMA] we get awesome performance and NBD can't match that”.  Which
> is quite likely to be true.  It would be nice to provide a
> standardized RDMA transport for NBD.
> 
> If we assume that NBD over RDMA was a thing, and you have the latest
> and most expensive 12x XDR Infiniband hardware, and we were able to
> saturate the link, then 4 GB could be transmitted in 13 ms!  That's
> less than my arbitrarily picked limit above :-)
> 
> (Agree with the rest of your email and the limits proposed.)

That's a good point, and I didn't think of it. Okay, so longer ranges
*can* make sense, in some cases, but then I would suggest we don't allow
structured requests without also block sizes, so that the server isn't
expected to wait for 2^64 - 1 bytes of data...

So that would make the format be like this:

- magic (32)
- request length (64)
- header length (8)
- flags (16)
- command (16)
- from (64)
- affected length (64)

Would it make sense to introduce more padding so that 64-bit values
are 64-bit aligned?

That would be:

- magic (32)
- header length (8)
- padding (24)
- flags (16)
- command (16)
- padding (32)
- request length (64)
- from (64)
- affected length (64)

I'm inclined to say no, because if we introduce so many padding bytes,
it becomes ridiculous. But then we probably should drop the padding
bytes after the header length, too...

-- 
Could you people please use IRC like normal people?!?

  -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
     Hacklab