Re: [Nbd] Question about the expected behaviour of nbd-server for async ops

[Goswin von Brederlow <goswin-v-b@...186...>]

Alex Bligh <alex@...872...> writes:

> I think you are misunderstanding the problem. A client sends a normal
> command, followed by a disconnect (which is legal). The server then
> accepts the normal command, processes it, and sends a reply (by which
> I mean "does a write() to the socket". The server then processes the
> NBD_CMD_DISC, which does a close() on the socket and exits the
> process. My concern is that this does not necessarily mean that the
> data enqueued in the socket buffer is in fact sent on the wire.
>
> This does not affect the current client in normal operations because
> it does not send the NBD_CMD_DISC until all replies have been received;
> however, this is a matter of observation, and not of coding guarantee.
>
> All I am saying is we should take action to ensure we do send any
> queued replies (queued here mean "processed and in the socket's
> SNDBUF") prior to any action which might cause them to be junked.
>
> If you look at Stephens UNIX Network Programming p202, on SO_LINGER,
> there are 3 possible behaviours.
>
> 1. l_onoff is 0: l_linger is ignored, close returns immediately
>   and the server wil try to deliver the data to the peer.

Which I believe is the default.

> 2. l_onoff is nonzero, and l_linger is zero. TCP aborts the connection
>   when it is closed: ***that is TCP discards any data still remaining
>   in the socket send buffer and sends an RST to the peer*** (this
>   is what we should avoid).

This would break things.

> 3. l_onoff is non-zero and l_linger is zero: the kernel lingers, i.e.
                                      ^^not^^
>   close blocks until either the data is sent and acknowledged or the
>   linker time expires (which one happens is returned in the error code).
>   (this is OK provided we use a sufficient linger time).

This would make the server quit a bit slower and probably make more
sense.

My concern here is what happens after the linger timeout? Does close
return and the kernel keeps sending the data in the background? Or does
it discard any data it didn't manage to send?

> That implies that l_onoff=0 l_linger=0 would be fine. However, under
> (e.g.) SVR4, close() can lose data. See (e.g.):
>
> http://tinyurl.com/odlj5
>> Closing a socket: if SO_LINGER has not been called on a socket, then
>> close() is not supposed to discard data. This is true on SVR4.2 (and,
>> apparently, on all non-SVR4 systems) but apparently not on SVR4; the use
>> of either shutdown() or SO_LINGER seems to be required to guarantee
>> delivery of all data.
>
> In general terms it is not sufficient to rely solely on close() to
> send data in portable programs.

I got that sense from googling about it too.

>>>> Here is what i think should happen:
>>>> - on recieving a NBD_CMD_DISC request you shutdown(fd, SHUT_RD)
>>>> - process and reply any pending requests
>>>> - fsync() /* implicit flush, just to be nice */
>>>> - shutdown(fd, SHUT_WR)
>>>> - close(fd)
>>>> - exit()
>>>
>>> That alone doesn't help (I am not sure we do the shutdown but
>>> it might be an improvement).
>
> I missed the shutdown(SHUT_WR) here - that will indeed do what is
> required.
>
> What I was saying is the shutdown(SHUT_RD) is unnecessary
> and insufficient (see below for why).

It stops the client from sending commands after a DISC. The client will
get an error on send if it tries. That might help find the cause of such
illegal behaviour in a client. I don't think it can hurt.

> I presume you mean fsync() the backing store (as opposed to the
> socket, here) - yes, I think that's a good idea. So I would do:
> 	fsync(backing store)
> 	shutdown(socket, SHUT_WR)
> 	close(fd)
> 	exit()
>
>> It (or close) blocks with SO_LINGER set or goes into background
>> otherwise. Also lingering is allways done on exit.
>
> I can't find a reference to the assertion that there is automatic lingering
> on a process exit on all platforms. My understanding is that exiting
> a process does an implicit close on fds, and that's all, but I migh
> be wrong.

I can't back say that it will block on all platforms. And it probably is
irelevant since we, as good programmers, do a close of the socket before
exiting, which I think invalidates the behaviour anyway.

>>> From all google can find me the kernel will still try to send any
>> remaining data in the outgoing socket buffer till the SO_LINGER timeout
>> or tcp timeout kills the socket for good. There seem to be no way to
>> "wait for all data to be send" on a socket prior to closing it.
>
> Sure, but nbd-server is a portable program.

I didn't see any indications different kernels would behave
differently. But aparently SVR4 does and requires a shutdown prior to
close. I think the combination of shutdown+close is the best one can do.

MfG
        Goswin