Re: Flushing all Buffers Before Exiting
On Mon 25 Mar 2019 at 10:47:08 (+0100), tomas@tuxteam.de wrote:
> On Sun, Mar 24, 2019 at 11:17:54AM -0500, David Wright wrote:
> > On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> > > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> > >
> > > > Reading the OP's problem, I wonder how you're meant to detect
> > > > "any whiff of a problem" [...]
> > >
> > > Torture tests.
> >
> > Like, multiply the number of sources by stealing a few more radio
> > scanners to connect up, which then all burst into life as the
> > police scour the neighbourhood for thieves?
>
> Tests require some modeling and phantasy. And some knowledge of
> the environment. Otherwise I'd call it "futzing around" -- still
> important, but at an earlier stage.
>
> We don't know OP's data sources; for modeling, one should have
> a couple of parameters, like expected average bandwidth, expected
> peak bandwith wrt interval and so on. For example, if data is
> coming in via serial 115kb/s or, say USB (which one?), we'd have
> the input bandwith. Knowing something about the process's digestion
> would tell us how much output it will produce for a given input.
> And so on.
>
> I've been doing some industrial data acquisition stuff. You end
> up doing this kind of estimations all the time: you don't want
> exceptional conditions to throw your systems into confusion --
> you want to have an idea on "how they fail". And then test, test
> and test, to know where you made bad assumptions.
>
> "Stealing radio scanners" would be definitely one possibility.
> I'd rather try to simulate them in software -- better for the
> nerves :-)
>
> For that, you'd have to have an idea about the interface the
> radio scanner has to your computer. Serial? (I strongly guess,
> but can't know ;-)
>
> We know a bit about the output: "20 to 30 lines per second".
> Assuming roughly 60 - 80 chars per line, that'd make 1200 to
> 2400 char/s, so roughly 12 - 14 kb/s. That's way below of
> what a serial interface handles. Pentium 90s did that under
> Linux back then, even a bunch of interfaces at a time.
>
> You had to put a bit of care into it, though.
Sure, but I didn't want to presume upon the OP. That can be a lot of
effort. (I too have worked on data acquisition with simultaneous
machine control. running on far less flexible OSes.)
> > When dealing with realtime real information coming in, over which you
> > have no control, it can be non-trivial to set up such scenarios.
> > That's why I thought it best to devise a method that's more
> > efficient than line buffering. After all, that's why buffering
> > was invented, wasn't it.
>
> Buffering is no magic: it's the tradeoff you have to make between
> bandwith, latency and loss granularity (i.e. how much you're willing
> to lose, should your "buffering process" fail to work properly).
>
> Also, just saying "buffering" here is confusing, since there are
> usually several levels of buffering at work: in the present case
> there is
>
> - in-process buffering (provided by the libraries (here Perl,
> the one you control with $| resp. ->autoflush()) which goes
> the way of the do-do if your process loses control: signal,
> memory corruption, whatnot)
>
> - OS level buffering (the one controlled with sync and friends),
> which offers somewhat stronger guarantees (because your OS
> kernel --ahem!-- never loses control [1]).
>
> It comes down to understanding what's going on, more or less.
> Given the few data points we have, I'm pretty confident in
> saying "write away, trust your hardware/OS, but watch out for
> possible problems. Make your back-of-the-envelope calculations
> and run a few tests to validate them".
>
> Your proposed "catching of signals" sounds elegant, but know that
> it'll only protect you during fair weather. For a SIGHUP or
> SIGUSR1 it'll be fine, for a SIGSEGV or a SIGILL (which is telling
> you that your process might be highly confused: perhaps the
> pointer to the buffer you're trying to flush points to something
> completely different!) it might be... not that fine. It's a bit
> like doing backups: you'd better done that earlier :-D
>
> Cheers
>
> [1] Database folks are a bit more restrictive than that and
> want to perform well even when the OS loses control. One
> can learn quite a bit from them.
I can see why the most important criterion with a database might be
protecting it from any sort of corruption, but I don't think we're
dealing with that here.
In my experience of these sorts of problems, the most important
criterion was not to waste the precious samples being analysed,
which meant preventing the OS from ever losing control. Again,
I don't think we're dealing with that here.
So these are issues beyond the scope of the OP's problem domain.
OTOH, IMO, trapping a USR1 signal before terminating the program
and, say, turning off the scanner, *is* within the scope of the
OP. I just don't understand why one needs to escalate it into
trapping *all* signals and the associated complications with
managing that.
Cheers,
David.
Reply to: