On Sun, Mar 24, 2019 at 11:17:54AM -0500, David Wright wrote: > On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote: > > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote: > > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote: > > > > > Reading the OP's problem, I wonder how you're meant to detect > > > "any whiff of a problem" [...] > > > > Torture tests. > > Like, multiply the number of sources by stealing a few more radio > scanners to connect up, which then all burst into life as the > police scour the neighbourhood for thieves? Tests require some modeling and phantasy. And some knowledge of the environment. Otherwise I'd call it "futzing around" -- still important, but at an earlier stage. We don't know OP's data sources; for modeling, one should have a couple of parameters, like expected average bandwidth, expected peak bandwith wrt interval and so on. For example, if data is coming in via serial 115kb/s or, say USB (which one?), we'd have the input bandwith. Knowing something about the process's digestion would tell us how much output it will produce for a given input. And so on. I've been doing some industrial data acquisition stuff. You end up doing this kind of estimations all the time: you don't want exceptional conditions to throw your systems into confusion -- you want to have an idea on "how they fail". And then test, test and test, to know where you made bad assumptions. "Stealing radio scanners" would be definitely one possibility. I'd rather try to simulate them in software -- better for the nerves :-) For that, you'd have to have an idea about the interface the radio scanner has to your computer. Serial? (I strongly guess, but can't know ;-) We know a bit about the output: "20 to 30 lines per second". Assuming roughly 60 - 80 chars per line, that'd make 1200 to 2400 char/s, so roughly 12 - 14 kb/s. That's way below of what a serial interface handles. Pentium 90s did that under Linux back then, even a bunch of interfaces at a time. You had to put a bit of care into it, though. > When dealing with realtime real information coming in, over which you > have no control, it can be non-trivial to set up such scenarios. > That's why I thought it best to devise a method that's more > efficient than line buffering. After all, that's why buffering > was invented, wasn't it. Buffering is no magic: it's the tradeoff you have to make between bandwith, latency and loss granularity (i.e. how much you're willing to lose, should your "buffering process" fail to work properly). Also, just saying "buffering" here is confusing, since there are usually several levels of buffering at work: in the present case there is - in-process buffering (provided by the libraries (here Perl, the one you control with $| resp. ->autoflush()) which goes the way of the do-do if your process loses control: signal, memory corruption, whatnot) - OS level buffering (the one controlled with sync and friends), which offers somewhat stronger guarantees (because your OS kernel --ahem!-- never loses control [1]). It comes down to understanding what's going on, more or less. Given the few data points we have, I'm pretty confident in saying "write away, trust your hardware/OS, but watch out for possible problems. Make your back-of-the-envelope calculations and run a few tests to validate them". Your proposed "catching of signals" sounds elegant, but know that it'll only protect you during fair weather. For a SIGHUP or SIGUSR1 it'll be fine, for a SIGSEGV or a SIGILL (which is telling you that your process might be highly confused: perhaps the pointer to the buffer you're trying to flush points to something completely different!) it might be... not that fine. It's a bit like doing backups: you'd better done that earlier :-D Cheers [1] Database folks are a bit more restrictive than that and want to perform well even when the OS loses control. One can learn quite a bit from them. -- tomás
Attachment:
signature.asc
Description: Digital signature