on binary logs (was: Re: /bin/sh)

[Gergely Nagy <algernon@balabit.hu>]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

>> I have no idea why people assume that a binary format means it can only
>> be processed with a special, proprietary tool. Binary simply means what
>> it means, binary and not text which means it's a more stream-lined and
>> machine-readable format as opposed to a text format with no formatting
>> at all.
>
> It means that it works completely differently from every existing Unix
> log parser on the planet.  syslog is hardly "no formatting at all".

Please allow me to disagree. syslog *is* the very definition of no
formatting at all. The single most interesting part of it, the message
is free form text. Good luck making sense of that. There is no common or
widely accepted way of writing syslog messages. There's the SDATA thing,
but hardly anyone uses that, and that is primarily for additional info:
the message itself is still free form chaos.

>> And, when it comes to processing, binary data is actually *easier* to
>> process. Everyone who has ever written a text parser themselves will
>> agree.
>
> I have written several, and I still prefer plain text.  I want to use
> the same tools to parse my logs that I have used for years, like
> logcheck.  Text files is the Unix way.

Except text files are bad when you want to search, index, classify or do
anything more complicated than grep or $PAGER. How many of the log
analytics / log browser things use text files? I know of none, and it is
no wonder. Text files are awful when you need random access, fast.

-- 
|8]