Re: Unifying logging by default
On Wed, Feb 20, 2019 at 02:19:02PM -0800, Russ Allbery wrote:
> Josh Triplett <josh@joshtriplett.org> writes:
> > While there are *absolutely* configurations in which system
> > administrators want to log to arbitrary locations and files, I would
> > like to propose that for consistency we should configure software to
> > unify logging into syslog and/or journald by default. In particular, I'd
> > propose that daemons and applications should default to logging to
> > syslog (working with any installed syslog implementation providing
> > /dev/log), and that daemons run via systemd units may detect or be
> > configured to log to stdout/stderr which will be wired to the sysadmin's
> > preferred log target.
>
> I've seen three main reasons why packages do not use syslog:
>
> * The log data is not line-structured. Example: /var/log/apt/history.log
> Dumping that as-is into syslog sounds like an awful idea. This would
> therefore block on an upstream feature request to change log formats
> first, with all that entails.
Both syslog and journald support multi-line log messages; I'd *love* to
see /var/log/aptitude and /var/log/apt/history.log end up in syslog or
journald.
I'd also have no problem with an exception explicitly saying that
something like /var/log/apt/term.log can continue to use a standalone
file iff there isn't a good solution for storing that in a structured
log message. (And even that *could* go into a structured log message
with some care.) I'm trying to handle the common case here.
> * Performance. Good examples here are Apache and INN, both of which have
> per-request logs that are intentionally not line-buffered and are logged
> directly to files, since losing some log messages is not as much of a
> concern as performance and disk I/O. I've tried to reroute things like
> that through syslog in the past, only to find syslog now taking a
> substantial portion of all available system resources trying to keep up.
Relatively few packages are going to be performance-bound on logging,
and those that do could absolutely recommend logging to standalone files
for scalable configurations, or even have configuration packages that
make it trivial to configure that destination. That said, Apache at
least tends to need a substantial amount of configuration for scaling,
and where and what to log is the least of that.
Also, while the default openlog/syslog/closelog doesn't scale all that
well, it's certainly quite possible to log asynchronously to syslog
without losing performance, and some syslog implementations are more
scalable than others.
The default configuration is never going to work for *everyone*; the
question then becomes what configuration provides the most value to the
most people. I would suggest that having unified log handling
system-wide makes sense as a default, and systems handling huge amounts
of traffic will often require other custom configuration by the sysadmin
anyway.
> * The package does its own log analysis. INN again is an excellent
> example: it has a bespoke but very comprehensive and incredibly useful
> log analysis package that ships with the software and requires access to
> the raw logs. Anything that moves those logs out of their expected
> locations will break that tool, which is a significant regression for
> our users.
And apache log analyzers would want similar things, yes. I'd certainly
be happy to help update such software to be able to extract log messages
from places like syslog or journald, and I'd also be happy in the
interim with such software continuing to log to standalone files. Again,
I'm aiming for the common case here, and I expect this to be a
years-long effort, not an overnight one.
- Josh Triplett
Reply to: