Re: Syslog/Rsyslog/Systemctl issue

[Freyja <Sioban@sioban.net>]

    There is an interesting part: I've already tried to solve this issue
    in December with some success except for 2 systemd services
    (udevd-control & udevd-kernel).
    It stopped again working the 25th of december.
    
    --
    I've found a journal related to 24th December in /var/log/journal:
    
    root in /var/log
    ❯ ls -la journal/
    total 44
    drwxr-sr-x+  3 root systemd-journal  4096 Aug 19  2021 .
    drwxr-xr-x. 33 root root            12288 Feb  1 06:25 ..
    drwxr-sr-x+  2 root systemd-journal 20480 Dec 24 11:35
    ef72ee158c9d4ae68bd60ae8adbb20af
    
    root in /var/log
    ❯ cd journal/ef72ee158c9d4ae68bd60ae8adbb20af/
    
    root in log/journal/ef72ee158c9d4ae68bd60ae8adbb20af
    ❯ ls -la
    total 98356
    drwxr-sr-x+ 2 root systemd-journal    20480 Dec 24 11:35 .
    drwxr-sr-x+ 3 root systemd-journal     4096 Aug 19  2021 ..
    -rw-r-----+ 1 root systemd-journal 75497472 Oct 24 14:14
    system.journal
    -rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14
    user-1000.journal
    -rw-r-----+ 1 root systemd-journal  8388608 Oct 23 19:56
    user-1014.journal
    -rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14
    user-1015.journal
    --
    
    And here are our logs from the 24th October.

Le 01/02/2023 à 13:26, Freyja a écrit :

Hi greg,

--
❯ df /run
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1629236  3732   1625504   1% /run
--

For /run/systemd/journal/socket this is another story.
The folder journal does not exist at all.

--
❯ ls -la /run/systemd/
total 0
drwxr-xr-x 17 root            root             400 Jan 31 16:12 .
drwxr-xr-x 37 root            root            1100 Feb  1 06:25 ..
drwxr-xr-x  2 root            root              40 Jan 31 10:45 ask-password
drwx------  2 root            root              80 Jan 31 17:31 ask-password-block
drwxr-xr-x  8 root            root             680 Jan 31 16:08 generator
drwxr-xr-x  4 root            root             360 Jan 31 16:08 generator.late
drwxr-xr-x  3 root            root             160 Jan 31 10:45 inaccessible
srw-rw-rw-  1 root            root               0 Jan 31 10:45 io.system.ManagedOOM
drwxr-xr-x  2 root            root              40 Jan 31 10:45 machines
drwxr-xr-x  5 systemd-network systemd-network  100 Jan 31 10:45 netif
srwxrwxrwx  1 root            root               0 Jan 31 10:45 notify
srwxrwxrwx  1 root            root               0 Jan 31 10:45 private
drwxr-xr-x  2 root            root              40 Jan 31 10:45 seats
drwxr-xr-x  2 root            root              40 Jan 31 10:45 sessions
drwxr-xr-x  2 root            root              40 Jan 31 10:45 shutdown
drwxr-xr-x  2 root            root              40 Jan 31 10:45 system
drwx------  2 root            root              40 Jan 31 10:45 unit-root
drwxr-xr-x  2 root            root            2060 Feb  1 13:09 units
drwxr-xr-x  2 root            root              60 Jan 31 10:45 userdb
drwxr-xr-x  2 root            root              40 Jan 31 10:45 users
--

I don't know why :/

Le 01/02/2023 à 13:22, Greg Wooledge a écrit :

On Wed, Feb 01, 2023 at 11:15:58AM +0100, debian@sioban.net wrote:

"journalctl -e" ends the 24th of October.

❯ journalctl -b
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24
14:14:33 CEST. --
-- No entries --

❯ systemctl start systemd-journald.service
A dependency job for systemd-journald.service failed. See 'journalctl -xe'
for details.

On my system:

unicorn:~$ systemctl cat systemd-journald.service | grep -i require
Requires=systemd-journald.socket
# /proc/<pid>/exe requires this capability. Thus if this capability is missing

unicorn:~$ systemctl cat systemd-journald.socket | grep Listen
ListenDatagram=/run/systemd/journal/socket
ListenStream=/run/systemd/journal/stdout

For you, that's at least *two* socket services that appear to be failing.
I suspect you've got something broken that's preventing these sockets
from being created, or being accessed.

Man, what I wouldn't give for a nice, clear "/run/systemd/journal/socket:
Permission denied" or "... No such file or directory" error!

/run is supposed to be mounted as a separate file system, entirely in
memory:

unicorn:~$ df /run
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1215596  1924   1213672   1% /run

And the ownerships and permissions on this socket are:

unicorn:~$ namei -l /run/systemd/journal/socket
f: /run/systemd/journal/socket
drwxr-xr-x root root /
drwxr-xr-x root root run
drwxr-xr-x root root systemd
drwxr-xr-x root root journal
srw-rw-rw- root root socket

If you've got anything different, that would be important.