psad, aptitude, man, mutt, all suddenly broken on my Lenny server. I'm very concerned.

[Sam Kuper <sam.kuper@uclmail.net>]

Dear all,

It's quite late where I live, and I've had a very long day, so I'm not
thinking at my best right now, which is why I'm asking for help sooner
than I'd normally like to. (Normally I'd try to do quite a bit more
research/investigation myself before seeking help from the mailing
list.) So please be patient with me.

On 6 Jan, I logged into a server I run, first logging in as a user and
then immediately using 'su' to become root, which is what I normally
do when I need to perform an administrative task.

Everything seemed fine. I was logging in in order to reroute my
DShield submission reports to go to reports@dshield.org instead of
having them sent to my own email address, due to the issue here[1]. I
edited the psad.conf file in Vim, ran /etc/init.d/psad restart , and
exited. I don't recall seeing any errors in response to any of those
commands, though in retrospect I suppose there's a slim chance I might
have missed an error message if there had been one: I was under some
time pressure.

This week, I noticed I'd been getting no DShield submission reports at
all, and this evening I decided to investigate and discovered that
psad was not emailing me lists of attacks either, which it normally
does.

So I logged into the server just now over SSH (the server's 80mi away,
unsupervised, in a trustworthy friend's basement; I use it for remote
backup), and opened mutt, and I can see that the last emails the
server sent were from psad and they were sent on 7 Jan. But I can't
read them to see what time they were sent. When I try to do that, I
get an error, "Could not create temporary file!"

Hmm, well, that's never happened to me before.

I tried running 'psad -S | less' and discovered that although I had
indeed restarted psad last time I logged in, it isn't running now. So
I ran '/etc/init.d/psad start', and got the result:

"Starting Port Scan Attack Detector and associated daemons: sh:
/var/log/psad/psad.iptout: Read-only file system
sh: /var/log/psad/psad.iptout: Read-only file system
sh: /var/log/psad/psad.iptout: Read-only file system
[*] Could not open /var/log/psad/fw_check: Read-only file system at
/usr/sbin/fwcheck_psad line 99.
[*] Could not open pidfile /var/run/psad/psad.pid: Read-only file system
touch: cannot touch `/var/run/psad.lock': Read-only file system"

Well, that's never happened to me either.

In some confusion, I tried, 'aptitude update', which produced: "bash:
/usr/bin/aptitude: Input/output error".

OK, never seen that before either :(

I've also noticed that if I try to use a man page, e.g. with the
command, 'man bash', I get an error along the lines, "Manual page
bash(1) line ?/? (END)".

I've never seen this error either.

I've done a bit of googling on these problems, but haven't found
anything yet that seems to relate specifically to my circumstances:
i.e. the times others have received these errors have been after using
XFS (I use EXT2 or EXT3 depending upon the partition), or they've been
running a dist-upgrade or suchlike, which I wasn't doing when the
server started malfunctioning.

I guess I should be checking some logs at this point, but frankly,
trying to troubleshoot a server this broken unassisted when I'm this
tired is a little more than I think it's wise to attempt.

I'd be very grateful, therefore, if anyone who reads this could please
make some suggestions about how to methodically go about diagnosing
the problem(s) and curing it/them.

Many thanks in advance,

Sam

[1] http://lists.dshield.org/pipermail/list/2009-January/027325.html