Re: psad, aptitude, man, mutt, all suddenly broken on my Lenny server. I'm very concerned.

["Andrew M.A. Cater" <amacater@galactic.demon.co.uk>]

On Thu, Jan 15, 2009 at 10:56:53PM +0000, Sam Kuper wrote:
> Dear all,
> 
> It's quite late where I live, and I've had a very long day, so I'm not
> thinking at my best right now, which is why I'm asking for help sooner
> than I'd normally like to. (Normally I'd try to do quite a bit more
> research/investigation myself before seeking help from the mailing
> list.) So please be patient with me.
> 
> On 6 Jan, I logged into a server I run, first logging in as a user and
> then immediately using 'su' to become root, which is what I normally
> do when I need to perform an administrative task.
> 
> Everything seemed fine. I was logging in in order to reroute my
> DShield submission reports to go to reports@dshield.org instead of
> having them sent to my own email address, due to the issue here[1]. I
> edited the psad.conf file in Vim, ran /etc/init.d/psad restart , and
> exited. I don't recall seeing any errors in response to any of those
> commands, though in retrospect I suppose there's a slim chance I might
> have missed an error message if there had been one: I was under some
> time pressure.
> 
> This week, I noticed I'd been getting no DShield submission reports at
> all, and this evening I decided to investigate and discovered that
> psad was not emailing me lists of attacks either, which it normally
> does.
> 
> So I logged into the server just now over SSH (the server's 80mi away,
> unsupervised, in a trustworthy friend's basement; I use it for remote
> backup), and opened mutt, and I can see that the last emails the
> server sent were from psad and they were sent on 7 Jan. But I can't
> read them to see what time they were sent. When I try to do that, I
> get an error, "Could not create temporary file!"
> 
> Hmm, well, that's never happened to me before.
> 
> I tried running 'psad -S | less' and discovered that although I had
> indeed restarted psad last time I logged in, it isn't running now. So
> I ran '/etc/init.d/psad start', and got the result:
> 
> "Starting Port Scan Attack Detector and associated daemons: sh:
> /var/log/psad/psad.iptout: Read-only file system
> sh: /var/log/psad/psad.iptout: Read-only file system
> sh: /var/log/psad/psad.iptout: Read-only file system
> [*] Could not open /var/log/psad/fw_check: Read-only file system at
> /usr/sbin/fwcheck_psad line 99.
> [*] Could not open pidfile /var/run/psad/psad.pid: Read-only file system
> touch: cannot touch `/var/run/psad.lock': Read-only file system"
> 
> Well, that's never happened to me either.
> 
> In some confusion, I tried, 'aptitude update', which produced: "bash:
> /usr/bin/aptitude: Input/output error".
> 
> OK, never seen that before either :(
> 
> I've also noticed that if I try to use a man page, e.g. with the
> command, 'man bash', I get an error along the lines, "Manual page
> bash(1) line ?/? (END)".
> 
> I've never seen this error either.
> 
> I've done a bit of googling on these problems, but haven't found
> anything yet that seems to relate specifically to my circumstances:
> i.e. the times others have received these errors have been after using
> XFS (I use EXT2 or EXT3 depending upon the partition), or they've been
> running a dist-upgrade or suchlike, which I wasn't doing when the
> server started malfunctioning.
> 
> I guess I should be checking some logs at this point, but frankly,
> trying to troubleshoot a server this broken unassisted when I'm this
> tired is a little more than I think it's wise to attempt.
> 
> I'd be very grateful, therefore, if anyone who reads this could please
> make some suggestions about how to methodically go about diagnosing
> the problem(s) and curing it/them.
> 
> Many thanks in advance,
> 
> Sam
> 
> [1] http://lists.dshield.org/pipermail/list/2009-January/027325.html
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org 
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Out of /var space? Large files "somewhere" taking up space.?

Mail may not work without things like /var/spool

Aptitude / apt certainly won't work without their cache

Reboot and let it do an fsck??? [Might get rid of the rogue read only?]

HTH,

AndyC