Re: we were attacked
Steve Kemp wrote:
Which we did, but nothing popped out at us in the access or error logs,
nor in the cgi-exec logs. We were stumped.
Looking in logs for this is pretty good, but I think if you start
seeing things owned by www-data on pretty-stock installations you
pretty much must assume that it is an Apache problem.
But, if I'm going to get labeled as appalling for trying to help the guy
out, then, next time, I'll just keep my trap shut and let the dude
suffer. In fact, maybe I'll go the whole nine yards and just criticize
the other posters who *do* try to help.
No the appalling part was you having a machine compromised
resetting it to a "good" state and then letting it get compromised
again, and again, and again.
Two things about this:
1 - When we started getting these intrusions, we were only a couple of
months away from switching to a 1and1 FC2 server. Since we
half-suspected that there was a flaw in some piece of software that was
on our current Debian box, we had a feeling of "this problem might
disappear once the move is done". Changing to an entirely brand-new
server trumps upgrading your packages in deciding if it's a problem with
the OS and applications or if it's a problem in a customer script. The
first time we saw the problem on the new server, we got really serious
2 - Again, we were stumped. The script I wrote to watch for the
intrusion could check for and terminate the processes faster, and more
consistently than any human. And the only thing we could have done to
*prevent* the intrusion would have been to unplug the ethernet cable
from the box and close the business. Keep in mind that the intrusion was
fairly innocuous. We were running two rootkit detectors as well as
another tool that compared all binaries in /bin, /sbin, /usr/bin, and
/usr/sbin against hashes of their originals. By all indications, no
existing files were being bothered. If they were, we *would* have
unplugged the machine.
Well, it would come and go. It would happen about 2-3 times, and then
we'd upgrade packages... change some Apache configs... and then the
problem would seem to go away. We didn't know if we had patched the hole
or if the guy had just moved on to other targets. Since we didn't know
exactly how they were getting in, we didn't know if certain package
upgrades should solve the problem or not. We just had to wait and see if
the problem came back.
(Actually it wasn't clear how many times you were reinfected from
your original mail so maybe I read it worse than it actually was.)
So, it would get quiet for a month... and then it would show up again.
We'd upgrade packages, and fuss around... problem would go away for a
couple of weeks...
And on and on it went for about a dozen times until we finally found it.