[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: we were attacked

Steve Kemp wrote:
Looking in logs for this is pretty good, but I think if you start
 seeing things owned by www-data on pretty-stock installations you
 pretty much must assume that it is an Apache problem.
Which we did, but nothing popped out at us in the access or error logs, nor in the cgi-exec logs. We were stumped.
But, if I'm going to get labeled as appalling for trying to help the guy out, then, next time, I'll just keep my trap shut and let the dude suffer. In fact, maybe I'll go the whole nine yards and just criticize the other posters who *do* try to help.

  No the appalling part was you having a machine compromised
 resetting it to a "good" state and then letting it get compromised
 again, and again, and again.
Two things about this:
1 - When we started getting these intrusions, we were only a couple of months away from switching to a 1and1 FC2 server. Since we half-suspected that there was a flaw in some piece of software that was on our current Debian box, we had a feeling of "this problem might disappear once the move is done". Changing to an entirely brand-new server trumps upgrading your packages in deciding if it's a problem with the OS and applications or if it's a problem in a customer script. The first time we saw the problem on the new server, we got really serious about it.

2 - Again, we were stumped. The script I wrote to watch for the intrusion could check for and terminate the processes faster, and more consistently than any human. And the only thing we could have done to *prevent* the intrusion would have been to unplug the ethernet cable from the box and close the business. Keep in mind that the intrusion was fairly innocuous. We were running two rootkit detectors as well as another tool that compared all binaries in /bin, /sbin, /usr/bin, and /usr/sbin against hashes of their originals. By all indications, no existing files were being bothered. If they were, we *would* have unplugged the machine.
  (Actually it wasn't clear how many times you were reinfected from
 your original mail so maybe I read it worse than it actually was.)
Well, it would come and go. It would happen about 2-3 times, and then we'd upgrade packages... change some Apache configs... and then the problem would seem to go away. We didn't know if we had patched the hole or if the guy had just moved on to other targets. Since we didn't know exactly how they were getting in, we didn't know if certain package upgrades should solve the problem or not. We just had to wait and see if the problem came back.

So, it would get quiet for a month... and then it would show up again. We'd upgrade packages, and fuss around... problem would go away for a couple of weeks...

And on and on it went for about a dozen times until we finally found it.

- Joe

Reply to: