Steve Kemp wrote:
Which we did, but nothing popped out at us in the access or error logs, nor in the cgi-exec logs. We were stumped.Looking in logs for this is pretty good, but I think if you start seeing things owned by www-data on pretty-stock installations you pretty much must assume that it is an Apache problem.
But, if I'm going to get labeled as appalling for trying to help the guy out, then, next time, I'll just keep my trap shut and let the dude suffer. In fact, maybe I'll go the whole nine yards and just criticize the other posters who *do* try to help.No the appalling part was you having a machine compromised resetting it to a "good" state and then letting it get compromised again, and again, and again.
Two things about this:1 - When we started getting these intrusions, we were only a couple of months away from switching to a 1and1 FC2 server. Since we half-suspected that there was a flaw in some piece of software that was on our current Debian box, we had a feeling of "this problem might disappear once the move is done". Changing to an entirely brand-new server trumps upgrading your packages in deciding if it's a problem with the OS and applications or if it's a problem in a customer script. The first time we saw the problem on the new server, we got really serious about it.
2 - Again, we were stumped. The script I wrote to watch for the intrusion could check for and terminate the processes faster, and more consistently than any human. And the only thing we could have done to *prevent* the intrusion would have been to unplug the ethernet cable from the box and close the business. Keep in mind that the intrusion was fairly innocuous. We were running two rootkit detectors as well as another tool that compared all binaries in /bin, /sbin, /usr/bin, and /usr/sbin against hashes of their originals. By all indications, no existing files were being bothered. If they were, we *would* have unplugged the machine.
Well, it would come and go. It would happen about 2-3 times, and then we'd upgrade packages... change some Apache configs... and then the problem would seem to go away. We didn't know if we had patched the hole or if the guy had just moved on to other targets. Since we didn't know exactly how they were getting in, we didn't know if certain package upgrades should solve the problem or not. We just had to wait and see if the problem came back.(Actually it wasn't clear how many times you were reinfected from your original mail so maybe I read it worse than it actually was.)
So, it would get quiet for a month... and then it would show up again. We'd upgrade packages, and fuss around... problem would go away for a couple of weeks...
And on and on it went for about a dozen times until we finally found it. - Joe