Re: Server hacked - next...?

[Theodore Knab <tjk@annapolislinux.org>]

>Is there a document somewhere, or procedure, to recover after this? This
>is a working and running system, so somehow need to be able to recover
>from this with minimal impact to end-users.

Just backup the datafiles, the password files and rebuild.
At least that is how I would do it.

>One of our servers was hacked (woody)... badly, from what I can see. A
>whole bunch of binaries have been modified, and strange processes are
>running on the server. The hack date appears to be jun 6.

I am curious as to how your server has gotten hacked.

More specifically, I run Debian on about 15 of my servers and none of them
have been hacked. Maybe I am just lucky, but I trust Debian stable and
Debian Sarge more than any other Distro.

To prevent hackers from getting in I follow the Debian Security HOWTO,
and I keep them all updated. Additionally, I run the minimum of
services. I only run services which are needed.

Hopefully these questions I have will help you secure your system a
little better in the future.

Questions:

1. How many users were able to login in to this box and call a shell ?

Most hackers use regular user accounts to gain access to higher
privilege parts of the OS.

2. Did you setup any security as defined in the Debian Security HOWTO ?
http://www.debian.org/doc/user-manuals#securing

    A. Did you run IP-tables or ipchains ?

    B. Did you use TCP wrappers in the /etc/hosts.allow and
    /etc/hosts.deny

Much of which follow comes from the Debian Security Howto.

3. How many services did you have running on the hacked machine ?

    i.e. sshd   daemon
         apache daemon
         mysqld

Limiting services to only the services needed helps make your systems
exposure to potential exploits less. Some people in the RedHat world 
install everything. This could result in problems if the system does not
get updated on daily basis.

4. Was the machine updated on a regular basis ?
    When was the last update ?
    Did you use the security updates ?

   /etc/apt/source.lists

   #for stable
   deb http://security.debian.org/ stable/updates main contrib non-free
    
   #for sarge
   deb http://security.debian.org/ sarge/updates main contrib non-free

5. How long was the root password and could anyone su to root ? 

Simple root passwords can be guessed. Don't type 'root' as the root
password or simple dictionary words.

6. Did you allow root logins from a remote location ?

    I do not. This is defined in the /etc/ssh/sshd_config
    with this line.
    
    PermitRootLogin no

    This will help limit dictionary attacks on root.

7. Did you allow telnet and FTP on this machine ?

    If I allow FTP, I do not give them shell access.

    For example, the PAM ssh file allows you to limit who can login to 
    a shell. 
    
    Pam limits shell access to with the following line in the /etc/pam.d/ssh file:

    auth        required    pam_listfile.so item=user sense=allow \
    file=/etc/sshusers-allowed onerr=fail

    In this example, the '/etc/sshusers-allowed' holds the names of people who can use ssh.

8. Do you have any external firewall ?
    If so what ?

-- 
            -\ - /-        
         --([0]-[0])--     
+--------oOOo-(_)-oOOo--------------------------+
| Theodore Knab                                 |
| Annapolis Linux LUG when not a sysadmin       |
+-----------------------------------------------+
|        oOOo                                   |
|       (    )    oOOo                          |
+--------\  (----(   )--------------------------+
          \_)     ) / 
                 (-/