[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Fwd: Debian Investigation Report after Server Compromises

  Hola gent

  Penso que us interessa el tema i aquí us reenvio el text del missatge 
fent-ne un resum en la llengua mare de les qüestions que com a simple usuari 
m'han cridat l'antenció.

  - Diuen que l'equip d'administració i el d'experts en seguretat finalment 
han indicat la detecció i les tasques de seguiment i comprovació derivades 
del problema de seguretat que varem tenir als servidors de Debian.

  - En el Timline veiem el procés de seguiment: la detecció de l'atac, 
procediment seguit.

Comentari: És d'esperar que a part de l'actualització del nucli apareguin 
d'altres canvis -- en especial en l'aplicació de pedaços per a temes tan 

  Crec que també fora important el saber com fer-nos-ho per a saber del cert 
que els nostres sistemes no han estat atacats amb un mètode més "pulit", o 
amb més èxit, per a que ens entenguem. Si algú de vosaltres pot fer-hi llum 
us ho agrairé. 

,--------------- Missatge reenviat (principi)

 Assumpte: Debian Investigation Report after Server Compromises
 De: Martin Schulze <joey@infodrom.org>
 Data: Tue, 02 Dec 2003 16:30:10 +0100
 Grup de notícies: linux.debian.announce

 The Debian Project                                http://www.debian.org/
 Debian Investigation Report                             press@debian.org
 December 2nd, 2003
 Debian Investigation Report after Server Compromises
 The Debian administration team and security experts are finally able
 to pinpoint the method used to break-in into four project machines.
 However, the person who did this has not yet been uncovered.
 The package archives were not altered by the intruder.
 The Debian administration and security teams have checked these
 archives (security, us, non-us) quite early on in the investigation
 and re-installation process.  That's why the project was able to open
 up the security archive again and confirm that the stable update
 (3.0r2) wasn't compromised.
 If the project had anticipated to get compromised at the same time the
 stable update was implemented, the involved people would have
 postponed it.  However, the updated packages were already installed in
 the stable archive and mirror servers at the time the break-ins were
 discovered, so it wasn't possible to hold it back anymore.
 Several methods based on different control data were used to verify
 the packages and to ensure that the archives weren't altered by the
  . externally stored lists of MD5 sums accumulated over the past weeks
    on not compromised machines
  . digitally signed .changes files from external debian-devel-changes
    archives on not compromised machines
  . digitally signed .changes files on the respective archive servers
  . externally stored mirror log files
 Below is the timeline of discovery and recovery of the compromised
 machines.  All times are in UTC.  Some times are only estimates since
 our conversation did not contain exact timestamps.
    Sep 28  01:33  Linus Torvalds releases 2.6.0-test6 with do_brk() fix
    Oct 02  05:18  Marcello Tosatti applies do_brk() boundary check
    Nov 19  17:00  Attacker logs into klecker with sniffed password
    Nov 19  17:08  Root-kit installed on klecker
    Nov 19  17:20  Attacker logs into master with same sniffed password
    Nov 19  17:47  Root-kit installed on master
    Nov 19  18:30  Attacker logs into murphy with service account from master
    Nov 19  18:35  Root-kit installed on murphy
    Nov 19  19:25  Oopses on murphy start
    Nov 20  05:38  Oopses on master start
    Nov 20  20:00  Discovery of Oopses on master and murphy
    Nov 20  20:54  Root-kit installed on gluck
    Nov 20  22:00  Confirmation that debian.org was compromised
    Nov 21  00:00  Deactivation of all accounts
    Nov 21  00:34  Shut down security.debian.org
    Nov 21  04:00  Shut down gluck (www, cvs, people, ddtp)
    Nov 21  08:30  Point www.debian.org to www.de.debian.org
    Nov 21  10:45  Public announcement
    Nov 21  16:47  Developer information updated
    Nov 21  17:10  Shut down murphy (lists)
    Nov 22  02:41  security.debian.org is back online
    Nov 25  07:40  lists.debian.org is back online
    Nov 28  22:39  Linux 2.4.23 released
 On the evening (GMT) of Thursday, November 20th, the admin team
 noticed several kernel oopses on master.  Since that system was
 running without problems for a long time, the system was about to be
 taken into maintenance for deeper investigation of potential hardware
 problems.  However, at the same time, a second machine, murphy, was
 experiencing exactly the same problems, which made the admins
 Also, klecker, murphy and gluck have "Advanced Intrusion Detection
 Environment" (package aide) installed to monitor filesystem changes
 and at around the same time it started warning that /sbin/init had
 been replaced and that the mtime and ctime values for
 /usr/lib/locale/en_US had changed.
 Further investigation revealed the cause for both these problems to be
 the SucKIT root-kit.  It includes password sniffing and detection
 evasion capabilities (i.e. tools to hide processes and files) which
 are installed directly into the kernel, which in turn caused the
 oopses that were noticed.
 Detailed Attack Analysis
 On Wednesday, November 19th, at approximately 5pm GMT, a sniffed
 password was used to log into an unprivileged developer account on the
 host klecker (.debian.org).  The attacker then retrieved the source
 code through HTTP for an (at that time) unknown local kernel exploit
 and gained root permissions via this exploit.  Afterwards, the SucKIT
 root-kit was installed.
 The same account and password data were then used to log into the
 machine master, to gain root permissions with the same exploit and
 also to install the SucKIT root-kit.
 The attacker then tried to get access to the host murphy with the same
 account.  This failed because murphy is a restricted machine and its
 only purpose is to act as list server to which only a small subset of
 developers can log into.  Since the initial login attempt didn't work
 the person used his root access on master to access an administrative
 account which was used for backup purposes and gained access to murphy
 as well.  The SucKIT root-kit was installed on this host as well.
 On the next day the attacker used a password sniffed on master to log
 into gluck, get root there and also install the SucKIT root-kit.
 The forensic analysis revealed exact dates and times when the program
 /sbin/init was overwritten and the root-kit installed.  The analysts
 also discovered the executable file which was used to gain root access
 on the machines, which was protected and obfuscated with Burneye.
 Upon unwrapping and disassembling the exploit, security experts
 discovered which kernel bug was utilised.
 An integer overflow in the brk system call was exploited to overwrite
 kernel memory (change page protection bits).  By doing so the attacker
 gained full control about the kernel memory space and was able to
 alter any value in memory.
 Even though this kernel bug was discovered in September by Andrew
 Morton and already fixed in recent pre-release kernels since October,
 its security implication wasn't considered that severe.  Hence, no
 security advisories were issued by any vendor.  However, after it was
 discovered to be used as a local root exploit the Common
 Vulnerabilities and Exposures project has assigned CAN-2003-0961 to
 this problem.  It is fixed in Linux 2.4.23 which was released last
 weekend and in the Debian advisory DSA 403.
 Linux 2.2.x is not vulnerable to this exploit because boundary
 checking is done before.  It is also believed that Sparc and PA-RISC
 kernels are not vulnerable since user and kernel addresses are stored
 in different address spaces on these architectures.
 Please understand that we cannot give away the used exploit to random
 people who we don't know.  So please don't ask us about it.
 After the machines were shut down, images of the compromised hard
 disks were created and stored on a separate machine.  They were
 distributed to the people doing the forensic analysis.  The three
 machines in the US (master, murphy, gluck) were reinstalled afterwards
 and their services re-instated one by one after investigation by the
 relevant service admin.
 On klecker, however, this was postponed for a scheduled maintenance so
 the security archive could be brought online again sooner than the
 other services.  At that time we also didn't have console access to
 klecker, so recovery had to be done remotely.  After a disk-image was
 made via serial console login to a local machine on a firewalled
 network connection, the root-kit was removed, the kernel exchanged and
 hardened, binaries double-checked and the security archive verified
 against several different external sources.  This machine will be
 re-installed in the next few weeks.
 As a security precaution all developer accounts were disabled in LDAP
 and SSH keys removed on the more important machines, so that no more
 machines could be compromised.  This, however, effectively disabled
 just about any public Debian work that involved uploading files and
 accessing the CVS repositories.
 All passwords used on quantz (i.e. all Alioth, arch and subversion
 passwords) have been invalidated as well.  All SSH authorized keys
 have been removed as well.  Please use the lost password system to
 receive a new password at:
 When all services are running again and the machines are sufficiently
 secured, LDAP will be reset so that developers can create a new
 password again (<http://db.debian.org/password.html>).  It can't
 currently be predicted when this will happen, though.
 Upon recovery SSH was re-installed on the compromised machines.
 Hence, there are new RSA host keys and key fingerprints for these
 hosts.  The keys will be included in LDAP as soon as they are created
 and can be taken from <http://db.debian.org/machines.cgi>.
 		    !!  Renew your passwords!  !!
 Since passwords were sniffed on the compromised hosts, any outgoing
 connection that involved a password is to be considered compromised as
 well, i.e. the password should be considered known to the attacker.
 It should therefore be changed immediately.
 Additionally, if somebody had access to a Debian machine and was using
 the same password or passphrase on other machines or keys we strongly
 advise to change the password or passphrase respectively as soon as
 If an SSH key was generated or stored on one of these machines and was
 used to log into other machines (i.e. by installing it in
 .ssh/authorized_keys), it should be removed as well.
 The secret GnuPG/PGP keys which were found on debian.org machines were
 also removed from the Debian keyrings and thus deactivated.
 Developers who are worried about their own machines should at least
 run chkrootkit and watch its output.  Matt Taggert maintains a
 backport of the current version for woody at the following address:
    deb http://lackof.org/taggart/debian woody/chkrootkit main
    deb-src http://lackof.org/taggart/debian woody/chkrootkit main
 Additionally, a detailed list of precaution issues is provided by
 Wichert Akkerman and Matt Taggart at:
 SucKIT Root-Kit
 SucKIT is a root-kit presented in Phrack issue 58, article 0x07
 ("Linux on-the-fly kernel patching without LKM", by sd & devik).  This
 is a fully working root-kit that is loaded through /dev/kmem, i.e. it
 does not need a kernel with support for loadable kernel modules.  It
 provides a password protected remote access connect-back shell
 initiated by a spoofed packet (bypassing most firewall
 configurations), and can hide processes, files and connections.
 Usually, SucKIT is launched as /sbin/init at system bootup, forks to
 install itself into the kernel, start up a backdoor, and launches a
 copy of the original "init" binary from the parent (with pid 1).  Any
 subsequent executions of /sbin/init are redirected to the original
 TESO's Burneye Protection
 Burneye is a means of obfuscating ELF binaries on the UNIX platform
 presented in Phrack issue 58, article 0x05 ("Armouring the ELF: Binary
 encryption on the UNIX platform", by grugq & scut).  Using tools like
 TESO's Burneye, an attacker can alter an executable program to encrypt
 its true purpose, hiding it from firewall filters, intrusion detection
 systems, anti-virus software and the prying eyes of investigators.
   . James Troup and Ryan Murray for their general work on all hosts
   . Adam Heath and Brian Wolfe for their work on master and murphy
   . Wichert Akkerman for his work on klecker
   . Dann Frazier and Matt Taggart for their work on gluck
   . Michael Stone and Robert van der Meulen for their forensics work
   . Marcus Meissner for disassembling the used exploit
   . Jaakko Niemi for his work on checking and re-enabling lists.debian.org
   . Colin Watson for his work on checking and re-enabling bugs.debian.org
   . Josip Rodin for his work on checking and re-enabling the lists web 
 Contact Information
 For further information, please visit the Debian web pages at
 <http://www.debian.org/> or send mail to <press@debian.org>.
 To UNSUBSCRIBE, email to debian-announce-request@lists.debian.org
 with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

`--------------- Missatge reenviat (fi)



######## Antoni Bella Perez ####################                             |
# http://www.terra.es/personal7/bella5/home.htm
## Correu-e :	<bella5 AT teleline DOT es>	##,
## ID de Jabber: vasten AT jabber DOT org	## i
col·laborador dels projectes:
	Debian en català: http://www.debian.org/index.ca.html
	KDE en català: http://ca.i18n.kde.org/
	T.P: http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=ca


Reply to: