Re: Debian Investigation Report after Server Compromises
[personal reply, and posting on -devel]
Hi Joey,
thanks for this report. I am aware that this is the result of tedious
work, and I really appreciate your efforts. Let me, however, ask a few
probably inconventient questions, and I surely hope that they won't be
ignored this time.
On Tue, Dec 02, 2003 at 03:23:51PM +0100, Martin Schulze wrote:
> Several methods based on different control data were used to verify
> the packages and to ensure that the archives weren't altered by the
> attacker:
>
> . externally stored lists of MD5 sums accumulated over the past weeks
> on not compromised machines
> . digitally signed .changes files from external debian-devel-changes
> archives on not compromised machines
> . digitally signed .changes files on the respective archive servers
> . externally stored mirror log files
(1) Were checks done on completely all archives?
- main archive on auric, including package pool and potato
- non-US
- security
(2) Are more details on the checks available? For example, are the
scripts that were used available to the public? Against which
servers did you check?
(3) Since there currently seem to be gaps in archival of .changes
files, can you positively say that every single file in all archives
was verified for its integrity?
> On Wednesday, November 19th, at approximately 5pm GMT, a sniffed
> password was used to log into an unprivileged developer account on the
> host klecker (.debian.org).
> The same account and password data were then used to log into the
> machine master,
> The attacker then tried to get access to the host murphy with the same
> account.
> On the next day the attacker used a password sniffed on master to log
> into gluck, get root there and also install the SucKIT root-kit.
(4) Did the attacker try to get user or root access on other project
boxes, including auric? Were these access attempts successful?
(5) Were the other project boxes, including auric, swept for root-kits
as well? Which methods were used to determine the other boxes
being clean?
> The forensic analysis revealed exact dates and times when the program
> /sbin/init was overwritten and the root-kit installed. The analysts
> also discovered the executable file which was used to gain root access
> on the machines, which was protected and obfuscated with Burneye.
> Upon unwrapping and disassembling the exploit, security experts
> discovered which kernel bug was utilised.
This is a pretty major and impressive achievement. My compliments on
being sucessful. Debian has proven again to be technically adept, I
really appreciate that.
> On klecker, however, this was postponed for a scheduled maintenance so
> the security archive could be brought online again sooner than the
> other services. At that time we also didn't have console access to
> klecker, so recovery had to be done remotely. After a disk-image was
> made via serial console login to a local machine on a firewalled
> network connection, the root-kit was removed, the kernel exchanged and
> hardened, binaries double-checked and the security archive verified
> against several different external sources. This machine will be
> re-installed in the next few weeks.
While my rationality says that this procedure is fine, my gut feelings
are not comfortable with this.
(6) Did klecker run with a known good system (for example, booted from
a CD) while the binaries were verified?
(7) Wouldn't it be possible to move security.debian.org to a different
machine while klecker is reinstalled sooner than "in the next few
weeks"?
(8) Will you repeat the scrutiny on the security archive after
klecker's reinstallation? Do you keep reference data around so
that this scrutiny will be easier and faster?
> The secret GnuPG/PGP keys which were found on debian.org machines were
> also removed from the Debian keyrings and thus deactivated.
(9) This most probably includes the Keys that are used to automatically
sign Release files, right? Will new Debian Archive Automatic
Signing Keys be generated?
> Thanks
>
> . James Troup and Ryan Murray for their general work on all hosts
> . Adam Heath and Brian Wolfe for their work on master and murphy
> . Wichert Akkerman for his work on klecker
> . Dann Frazier and Matt Taggart for their work on gluck
> . Michael Stone and Robert van der Meulen for their forensics work
> . Marcus Meissner for disassembling the used exploit
> . Jaakko Niemi for his work on checking and re-enabling lists.debian.org
> . Colin Watson for his work on checking and re-enabling bugs.debian.org
> . Josip Rodin for his work on checking and re-enabling the lists web archives
Let me say "Thank you" as well.
This announcement has greatly raised my trust in the project again,
and I really appreciate the openness. I hope that you will be able to
answer the questions I have raised.
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Karlsruhe, Germany | lose things." Winona Ryder | Fon: *49 721 966 32 15
Nordisch by Nature | How to make an American Quilt | Fax: *49 721 966 31 29
Reply to: