Strange removal of /sbin and /lib
Hey all,
Last night one of my Debian-running laptops was made inoperable by
the removal of /sbin and /bin. I'm not quite sure why or how this
happened, but I'll tell you what I did from start to finish. The laptop
itself is an old Toshiba Satellite powered by a Celeron 500 with 64
megabytes of RAM. The kernel is 2.6.0, pre-emptible, compiled and
installed via kernel-package. It was running unstable, with the apt
repository being linux.stanford.edu. The computer was running X windows
w/ fluxbox as the window manager, logged in as an unpriveleged user. The
other services were an ssh server (default settings), xfs, xfstt, and
automount. Only one kernel module was running, that one being
ndiswrapper (ndiswrapper.sourceforge.net) to enable the WPC54G wireless
card to function under Linux with Windows drivers. The hard drive is
partitioned into one ext2 partition encompassing the whole system.
I began by opening up an xterm window, su'ing to root, and running
'apt-get update' followed by 'apt-get dist-upgrade'. It asked me if I
wanted to upgrade 30 odd packages as well as install 1 new one, and
having done this many times I glossed over the list, which included a
libc update, and hit 'Y'. It downloaded the 30 packages, after which it
said something on the lines of: "Can't find ldconfig in your path" and
"Can't find start-stop-daemon in your path" and "Check and make sure
/sbin is in your path". I had never gotten this response before, so I
checked my path, and it was fine. But I couldn't find ldconfig or
start-stop-daemon with locate, and then I saw that the /sbin directory
was gone. No trace of it was left.
Knowing that the computer was a very short time away from death
(lack of init and other essential processes), I ssh'ed into another
Debian computer I have (also running unstable), and quickly scp'ed over
its /sbin directory, hoping to repair some of the damage. Once I had
that, I thought I was OK, so I ran apt-get dist-upgrade again to make
sure. It began upgrading packages. When it reached the libc upgrade, I
saw the customary "Setting up libc6" message followed by the date/time
and "Run tzconfig if you want to change the time zone". After that, the
xterm hung. It wouldn't respond to anything. Furthermore, I tried
opening a new xterm, or any other process. Nothing opened up.
I hit ctrl-alt-backspace and hopped out of X, and tried to login
through the command line. I received a string of error messages
involving "Cannot open /sbin/getty" and "Getty sent messages too fast"
or something like that. I tried to ssh into the machine through a
different machine, without luck. No user/password combination would
work. Control-alt-delete didn't work to reboot ("Could not find
/sbin/shutdown"), so I hard-rebooted. After that, it didn't boot
("Kernel panic, could not load init").
The first thing I did was boot up the laptop with a Woody 3.0 CD,
and open up a shell. I ran e2fsck -c hoping to find some badblocks on
the hard drive, but I found none. Some minor filesystem corruption was
fixed, but errors that you would expect to find on a machine that was
not cleanly unmounted. I then tried to run init, just for the hell of
it, and it gave me an error about an incompatbile GLIBC (which was true,
given that it expected the one from unstable, and got the one used on
this Woody CD). But this prompted me to check / again, and I noticed
that /lib was also gone. I then checked various logs in /var/log,
including auth.log, syslog, messages, kern.log, and a few others, but
could find nothing describing what had happened. I saw in auth.log the
various failures of PAM to try and authenticate me when I logged in
remotely, but nothing that hinted at any break-in or failure that caused
this cascade.
At this point I'm a bit stumped. I'm ready to cut my losses and
either find replacement /sbin and /lib directories, or outright
re-partition and re-install Debian, but I'm entirely perplexed as to how
this catastrophic failure happened. Did some hardware malfunction? Was
one of the packages corrupt or broken, and apt-get somehow deleted the
directories? Did somebody regardless manage to break in and remove them?
If anybody has any ideas, I'm all ears. And if anyone has any pointers
as to where to look on the laptop for possibly more information, I'm
also open to that.
Thanks!
-Adar
Reply to: