Colo Servers, Telco-Grade Configuration, Embedded/Turnkey Systems

[debbi@starshine.org]

 I have a question related to a variety of similar common server
 configurations.  In many cases servers are remote; they have no
 console, or the console is almost always unattended.

 There are a number of settings that will make a system (even a 
 PC) much more robust under these circumstances.  Currently I have
 to hack them into the rc*, /etc, /proc and other places by hand
 when I'm setting up such a system.

 Here's a sample of some of the settings that I change/use:

 	Move the /usr/bin/logger to /bin (so it's on the root
	fs before /usr is mounted).

	Change the ifconfig and networking to enable networking
	before fsck, mounting, etc.

	Put sshd (and libz and libcrypto) on the root filesystem 
	and start it from a respawn (-D) option in /etc/inittab.

	Likewise for syslogd and crond,

	Tune the kernel for panic=60 (don't hang at a crashdump
	forever), reboot after a small timeout.

	Configure lilo and the kernel for serial console; add
	dumb-terminal getty entries (if I have some sort of terminal 
	server in that rack/cabinet, sometimes I just crosslink the 
	serial ports from nearby boxes --- so I can get to the serial
	console of any one box from its neighbor (a daisy chain)
	[I'd like to figure out how to do that so each null modem
	can be bidrectionally attached --- each as a monitoring getty
	but doesn't emit an issue file until it recieves a break or
	<Enter> or something].

	Install software watchdog and loaddog utilities. (Sometimes,
	I install hardware watchdog timer cards).

	I go through rc scripts and change as many of them to use 
	logger -s for their errors and warnings as I can.  

 I do various other tweaks as they occur to me and has required by 
 particular systems.

 Obviously (being a lazy bum) I'd like to have some of these merged
 into Debian policies, or at least made compatible with policy and
 automated into an installable package.
 
 I'd like to see Debian GNU/Linux come to completely dominate the 
 colocated and telco-grade remote/isolated server markets.  These 
 sorts of unattended systems (sometimes called "embedded" or "turnkey"
 systems) are a huge marketshare among full-time professional system
 administrators.  
 
 It's pretty common to want maximal robustness and secure remote 
 accessibility for such systems.  Driving across town and "getting
 into the cage" (dealing with Pinkerton's dweebs and fingerprint
 or palm reader machines at the front door, etc) is a hassle.  Linux
 is far better for this than *cough* any Microsoft OS; and it's
 almost as good as any other UNIX (better than any other PC UNIX).

 But, it could be better.  

 What do I want?

 I want to see a version of syslog that can be started off the
 root fs before any other filesystems are started, even before
 the network interfaces are configured.  It would buffer its
 messages in RAM (malloc'd) until it can open all of its
 pipes, files, and sockets.  It would have a signal handler that
 would tell it when to try opening /var/log, and it's UDP/TCP sockets
 tunnels, or whatever, et al (and/or it could try periodically until 
 those succeed).  Maybe another signal (USR2?) could tell it to 
 give up on any dead sinks that were listed in /etc/syslog.conf
 (posting a warning/error to any of the successfully opened sinks).

 I want to see logger moved to the rootfs.  I'd like to see all
 rc scripts use it for all error and warning output (using it with -s
 causes those to be tee'd to stderr, too).

 I want to see sshd and its requisite libraries moved to the rootfs
 (and optionally put in inittab with a respawn and the necessary
 --- and quite recent! --- -D (don't fork/daemonize) option).

 I'd like to see static network configuration (optionally) resequenced
 so that it occurs before fsck/mounting.

 I'd love to see a (tricky) option that would run the rc scripts
 under a detached screen session (if screen was installed) so that
 any sort of rc script "hang at fsck" type conditions could be
 handled remotely (assuming we got far enough to get sshd and screen
 started --- I'd ssh into the "down" system, re-attach to the screen
 session and handle the problem interactively).  Of course we'd like
 for all scripts to have and ABSOLUTELY UNATTENDED, DON'T STOP FOR
 ANYTHING, robustness --- but we might not ever quite get there ---
 so a run under screen hack might help for that last 1%.

 I'd like to see an installable package that would configure the
 kernel, bootloader, and inittab for serial console and login.

 I'd like to see a 'task-unattended-server' that would include
 software watchdog and panic= settings.

 I'd like to see other ideas for the things I've missed.

 I'd like a winning lottery ticket in that California lottery,
 too.  (Think of what SPI and FSF could do with little portions 
 of that!).