System hits kern.maxproc, crashes
Hi,
Today I noticed a Wheezy kfreebsd-amd64 box had become unresponsive.
It seems it had exhausted kern.maxproc (default 6164), and there had
been ssh bruteforcing activity immediately prior to that.
Coincidentally there had been 6041 ssh connections/disconnections over a
7-hour period before the process limit was exhausted.
With MaxStartups unset in sshd_config, the default is for sshd (verified
by running with -T option) to spawn no more than 10 login processes at a
time. UseDNS is enabled by default, so each connection attempt would
also trigger a DNS lookup.
In fact I only permit public key authentication anyway, so the attacker
had been pointlessly connecting, supplying a username and getting
disconnected. When I do this myself, the sshd child process appears to
exit cleanly. So how did it ever exceed kern.maxproc?
> auth.log:Jul 24 12:36:19 localhost sshd[85122]: reverse mapping checking getaddrinfo for 93-95-10-4.static.rockford-it.net [93.95.10.4] failed - POSSIBLE BREAK-IN ATTEMPT!
> auth.log:Jul 24 12:36:19 localhost sshd[85124]: input_userauth_request: invalid user shikiuchi [preauth]
> auth.log:Jul 24 12:36:19 localhost sshd[85124]: Invalid user shikiuchi from 93.95.10.4
> auth.log:Jul 24 12:36:19 localhost sshd[85124]: Received disconnect from 93.95.10.4: 11: Bye Bye [preauth]
> auth.log:Jul 24 12:36:19 localhost sshd[85124]: reverse mapping checking getaddrinfo for 93-95-10-4.static.rockford-it.net [93.95.10.4] failed - POSSIBLE BREAK-IN ATTEMPT!
> auth.log:Jul 24 12:36:20 localhost sshd[85126]: fatal: fork of unprivileged child failed
> kern.log:Jul 24 12:36:20 localhost kernel: maxproc limit exceeded by uid 0, please see tuning(7) and login.conf(5).
Also Exim, which only listens locally, was unable to spawn the next
queue runner job due to the system-wide maxproc limit being reached:
> exim4/mainlog:2012-07-24 12:44:37 daemon: fork of queue-runner process failed: Resource temporarily unavailable
> kern.log:Jul 24 12:44:37 localhost kernel: maxproc limit exceeded by uid 103, please see tuning(7) and login.conf(5).
That's all from the system logs around this time.
I was unable to log in after this, so had to reset the box.
Regards,
--
Steven Chamberlain
steven@pyro.eu.org
Reply to: