Bug#670398: linux-image-2.6.32-5-amd64: SSH logins hang while hpet interrupts multiply on Intel Nehalem CPUs
Package: linux-image-2.6.32-5-amd64
Version: 2.6.32-41squeeze2
Severity: important
Hi,
since about December 2011 we've seen systems were SSH sessions suddenly hang and
further logins on the physical TTY or via SSH are no longer possible. In some cases
ssh logins still work and you see motd and mayeb can even issue one or two commands.
(I've brought this issue up on debian-user in march with a private reply from a
fellow DD yesterday http://lists.debian.org/debian-user/2012/03/msg01204.html)
Over time we observed that ssh logins without PTS (ssh -T) still work. Looking at
other sessions sshd was in state and D entries in /dev/pts/ were created correctly.
Searching through munin graphs we could narrow down the starting point of this issue
to the point when the hpet interrupts for one CPU core multiplied. Sometimes they
multiplied by six. Looking further we've found the Kernel [events/$x] in state D
where $x is the number of the CPU core which has the high number of hpet interrupts.
When we started strace -f on the sshd master process everything works until you logout.
Then you'll again see the forked sshd process hanging in state D.
Up to that point we've seen this issue exclusively on Linux 2.6.32 based systems,
most often on Debian/Squeeze and less often on Ubuntu 10.04 and once or twice on
a RHEL 6.1 system.
Searching further I've seen references on a Dell PowerEdge mailinglist referencing
RedHat BZ#750201 and Intel CPU errata number AAO67 for Nehalem (rapid C state switching).
The RedHat bug is currently non-public but through our technical contact at RedHat I was
able to receive a summary of this bug and other referenced bugs which describe more or
less exactly our issue.
According to RedHat that should be fixed in their Kernel 2.6.32-220.7.1.el6
citing the following in the changelog:
- [x86] hpet: Disable per-cpu hpet timer if ARAT is supported (Prarit Bhargava) [772884 750201]
- [x86] Improve TSC calibration using a delayed workqueue (Prarit Bhargava) [772884 750201]
- [kernel] clocksource: Add clocksource_register_hz/khz interface (Prarit Bhargava) [772884 750201]
- [kernel] clocksource: Provide a generic mult/shift factor calculation (Prarit Bhargava) [772884 750201]
(Maybe that helps to track down the relevant changes.)
As a workaround it could work to disable C-states in the BIOS or on the Kernel commandline
with intel_idle.max_cstate=0 processor.max_cstate=1.
Since we run into that issue only from time to time on the same system we could not yet
verify either workaround. Rumours indicate that sometimes disabling it in the BIOS did
not help because the Kernel enabled C-states again.
My current guess is that it's somehow related to the Intel Nehalem CPU bug and only happens
if you have a high single threaded load which leads to one or core cores are switched into
a C-6 sleep state so that they can overclock one core. Marketing name is TurboBoost.
Regarding the CPUs I know this happens with:
- Intel X3430
- Intel X3450
- Intel L3426
We see it in almost all cases on Dell R210 with the X3430 CPUs.
Rumours claim it also happens with other Dell models based on other CPUs from the
Intel Nehalem series with TurboBoost.
Would be great if someone could track down the needed changes and incorporate those
into a point release. In general I would be available for testing but we still have
no way reproduce it beside waiting a few month. :(
Regards,
Sven
-- System Information:
Debian Release: 6.0.4
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-0.bpo.1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-2.6.32-5-amd64 depends on:
ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management sy
ii initramfs-tools [linux-init 0.99~bpo60+1 tools for generating an initramfs
ii linux-base 3.4~bpo60+1 Linux image base package
ii module-init-tools 3.12-2 tools for managing Linux kernel mo
Versions of packages linux-image-2.6.32-5-amd64 recommends:
pn firmware-linux-free <none> (no description available)
Versions of packages linux-image-2.6.32-5-amd64 suggests:
pn grub | lilo <none> (no description available)
pn linux-doc-2.6.32 <none> (no description available)
Reply to: