Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> Please try 2.6.34 from experimental.
I've now replicated the problem on my coffee table with a temporary
(intermittent) loan of an identical machine from Jump networks.
I can get 2.6.26-21lenny4 and 2.6.26-24 to crash on demand very
easily. In none of the crashes do I get any kind of kernel log
messages (eg, soft lockup warnings).
2.6.34 wasn't in experimental any more by the time I had all this set
up. Instead I tried linux-image-2.6.35-rc5-686-bigmem version
2.6.35~rc5-1~experimental.1. I wasn't able to reproduce the problem
with 2.6.35.
> Even if you can't get a process dump, you can get some useful
> information with:
> 'd' - show locks held
> 'l' - show backtrace for active CPUs
> 'w' - show uninterruptible tasks
2.6.24 doesn't seem to have the "d" debug key. It says:
[ 723.072717] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full
kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw
Sync showTasks Unmount shoW-blocked-tasks
But I have the other information. It's in the attached logfile. On
my coffee table I was able to set the serial rate to something
sensible :-).
The first 200 lines or so are just me setting things up and checking
that debug keys are working and the output is being collected. The
message
[ 612.446061] EXT3-fs: mounted filesystem with ordered data mode.
is due to the backup process mounting (readonly) its LVM snapshot.
I'll probably have to give the machine back for a bit now, so if you'd
like to think about what set of tests you'd like me to do next that
would be great. Next time it will be quicker to set up as I can keep
the disks in between :-).
Thanks.
Ian.
Thanks for using picocom
liberator:~> really picocom -b 9600 /dev/ttyUSB0 [4P[1@1[1@1[1@5[1@2[1@2[1@0[1@0[1P[1P[1P[1@0[1@0
picocom v1.4
port is : /dev/ttyUSB0
flowcontrol : none
baudrate is : 115200
parity is : none
databits are : 8
escape is : C-a
noinit is : no
noreset is : no
nolock is : no
send_cmd is : ascii_xfr -s -v -l10
receive_cmd is : rz -vv
Terminal ready
Thanks for using picocom
liberator:~> really picocom -b 115200 /dev/ttyUSB0 [6P[1@9[1@6[1@0[1@0
picocom v1.4
port is : /dev/ttyUSB0
flowcontrol : none
baudrate is : 9600
parity is : none
databits are : 8
escape is : C-a
noinit is : no
noreset is : no
nolock is : no
send_cmd is : ascii_xfr -s -v -l10
receive_cmd is : rz -vv
Terminal ready
-rescue:~>
-rescue:~>
-rescue:~> really stty </de 115200
ÿ
Thanks for using picocom
liberator:~> really picocom -b 9600 /dev/ttyUSB0 [4P[1@1[1@1[1@1[1P[1@5[1@2[1@0[1@0
picocom v1.4
port is : /dev/ttyUSB0
flowcontrol : none
baudrate is : 115200
parity is : none
databits are : 8
escape is : C-a
noinit is : no
noreset is : no
nolock is : no
send_cmd is : ascii_xfr -s -v -l10
receive_cmd is : rz -vv
Terminal ready
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~> d h 00:37:01 up 3 min, 3 users, load average: 1.33, 0.78, 0.32
-rescue:~>
-rescue:~> cat /proc/syske /kernel/sysrq
1
-rescue:~>
-rescue:~> #
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~>
*** break sent ***
[ 415.071786] SysRq : Show Memory
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~> cat /p[Kcat /etc/sysc
sysconfig/ sysctl.conf.dpkg-dist sysctl.d/
sysctl.conf sysctl.conf~
-rescue:~> cat /etc/sysc
sysconfig/ sysctl.conf.dpkg-dist sysctl.d/
sysctl.conf sysctl.conf~
-rescue:~> cat /etc/sysctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
#net/ipv4/icmp_echo_ignore_broadcasts=1
kernel/printk= 4 4 1 4
fs/file-max= 200000
#fs/inode-max= 32768
-rescue:~> sysctl kernel/printk= 4 4 1 4
error: Malformed setting "kernel/printk="
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> sysctl kernel/printk= 4 4 1 4printk[1@ = [1@'4 4 1 4'
kernel.printk = 4 4 1 4
error: Malformed setting "="
error: "4 4 1 4" is an unknown key
-rescue:~> sysctl kernel/printk = '4 4 1 4' [1P
kernel.printk = 4 4 1 4
error: Malformed setting "="
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> sysctl kernel/printk = 4 4 1 4kernel/printk[1P[1P
kernel.printk = 4 4 1 4
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> man sysctl
[?1049h[?1h=SYSCTL(8) SYSCTL(8)[m
[m
[m
[m
[1mNAME[0m[m
sysctl - configure kernel parameters at runtime[m
[m
[1mSYNOPSIS[0m[m
[1msysctl[0m [[1m-n[0m] [[1m-e[0m] [4mvariable[24m ...[m
[1msysctl[0m [[1m-n[0m] [[1m-e[0m] [[1m-q[0m] [1m-w[0m [4mvariable[24m=[4mvalue[24m ...[m
[1msysctl[0m [[1m-n[0m] [[1m-e[0m] [[1m-q[0m] [1m-p[0m [[4mfilename[24m][m
[1msysctl[0m [[1m-n[0m] [[1m-e[0m] [1m-a[0m[m
[1msysctl[0m [[1m-n[0m] [[1m-e[0m] [1m-A[0m[m
[m
[1mDESCRIPTION[0m[m
[1msysctl[0m is used to modify kernel parameters at runtime. The parameters[m
available are those listed under /proc/sys/. Procfs is required for[m
[1msysctl[0m support in Linux. You can use [1msysctl[0m to both read and write[m
sysctl data.[m
[m
[1mPARAMETERS[0m[m
[4mvariable[24m[m
The name of a key to read from. An example is kernel.ostype.[m
[7m Manual page sysctl(8) line 1[27m[K
gzip: stdout: No space left on device
[K[?1l>[?1049l-rescue:~> man sysctlsysctl kernel/printk 4 4 1 4[1P[1@=[1@'4 4 1 4'
error: permission denied on key 'kernel.printk'
-rescue:~> sysctl kernel/printk='4 4 1 4'eal
-rescue:~> [1@r[1@e[1@a[1@l[1@l[1@y[1@
kernel.printk = 4 4 1 4
-rescue:~> really sysctl kernel/printk='4 4 1 4'[1P[1@7
kernel.printk = 7 4 1 4
-rescue:~> really sysctl kernel/printk='7 4 1 4''[1P[1@0
kernel.printk = 0 4 1 4
-rescue:~>
*** break sent ***
[ 551.797057] SysRq : Show Memory
-rescue:~>
-rescue:~> really sysctl kernel/printk='0 4 1 4'7 4 1 4'
kernel.printk = 7 4 1 4
-rescue:~> 00:42:01 up 8 min, 3 users, load average: 0.52, 0.81, 0.47
*** break sent ***
[ 560.972713] SysRq : Show Memory
[ 560.972713] Mem-info:
[ 560.972713] DMA per-cpu:
[ 560.972713] CPU 0: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 1: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 2: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 3: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 4: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 5: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 6: hi: 0, btch: 1 usd: 0
[ 560.972713] CPU 7: hi: 0, btch: 1 usd: 0
[ 560.972713] Normal per-cpu:
[ 560.972713] CPU 0: hi: 186, btch: 31 usd: 83
[ 560.972713] CPU 1: hi: 186, btch: 31 usd: 51
[ 560.972713] CPU 2: hi: 186, btch: 31 usd: 66
[ 560.972713] CPU 3: hi: 186, btch: 31 usd: 44
[ 560.972713] CPU 4: hi: 186, btch: 31 usd: 131
[ 560.972713] CPU 5: hi: 186, btch: 31 usd: 104
[ 560.972713] CPU 6: hi: 186, btch: 31 usd: 160
[ 560.972713] CPU 7: hi: 186, btch: 31 usd: 39
[ 560.972713] HighMem per-cpu:
[ 560.972713] CPU 0: hi: 186, btch: 31 usd: 166
[ 560.972713] CPU 1: hi: 186, btch: 31 usd: 34
[ 560.972713] CPU 2: hi: 186, btch: 31 usd: 147
[ 560.972713] CPU 3: hi: 186, btch: 31 usd: 175
[ 560.972713] CPU 4: hi: 186, btch: 31 usd: 48
[ 560.972713] CPU 5: hi: 186, btch: 31 usd: 157
[ 560.972713] CPU 6: hi: 186, btch: 31 usd: 86
[ 560.972713] CPU 7: hi: 186, btch: 31 usd: 10
[ 560.972713] Active:10403 inactive:29112 dirty:49 writeback:0 unstable:0
[ 560.972713] free:984661 slab:4007 mapped:1923 pagetables:184 bounce:0
[ 560.972713] DMA free:12820kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
[ 560.972713] lowmem_reserve[]: 0 873 5064 5064
[ 560.972713] Normal free:818384kB min:3744kB low:4680kB high:5616kB active:5128kB inactive:4820kB present:894080kB pages_scanned:0 all_unreclaimable? no
[ 560.972713] lowmem_reserve[]: 0 0 33528 33528
[ 560.972713] HighMem free:3107440kB min:512kB low:5008kB high:9504kB active:36484kB inactive:111628kB present:4291584kB pages_scanned:0 all_unreclaimable? no
[ 560.972713] lowmem_reserve[]: 0 0 0 0
[ 560.972713] DMA: 5*4kB 4*8kB 4*16kB 3*32kB 3*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12820kB
[ 560.972713] Normal: 62*4kB 15*8kB 16*16kB 11*32kB 42*64kB 21*128kB 12*256kB 4*512kB 2*1024kB 5*2048kB 194*4096kB = 818384kB
[ 560.972713] HighMem: 120*4kB 28*8kB 13*16kB 9*32kB 3*64kB 4*128kB 1*256kB 1*512kB 2*1024kB 1*2048kB 757*4096kB = 3107440kB
[ 560.972713] 36625 total pagecache pages
[ 560.972713] Swap cache: add 0, delete 0, find 0/0
[ 560.972713] Free swap = 8388592kB
[ 560.972713] Total swap = 8388592kB
[ 560.972713] 1310720 pages of RAM
[ 560.972713] 1081344 pages of HIGHMEM
[ 560.972713] 276248 reserved pages
[ 560.972713] 11078 pages shared
[ 560.972713] 0 pages swap cached
[ 560.972713] 49 pages dirty
[ 560.972713] 0 pages writeback
[ 560.972713] 1923 pages mapped
[ 560.972713] 4007 pages slab
[ 560.972713] 184 pages pagetables
-rescue:~>
-rescue:~>
-rescue:~> [ 612.446061] kjournald starting. Commit interval 5 seconds
[ 612.446061] EXT3-fs: mounted filesystem with ordered data mode.
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~> echo hi
hi
-rescue:~>
-rescue:~>
-rescue:~>
-rescue:~> ls
*** break sent ***
[ 677.678494] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks
*** break sent ***
[ 694.121182] SysRq : Show Blocked State
[ 694.121182] task PC stack pid father
[ 694.121182] pdflush D 00000000 0 263 2
[ 694.121182] f75a3760 00000046 c011f085 00000000 00000003 f75a38ec c3819020 00000000
[ 694.121182] 00000001 00011b31 c012126f 00000000 00000000 00000000 0000030c ffffffff
[ 694.121182] f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f75a3760 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c011f085>] __wake_up_common+0x2e/0x58
[ 694.121182] [<c012126f>] __wake_up+0x29/0x39
[ 694.121182] [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[ 694.121182] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 694.121182] [<c015fc05>] mempool_alloc+0x1c/0xba
[ 694.121182] [<c01dd76c>] submit_bio+0xc6/0xcd
[ 694.121182] [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[ 694.121182] [<c0199e27>] submit_bh+0xcf/0xed
[ 694.121182] [<c019b7d2>] __block_write_full_page+0x1fa/0x2da
[ 694.121182] [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[ 694.121182] [<c019b993>] block_write_full_page+0xe1/0xea
[ 694.121182] [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[ 694.121182] [<f8c8d889>] ext3_ordered_writepage+0xab/0x13d [ext3]
[ 694.121182] [<f8c8b030>] bget_one+0x0/0x7 [ext3]
[ 694.121182] [<c01628ed>] __writepage+0x8/0x21
[ 694.121182] [<c0162d68>] write_cache_pages+0x16a/0x27b
[ 694.121182] [<c01628e5>] __writepage+0x0/0x21
[ 694.121182] [<c0162e93>] generic_writepages+0x1a/0x21
[ 694.121182] [<c0162ec3>] do_writepages+0x29/0x30
[ 694.121182] [<c0196949>] __writeback_single_inode+0x127/0x251
[ 694.121182] [<f89626fe>] dm_table_any_congested+0x32/0x4e [dm_mod]
[ 694.121182] [<c0196d40>] sync_sb_inodes+0x17c/0x233
[ 694.121182] [<c01970b7>] writeback_inodes+0x53/0x99
[ 694.121182] [<c0163ad9>] pdflush+0x0/0x1cc
[ 694.121182] [<c0163794>] wb_kupdate+0x7b/0xdb
[ 694.121182] [<c0163c08>] pdflush+0x12f/0x1cc
[ 694.121182] [<c0163719>] wb_kupdate+0x0/0xdb
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] kjournald D 00000000 0 1550 2
[ 694.121182] f78984a0 00000046 c011f085 00000000 00000003 f789862c c3866020 00000007
[ 694.121182] 00000001 000122e8 c012126f 00000000 00000000 00000000 00000000 ffffffff
[ 694.121182] f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f78984a0 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c011f085>] __wake_up_common+0x2e/0x58
[ 694.121182] [<c012126f>] __wake_up+0x29/0x39
[ 694.121182] [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[ 694.121182] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 694.121182] [<c015fc05>] mempool_alloc+0x1c/0xba
[ 694.121182] [<c01dd76c>] submit_bio+0xc6/0xcd
[ 694.121182] [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[ 694.121182] [<c0199e27>] submit_bh+0xcf/0xed
[ 694.121182] [<f8c59d3b>] journal_commit_transaction+0x690/0xb3d [jbd]
[ 694.121182] [<c01309b4>] lock_timer_base+0x19/0x35
[ 694.121182] [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] md127_resync D 3ce2eced 0 2324 2
[ 694.121182] f68f1b80 00000046 c011f085 3ce2eced 00000089 f68f1d0c c383a020 00000003
[ 694.121182] 00000001 00000282 00000001 00000000 00000000 00000003 f6592ac0 f6592b14
[ 694.121182] f6592ac0 f6592b14 f6592af4 f6592ac0 f89cf48b 00000000 f68f1b80 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c011f085>] __wake_up_common+0x2e/0x58
[ 694.121182] [<f89cf48b>] raise_barrier+0x100/0x133 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d1291>] sync_request+0x1dd/0x59a [raid1]
[ 694.121182] [<f89e8b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[ 694.121182] [<f89ef9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 694.121182] [<c02c8f8d>] schedule+0x63b/0x66d
[ 694.121182] [<f89ef310>] md_thread+0x0/0xcd [md_mod]
[ 694.121182] [<f89ef3c7>] md_thread+0xb7/0xcd [md_mod]
[ 694.121182] [<f89ef310>] md_thread+0x0/0xcd [md_mod]
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] kjournald D 00000000 0 2483 2
[ 694.121182] f7670e20 00000046 c011f085 00000000 00000003 f7670fac c3819020 00000000
[ 694.121182] 00000001 00014482 c012126f 00000000 00000000 00000000 00000000 ffffffff
[ 694.121182] f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f7670e20 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c011f085>] __wake_up_common+0x2e/0x58
[ 694.121182] [<c012126f>] __wake_up+0x29/0x39
[ 694.121182] [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[ 694.121182] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 694.121182] [<c015fc05>] mempool_alloc+0x1c/0xba
[ 694.121182] [<c01dd76c>] submit_bio+0xc6/0xcd
[ 694.121182] [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[ 694.121182] [<c0199e27>] submit_bh+0xcf/0xed
[ 694.121182] [<f8c59d3b>] journal_commit_transaction+0x690/0xb3d [jbd]
[ 694.121182] [<c01309b4>] lock_timer_base+0x19/0x35
[ 694.121182] [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] kjournald D d3d0808b 0 2484 2
[ 694.121182] f61775a0 00000046 c38372f8 d3d0808b 00000089 f617772c c383a020 00000003
[ 694.121182] c383a454 c38372f8 000048f9 00000000 d632cf95 00000089 00000246 c013893b
[ 694.121182] f613ec14 f44a7cfc f613ec74 f613ec00 f8c5976b f44a7cc0 c383a020 f593d800
[ 694.121182] Call Trace:
[ 694.121182] [<c013893b>] prepare_to_wait+0x12/0x4d
[ 694.121182] [<f8c5976b>] journal_commit_transaction+0xc0/0xb3d [jbd]
[ 694.121182] [<c02c8f8d>] schedule+0x63b/0x66d
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<c0130a14>] try_to_del_timer_sync+0x44/0x4a
[ 694.121182] [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] kjournald D 00000000 0 2485 2
[ 694.121182] f6a651c0 00000046 c011f085 00000000 00000003 f6a6534c c3845020 00000004
[ 694.121182] 00000001 00013f05 c012126f 00000000 00000000 00000000 0000030c ffffffff
[ 694.121182] f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f6a651c0 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c011f085>] __wake_up_common+0x2e/0x58
[ 694.121182] [<c012126f>] __wake_up+0x29/0x39
[ 694.121182] [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[ 694.121182] [<c0161cc3>] get_page_from_freelist+0xc1/0x3e9
[ 694.121182] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 694.121182] [<c015fc05>] mempool_alloc+0x1c/0xba
[ 694.121182] [<c01dd76c>] submit_bio+0xc6/0xcd
[ 694.121182] [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[ 694.121182] [<c0199e27>] submit_bh+0xcf/0xed
[ 694.121182] [<c019ccab>] sync_dirty_buffer+0x4d/0x91
[ 694.121182] [<f8c59ed6>] journal_commit_transaction+0x82b/0xb3d [jbd]
[ 694.121182] [<c01309b4>] lock_timer_base+0x19/0x35
[ 694.121182] [<c0130a14>] try_to_del_timer_sync+0x44/0x4a
[ 694.121182] [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[ 694.121182] [<c013876b>] kthread+0x38/0x5d
[ 694.121182] [<c0138733>] kthread+0x0/0x5d
[ 694.121182] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 694.121182] =======================
[ 694.121182] summer D 00000000 0 3433 3430
[ 694.121182] f7806560 00000082 f6827a90 00000000 00000003 f78066ec c3866020 00000007
[ 694.121182] 00000001 00011aaa c012126f 00000000 00000000 00000000 00000000 ffffffff
[ 694.121182] f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f7806560 c01222b0
[ 694.121182] Call Trace:
[ 694.121182] [<c012126f>] __wake_up+0x29/0x39
[ 694.121182] [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[ 694.121182] [<c01222b0>] default_wake_function+0x0/0x8
[ 694.121182] [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[ 694.121182] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 694.121182] [<c01dd76c>] submit_bio+0xc6/0xcd
[ 694.121182] [<c015dbd9>] add_to_page_cache+0x64/0x7d
[ 694.121182] [<c01a126d>] mpage_end_io_read+0x0/0x53
[ 694.121182] [<c01a0546>] mpage_bio_submit+0x19/0x1d
[ 694.121182] [<c01a136e>] mpage_readpages+0xae/0xb8
[ 694.121182] [<f8c8ba5f>] ext3_readpages+0x0/0x15 [ext3]
[ 694.121182] [<c0163e40>] __do_page_cache_readahead+0xea/0x163
[ 694.121182] [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[ 694.121182] [<c016418d>] page_cache_sync_readahead+0x15/0x19
[ 694.121182] [<c015f429>] generic_file_aio_read+0x200/0x4e1
[ 694.121182] [<c016e9a7>] vma_adjust+0x1c6/0x399
[ 694.121182] [<c017e4d2>] do_sync_read+0xbf/0xfe
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<c01c4da9>] security_file_permission+0xc/0xd
[ 694.121182] [<c017e413>] do_sync_read+0x0/0xfe
[ 694.121182] [<c017ec62>] vfs_read+0x81/0x11e
[ 694.121182] [<c017f0b3>] sys_read+0x3c/0x63
[ 694.121182] [<c0108857>] sysenter_past_esp+0x78/0xb1
[ 694.121182] =======================
[ 694.121182] ls D f48dc588 0 3465 3453
[ 694.121182] f7670a00 00000082 f48dc4e0 f48dc588 f3b735c0 f7670b8c c3850020 00000005
[ 694.121182] f6390b40 00011d6d c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f4c5be70 f60ff2c0 c3802624 f7c4047c f8c58617 00000000 f48fc70c 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c015d5fa>] generic_file_mmap+0x2a/0x3e
[ 694.121182] [<c016f604>] mmap_region+0x1c7/0x392
[ 694.121182] [<c016fb07>] do_mmap_pgoff+0x25d/0x2b0
[ 694.121182] [<c0167004>] sys_mmap_pgoff+0x9b/0xc2
[ 694.121182] [<c01088d2>] syscall_call+0x7/0xb
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] bash D f3305be0 0 3466 3132
[ 694.121182] f597ca00 00000082 f1d7ac50 f3305be0 f41836a0 f597cb8c c385b020 00000006
[ 694.121182] f6390ae0 00012e32 c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f5473d8c f60ff2c0 c380080c f7deeafc f8c58617 00000000 f48fcaa4 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[ 694.121182] [<c017e4d2>] do_sync_read+0xbf/0xfe
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c98905>] ext3_xattr_security_get+0x31/0x3b [ext3]
[ 694.121182] [<c01c4da9>] security_file_permission+0xc/0xd
[ 694.121182] [<c017e413>] do_sync_read+0x0/0xfe
[ 694.121182] [<c017ec62>] vfs_read+0x81/0x11e
[ 694.121182] [<c0181c56>] kernel_read+0x32/0x43
[ 694.121182] [<c0181d27>] prepare_binprm+0xc0/0xc4
[ 694.121182] [<c0182b07>] do_execve+0xd2/0x1c6
[ 694.121182] [<c010713b>] sys_execve+0x2a/0x4a
[ 694.121182] [<c0108857>] sysenter_past_esp+0x78/0xb1
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] cat D f3305ba8 0 3467 3311
[ 694.121182] f5824f20 00000086 f7031668 f3305ba8 f1d7ac88 f58250ac c385b020 00000006
[ 694.121182] f6390b40 000144cd c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f4cdbe70 f60ff2c0 c3802624 f7c4047c f8c58617 00000000 f48fcbf8 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c015d5fa>] generic_file_mmap+0x2a/0x3e
[ 694.121182] [<c016f604>] mmap_region+0x1c7/0x392
[ 694.121182] [<c016fb07>] do_mmap_pgoff+0x25d/0x2b0
[ 694.121182] [<c0167004>] sys_mmap_pgoff+0x9b/0xc2
[ 694.121182] [<c01088d2>] syscall_call+0x7/0xb
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] cron D f71b3908 0 3468 2969
[ 694.121182] f7b23200 00000082 f7057160 f71b3908 f71074a8 f7b2338c c385b020 00000006
[ 694.121182] f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f54cbd80 f60ff2c0 c3800c74 f7dec6ec f8c58617 00000000 f48fca04 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c01864e2>] __link_path_walk+0x7d8/0xb24
[ 694.121182] [<c018d022>] __d_lookup+0x98/0xd7
[ 694.121182] [<c0186865>] path_walk+0x37/0x70
[ 694.121182] [<c0186b14>] do_path_lookup+0x122/0x184
[ 694.121182] [<c01874f6>] __path_lookup_intent_open+0x42/0x72
[ 694.121182] [<c0187575>] path_lookup_open+0xf/0x13
[ 694.121182] [<c0187600>] do_filp_open+0x87/0x684
[ 694.121182] [<c011b3e1>] do_page_fault+0x4b2/0x8f9
[ 694.121182] [<c011b448>] do_page_fault+0x519/0x8f9
[ 694.121182] [<c017ce8c>] do_sys_open+0x40/0xb0
[ 694.121182] [<c017cf40>] sys_open+0x1e/0x23
[ 694.121182] [<c01088d2>] syscall_call+0x7/0xb
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] cron D f1e4ce80 0 3469 2969
[ 694.121182] f7661620 00000082 f3b3b390 f1e4ce80 f1e4cef0 f76617ac c3866020 00000007
[ 694.121182] f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f442ddc8 f60ff2c0 c3802b10 f7c40684 f8c58617 00000000 f48fca18 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[ 694.121182] [<c017e4d2>] do_sync_read+0xbf/0xfe
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<c01c4da9>] security_file_permission+0xc/0xd
[ 694.121182] [<c017e413>] do_sync_read+0x0/0xfe
[ 694.121182] [<c017ec62>] vfs_read+0x81/0x11e
[ 694.121182] [<c017f0b3>] sys_read+0x3c/0x63
[ 694.121182] [<c01088d2>] syscall_call+0x7/0xb
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] cron D 00000002 0 3470 2969
[ 694.121182] f76605a0 00000086 f799eaf8 00000002 00000000 f766072c c3845020 00000004
[ 694.121182] 00000000 000144ce 0003404e 00000000 00000000 00000000 00000000 ffffffff
[ 694.121182] f613ec38 f613ec00 f4433e00 f44a7cc0 f8c58a05 f48fc360 00000002 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<f8c58a05>] start_this_handle+0x23b/0x2aa [jbd]
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<f8c58af1>] journal_start+0x7d/0xa9 [jbd]
[ 694.121182] [<f8c8ded9>] ext3_dirty_inode+0x21/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[ 694.121182] [<c017e4d2>] do_sync_read+0xbf/0xfe
[ 694.121182] [<c013882c>] autoremove_wake_function+0x0/0x2d
[ 694.121182] [<c018c410>] fcntl_setlk+0x202/0x22a
[ 694.121182] [<c01c4da9>] security_file_permission+0xc/0xd
[ 694.121182] [<c017e413>] do_sync_read+0x0/0xfe
[ 694.121182] [<c017ec62>] vfs_read+0x81/0x11e
[ 694.121182] [<c017f0b3>] sys_read+0x3c/0x63
[ 694.121182] [<c0108857>] sysenter_past_esp+0x78/0xb1
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] sh D f7c2cef0 0 3472 3471
[ 694.121182] f7b9ce20 00000086 f7c11e10 f7c2cef0 f7c2c2b0 f7b9cfac c3866020 00000007
[ 694.121182] f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff
[ 694.121182] f4547d80 f60ff2c0 c3800c74 f7dec6ec f8c58617 00000000 f48fc5f4 00000000
[ 694.121182] Call Trace:
[ 694.121182] [<c019a2ea>] __getblk+0x27/0x24e
[ 694.121182] [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[ 694.121182] [<c0138859>] wake_bit_function+0x0/0x3c
[ 694.121182] [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[ 694.121182] [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[ 694.121182] [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[ 694.121182] [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[ 694.121182] [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[ 694.121182] [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[ 694.121182] [<c018f21e>] touch_atime+0xc7/0xd1
[ 694.121182] [<c01864e2>] __link_path_walk+0x7d8/0xb24
[ 694.121182] [<c018d022>] __d_lookup+0x98/0xd7
[ 694.121182] [<c0186865>] path_walk+0x37/0x70
[ 694.121182] [<c0186b14>] do_path_lookup+0x122/0x184
[ 694.121182] [<c01874f6>] __path_lookup_intent_open+0x42/0x72
[ 694.121182] [<c0187575>] path_lookup_open+0xf/0x13
[ 694.121182] [<c0187600>] do_filp_open+0x87/0x684
[ 694.121182] [<c011b3e1>] do_page_fault+0x4b2/0x8f9
[ 694.121182] [<c011b448>] do_page_fault+0x519/0x8f9
[ 694.121182] [<c017ce8c>] do_sys_open+0x40/0xb0
[ 694.121182] [<c017cf40>] sys_open+0x1e/0x23
[ 694.121182] [<c01088d2>] syscall_call+0x7/0xb
[ 694.121182] [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[ 694.121182] =======================
[ 694.121182] Sched Debug Version: v0.07, 2.6.26-2-686-bigmem #1
[ 694.121182] now at 639513.043198 msecs
[ 694.121182] .sysctl_sched_latency : 80.000000
[ 694.121182] .sysctl_sched_min_granularity : 16.000000
[ 694.121182] .sysctl_sched_wakeup_granularity : 40.000000
[ 694.121182] .sysctl_sched_child_runs_first : 0.000001
[ 694.121182] .sysctl_sched_features : 895
[ 694.121182]
[ 694.121182] cpu#0, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 72329
[ 694.121182] .nr_load_updates : 18539
[ 694.121182] .nr_uninterruptible : 4294967243
[ 694.121182] .jiffies : 84895
[ 694.121182] .next_balance : 0.084896
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 639560.113202
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 26
[ 694.121182]
[ 694.121182] cfs_rq[0]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 16348.622085
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 0.000000
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#1, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 29556
[ 694.121182] .nr_load_updates : 5341
[ 694.121182] .nr_uninterruptible : 4294967270
[ 694.121182] .jiffies : 84932
[ 694.121182] .next_balance : 0.084934
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 639080.236160
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 0
[ 694.121182]
[ 694.121182] cfs_rq[1]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 7320.183309
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : -9028.438776
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#2, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 66329
[ 694.121182] .nr_load_updates : 5613
[ 694.121182] .nr_uninterruptible : 4294967199
[ 694.121182] .jiffies : 84970
[ 694.121182] .next_balance : 0.084972
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 639084.061939
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 60
[ 694.121182]
[ 694.121182] cfs_rq[2]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 12327.667563
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : -4020.954522
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#3, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 1898699
[ 694.121182] .nr_load_updates : 92084
[ 694.121182] .nr_uninterruptible : 4294966813
[ 694.121182] .jiffies : 85008
[ 694.121182] .next_balance : 0.085009
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 640051.975065
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 0
[ 694.121182]
[ 694.121182] cfs_rq[3]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 55749.380669
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 39400.758584
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#4, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 65011
[ 694.121182] .nr_load_updates : 7806
[ 694.121182] .nr_uninterruptible : 214
[ 694.121182] .jiffies : 85046
[ 694.121182] .next_balance : 0.085047
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 637517.396452
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 511
[ 694.121182] .cpu_load[2] : 750
[ 694.121182] .cpu_load[3] : 853
[ 694.121182] .cpu_load[4] : 868
[ 694.121182]
[ 694.121182] cfs_rq[4]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 45024.900733
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 28676.278648
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#5, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 34811
[ 694.121182] .nr_load_updates : 5207
[ 694.121182] .nr_uninterruptible : 128
[ 694.121182] .jiffies : 85084
[ 694.121182] .next_balance : 0.085085
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 640352.338449
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 30
[ 694.121182]
[ 694.121182] cfs_rq[5]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 26072.394124
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 9723.772039
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#6, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 32304
[ 694.121182] .nr_load_updates : 11575
[ 694.121182] .nr_uninterruptible : 172
[ 694.121182] .jiffies : 85121
[ 694.121182] .next_balance : 0.085124
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 640500.418721
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 0
[ 694.121182]
[ 694.121182] cfs_rq[6]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 37684.668142
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 21336.046057
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
[ 694.121182] cpu#7, 1862.045 MHz
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_switches : 2325749
[ 694.121182] .nr_load_updates : 10926
[ 694.121182] .nr_uninterruptible : 159
[ 694.121182] .jiffies : 85159
[ 694.121182] .next_balance : 0.085162
[ 694.121182] .curr->pid : 0
[ 694.121182] .clock : 640653.165765
[ 694.121182] .cpu_load[0] : 0
[ 694.121182] .cpu_load[1] : 0
[ 694.121182] .cpu_load[2] : 0
[ 694.121182] .cpu_load[3] : 0
[ 694.121182] .cpu_load[4] : 0
[ 694.121182]
[ 694.121182] cfs_rq[7]:
[ 694.121182] .exec_clock : 0.000000
[ 694.121182] .MIN_vruntime : 0.000001
[ 694.121182] .min_vruntime : 31973.568990
[ 694.121182] .max_vruntime : 0.000001
[ 694.121182] .spread : 0.000000
[ 694.121182] .spread0 : 15624.946905
[ 694.121182] .nr_running : 0
[ 694.121182] .load : 0
[ 694.121182] .nr_spread_over : 0
[ 694.121182]
[ 694.121182] runnable tasks:
[ 694.121182] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 694.121182] ----------------------------------------------------------------------------------------------------------
[ 694.121182]
*** break sent ***
[ 708.530043] SysRq : Show backtrace of all active CPUs
[ 708.530043] CPU4:
[ 708.530043]
[ 708.530043] Pid: 0, comm: swapper Tainted: G W (2.6.26-2-686-bigmem #1)
[ 708.530043] EIP: 0060:[<f884908c>] EFLAGS: 00000202 CPU: 4
[ 708.530043] EIP is at acpi_idle_enter_bm+0x2a0/0x315 [processor]
[ 708.530043] EAX: c03d5430 EBX: 00001033 ECX: 00005050 EDX: 0346d000
[ 708.530043] ESI: 00000000 EDI: 002cad1d EBP: 002c9cea ESP: f74b3f84
[ 708.530043] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 708.530043] CR0: 8005003b CR2: b77861be CR3: 00382000 CR4: 000006f0
[ 708.530043] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 708.530043] DR6: ffff0ff0 DR7: 00000400
[ 708.530043] [<c025b4f9>] ? cpuidle_idle_call+0x5b/0x86
[ 708.530043] [<c025b49e>] ? cpuidle_idle_call+0x0/0x86
[ 708.530043] [<c01075d3>] ? cpu_idle+0xb0/0xd0
[ 708.530043] =======================
*** break sent ***
[ 723.072717] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks
Reply to: