Re: rcu_sched self-detected stall on CPU { 0}
On Friday, March 21, 2014 14:22:18 Rainer Dorsch wrote:
> Hello,
>
> I do not remember when I see last term syslogd printing messages on the
> terminal in X (konsole in KDE to be precise). But I do remember that these
> messages indicated serious issues :0(
>
> The worst thing is I do not even understand what the system wanted to tell
> me. From the syslog, I can speculate on a rcu_sched self-detect stall
> (therefore I copied debian-kernel). I am running kernel 3.13 since a few
> days. Can anybody help here?
>
> I attach the console output and the relevant section of /var/log/syslog
>
> Console output:
>
> Message from syslogd@www at Fri Mar 21 14:28:50 2014 ...
> www vmunix: [118396.260001] CPU: 0 PID: 29 Comm: kswapd0 Tainted: G
> C O 3.13-1-686-pae #1 Debian 3.13.5-1
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260001] Hardware name: System manufacturer P5E-VM
> HDMI/P5E-VM HDMI, BIOS 0405 12/17/2007
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260001] task: f75344a0 ti: f76d6000 task.ti: f76d6000
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260001] Stack:
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260001] Call Trace:
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260001] Code: 00 48 75 fd 48 c3 8d 74 26 00 8d bc 27 00
> 00 00 00 ff 15 30 0c 59 c1 f3 c3 90 8d b4 26 00 00 00 00 c1 e0 02 64 8b 15
> 9c be 63 c1 <6b> d2 3e f7 e2 8d 42 01 ff 15 30 0c 59 c1 f3 c3 8d b6 00 00
> 00
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260211] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G
> C O 3.13-1-686-pae #1 Debian 3.13.5-1
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260214] Hardware name: System manufacturer P5E-VM
> HDMI/P5E-VM HDMI, BIOS 0405 12/17/2007
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260216] task: f7494da0 ti: f749c000 task.ti: f749c000
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260233] Stack:
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260250] Call Trace:
>
> Message from syslogd@www at Fri Mar 21 14:29:10 2014 ...
> www vmunix: [118396.260268] Code: 00 00 8d bc 27 00 00 00 00 fa c3 8d b4 26
> 00 00 00 00 8d bc 27 00 00 00 00 fb c3 8d b4 26 00 00 00 00 8d bc 27 00 00
> 00 00 fb f4 <c3> 8d b6 00 00 00 00 8d bc 27 00 00 00 00 f4 c3 8d b4 26 00
> 00
>
>
> Content of syslog:
>
> Mar 21 14:23:53 www ddclient[3535]: WARNING: file /etc/ddclient.conf, line
> 8: Invalid Value for keyword 'login' = ''
> Mar 21 14:26:50 www vmunix: [118276.128002] [sched_delayed] sched: RT
> throttling activated
> Mar 21 14:28:50 www vmunix: [118396.260001] INFO: rcu_sched self-detected
> stall on CPU { 0} (t=5250 jiffies g=2105139 c=2105138 q=2179)
> Mar 21 14:28:50 www vmunix: [118396.260001] sending NMI to all CPUs:
> Mar 21 14:28:50 www vmunix: [118396.260001] NMI backtrace for cpu 0
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] CPU: 0 PID: 29 Comm: kswapd0
> Tainted: G C O 3.13-1-686-pae #1 Debian 3.13.5-1
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] Hardware name: System
> manufacturer P5E-VM HDMI/P5E-VM HDMI, BIOS 0405 12/17/2007
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] task: f75344a0 ti: f76d6000
> task.ti: f76d6000
> Mar 21 14:29:10 www vmunix: [118396.260001] EIP: 0060:[<c122173a>] EFLAGS:
> 00000006 CPU: 0
> Mar 21 14:29:10 www vmunix: [118396.260001] EIP is at
> __const_udelay+0xa/0x20 Mar 21 14:29:10 www vmunix: [118396.260001] EAX:
> 01062560 EBX: 00002710 ECX: fffff000 EDX: 00a30dac
> Mar 21 14:29:10 www vmunix: [118396.260001] ESI: c1576440 EDI: c1576440
> EBP: 00000000 ESP: f76d7b7c
> Mar 21 14:29:10 www vmunix: [118396.260001] DS: 007b ES: 007b FS: 00d8 GS:
> 00e0 SS: 0068
> Mar 21 14:29:10 www vmunix: [118396.260001] CR0: 8005003b CR2: b6930d40
> CR3: 2c03a000 CR4: 000007f0
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] Stack:
> Mar 21 14:29:10 www vmunix: [118396.260001] c10392ed c14cccc9 f79b7840
> c109a39a c14dad94 00001482 00201f33 00201f32
> Mar 21 14:29:10 www vmunix: [118396.260001] 00000883 00006bae f79b7840
> c1576440 c15b31ac 00000000 00000000 00000082
> Mar 21 14:29:10 www vmunix: [118396.260001] f75344a0 00000000 00000000
> 00006bae c1059044 f76d7ca0 f76d7ca0 47052501
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] Call Trace:
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c10392ed>] ?
> arch_trigger_all_cpu_backtrace+0x4d/0x70
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c109a39a>] ?
> rcu_check_callbacks+0x37a/0x5b0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1059044>] ?
> update_process_times+0x34/0x60
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c10a48dc>] ?
> tick_sched_handle.isra.12+0x1c/0x50
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c10a493f>] ?
> tick_sched_timer+0x2f/0x60
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c106b487>] ?
> __remove_hrtimer+0x27/0x80
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c106b68b>] ?
> __run_hrtimer+0x6b/0x190
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c10a4910>] ?
> tick_sched_handle.isra.12+0x50/0x50
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c106c2a1>] ?
> hrtimer_interrupt+0x201/0x2c0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c11f6b03>] ?
> blk_queue_bio+0x103/0x2e0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1037467>] ?
> local_apic_timer_interrupt+0x27/0x50
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1051e5d>] ?
> irq_enter+0xd/0x60 Mar 21 14:29:10 www vmunix: [118396.260001]
> [<c10375fb>] ?
> smp_apic_timer_interrupt+0x2b/0x50
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c140fbec>] ?
> apic_timer_interrupt+0x34/0x3c
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c104007b>] ?
> kvm_read_and_reset_pf_reason+0x2b/0x40
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1127420>] ?
> scan_swap_map+0x120/0x400
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1127819>] ?
> get_swap_page+0x119/0x150
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1124c9c>] ?
> add_to_swap+0xc/0x60
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1102351>] ?
> shrink_page_list+0x4e1/0x8d0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1102c7a>] ?
> shrink_inactive_list+0x16a/0x460
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c110353a>] ?
> shrink_lruvec+0x2ca/0x5e0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c10736ed>] ?
> ttwu_do_wakeup+0xd/0xf0
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c11038a0>] ?
> shrink_zone+0x50/0x160
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c11046f7>] ?
> kswapd+0x487/0x950 Mar 21 14:29:10 www vmunix: [118396.260001]
> [<c1104270>] ?
> mem_cgroup_shrink_node_zone+0x100/0x100
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1068c6e>] ?
> kthread+0x9e/0xc0 Mar 21 14:29:10 www vmunix: [118396.260001]
> [<c14161f7>] ?
> ret_from_kernel_thread+0x1b/0x28
> Mar 21 14:29:10 www vmunix: [118396.260001] [<c1068bd0>] ?
> kthread_create_on_node+0x110/0x110
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260001] Code: 00 48 75 fd 48 c3 8d 74
> 26 00 8d bc 27 00 00 00 00 ff 15 30 0c 59 c1 f3 c3 90 8d b4 26 00 00 00 00
> c1 e0 02 64 8b 15 9c be 63 c1 <6b> d2 3e f7 e2 8d 42 01 ff 15 30 0c 59 c1
> f3 c3 8d b6 00 00 00
> Mar 21 14:29:10 www vmunix: [118396.260207] NMI backtrace for cpu 1
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260211] CPU: 1 PID: 0 Comm: swapper/1
> Tainted: G C O 3.13-1-686-pae #1 Debian 3.13.5-1
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260214] Hardware name: System
> manufacturer P5E-VM HDMI/P5E-VM HDMI, BIOS 0405 12/17/2007
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260216] task: f7494da0 ti: f749c000
> task.ti: f749c000
> Mar 21 14:29:10 www vmunix: [118396.260219] EIP: 0060:[<c1040692>] EFLAGS:
> 00200286 CPU: 1
> Mar 21 14:29:10 www vmunix: [118396.260223] EIP is at
> native_safe_halt+0x2/0x10
> Mar 21 14:29:10 www vmunix: [118396.260225] EAX: ffffffed EBX: edb4421f
> ECX: 01000000 EDX: 00000000
> Mar 21 14:29:10 www vmunix: [118396.260227] ESI: f749c000 EDI: 00000001
> EBP: 00000000 ESP: f749df68
> Mar 21 14:29:10 www vmunix: [118396.260229] DS: 007b ES: 007b FS: 00d8 GS:
> 00e0 SS: 0068
> Mar 21 14:29:10 www vmunix: [118396.260231] CR0: 8005003b CR2: 089510e0
> CR3: 2ce94000 CR4: 000007f0
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260233] Stack:
> Mar 21 14:29:10 www vmunix: [118396.260234] c1015cd4 edb4421f f749c000
> 00000000 c1016396 c10919a9 10fe296c edb4421f
> Mar 21 14:29:10 www vmunix: [118396.260240] 4d11762a 00000000 00000000
> 00000000 c1035b8e ffffffff 00000000 00000000
> Mar 21 14:29:10 www vmunix: [118396.260245] 00000000 9da374a5 4d11762a
> 01020800 00000000 00000000 00000000 00000000
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260250] Call Trace:
> Mar 21 14:29:10 www vmunix: [118396.260255] [<c1015cd4>] ?
> default_idle+0x14/0xa0
> Mar 21 14:29:10 www vmunix: [118396.260258] [<c1016396>] ?
> arch_cpu_idle+0x16/0x20
> Mar 21 14:29:10 www vmunix: [118396.260263] [<c10919a9>] ?
> cpu_startup_entry+0x1b9/0x200
> Mar 21 14:29:10 www vmunix: [118396.260266] [<c1035b8e>] ?
> start_secondary+0x1ce/0x240
> Mar 21 14:29:10 www syslogd: /dev/:0: No such file or directory
> Mar 21 14:29:10 www syslogd: /dev/:1: No such file or directory
> Mar 21 14:29:10 www vmunix: [118396.260268] Code: 00 00 8d bc 27 00 00 00
> 00 fa c3 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 fb c3 8d b4 26 00 00 00
> 00 8d bc 27 00 00 00 00 fb f4 <c3> 8d b6 00 00 00 00 8d bc 27 00 00 00 00
> f4 c3 8d b4 26 00 00
> Mar 21 14:29:10 www ddclient[3535]: WARNING: file /etc/ddclient.conf, line
> 8: Invalid Value for keyword 'login' = ''
> Mar 21 14:30:25 www vmunix: [118490.575735] [drm:intel_crtc_cursor_set],
> cursor off
>
> Any hint is welcome.
>
> Many thanks,
> Rainer
Very rough guess here - and I am probably wrong. The combination of the
tainted swap and the /dev messages would lead me to first check the hard
drive. Boot into a rescue disk and run fsck and see what you get.
Good luck, and I will be watching this thread to see what the answer is.
Mike
--
Mike McGinn KD2CNU
Ex Uno Plurima
No electrons were harmed in sending this message, some were inconvenienced.
** Registered Linux User 377849
Reply to: