Bug#787817: qla2xxx target - NULL pointer dereference
Hi,
I also tried tu use kernel 4.0.0-1 from sid. The FC target crashes
in the same time (after finishing transfer from Veeam; probably sync
at the end, because iostat shows much larger IO size), however the
trace is another, so I am attaching this.
Regards,
|
Tomasz Charoński
Storage Administrator
|
 |
Ul. M. Palacza 113
60-273 Poznań
tel.: +48 61 859 70 07
fax: +48 61 623 25 79
www.beyond.pl
biuro@beyond.pl
NIP: 782-23-24-152
KRS: 0000237620
Sąd Rejonowy Poznań-Nowe Miasto i Wilda w Poznaniu
VIII Wydział Gospodarczy Krajowego Rejestru Sądowego
Kapitał Zakładowy: 2 539 700,00 PLN
|
|
UWAGA: Ta wiadomość oraz wszelkie załączniki są
przeznaczone wyłącznie dla adresata korespondencji
e-mail. Jeśli otrzymałeś ten e-mail przez pomyłkę,
prosimy niezwłocznie powiadom nadawcę i usuń przesyłkę
ze swojego systemu wraz ze wszystkimi załącznikami.
Kopiowanie, dystrybucja, ujawnianie i inne
wykorzystanie przesyłki e-mail, bez zgody nadawcy,
jest zakazane.
|
--- Treść przekazanej wiadomości ---
Hello,
it's duplicate of: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787817
I have serious problem with qla2xxx target. After few days of
successful
working, there is a BUG in kernel, which causes FC target death.
It
happens repeatedly, usually few minutes after end of big transfer.
I am
attaching the dmesg.
May be related:
https://lists.debian.org/debian-kernel/2014/10/msg00247.html
(however at
me it doesn't appear at qlt_create_sess)
There was a lot of patches to qla2xxx in 3.17 and 3.18:
http://comments.gmane.org/gmane.linux.kernel.stable/111225
https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg33080.html
I am using Debian 8 Jessie with kernel 3.16.0-4-amd64.
Regards,
--
|
Tomasz Charoński
Storage Administrator
|
 |
Ul. M. Palacza 113
60-273 Poznań
tel.: +48 61 859 70 07
fax: +48 61 623 25 79
www.beyond.pl
biuro@beyond.pl
NIP: 782-23-24-152
KRS: 0000237620
Sąd Rejonowy Poznań-Nowe Miasto i Wilda w Poznaniu
VIII Wydział Gospodarczy Krajowego Rejestru Sądowego
Kapitał Zakładowy: 2 539 700,00 PLN
|
|
UWAGA: Ta wiadomość oraz wszelkie załączniki są
przeznaczone wyłącznie dla adresata korespondencji
e-mail. Jeśli otrzymałeś ten e-mail przez pomyłkę,
prosimy niezwłocznie powiadom nadawcę i usuń przesyłkę
ze swojego systemu wraz ze wszystkimi załącznikami.
Kopiowanie, dystrybucja, ujawnianie i inne
wykorzystanie przesyłki e-mail, bez zgody nadawcy,
jest zakazane.
|
|
Jun 11 11:45:41 debian8 kernel: [250981.461697] ------------[ cut here ]------------
Jun 11 11:45:41 debian8 kernel: [250981.461709] WARNING: CPU: 10 PID: 0 at /build/linux-Xbe5gu/linux-4.0.2/kernel/watchdog.c:290 watchdog_overflow_callback+0x92/0xc0()
Jun 11 11:45:41 debian8 kernel: [250981.461710] Watchdog detected hard LOCKUP on cpu 10
Jun 11 11:45:41 debian8 kernel: [250981.461712] Modules linked in: tcm_fc libfc ib_srpt ib_cm ib_sa ib_mad ib_core ib_addr iscsi_target_mod tcm_qla2xxx tcm_loop vhost_scsi vhost target_core_file target_core_iblock target_core_pscsi target_core_mod configfs ses enclosure mpt2sas dm_round_robin dm_multipath scsi_dh nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc bonding zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 iTCO_wdt lrw gf128mul drm iTCO_vendor_support glue_helper dm_mod joydev evdev ablk_helper cryptd pcspkr sb_edac edac_core mei_me lpc_ich mei i2c_i801 ipmi_si mfd_core ioatdma 8250_fintek ipmi_msghandler tpm_tis processor shpchp tpm thermal_sys wmi button autofs4 ext4 crc16 mbcache jbd2 sg sd_mod hid_generic usbhid hid isci crc32c_intel ahci ehci_pci libsas libahci qla2xxx ehci_hcd mpt3sas igb raid_class scsi_transport_fc libata i2c_algo_bit scsi_transport_sas i2c_core usbcore dca ptp usb_common pps_core scsi_mod
Jun 11 11:45:41 debian8 kernel: [250981.461772] CPU: 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.0-1-amd64 #1 Debian 4.0.2-1
Jun 11 11:45:41 debian8 kernel: [250981.461773] Hardware name: Supermicro SSG-2027R-AR24NV/X9DRH-IF-NV, BIOS 3.0 12/11/2013
Jun 11 11:45:41 debian8 kernel: [250981.461775] 0000000000000000 ffffffff8172ddf0 ffffffff8155df6b ffff88203f005b20
Jun 11 11:45:41 debian8 kernel: [250981.461777] ffffffff8106ce91 ffff882038741800 0000000000000000 ffff88203f005c40
Jun 11 11:45:41 debian8 kernel: [250981.461778] ffff88203f005ef8 0000000000000000 ffffffff8106cf0a ffffffff8172ddc8
Jun 11 11:45:41 debian8 kernel: [250981.461780] Call Trace:
Jun 11 11:45:41 debian8 kernel: [250981.461782] <NMI> [<ffffffff8155df6b>] ? dump_stack+0x40/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461793] [<ffffffff8106ce91>] ? warn_slowpath_common+0x81/0xb0
Jun 11 11:45:41 debian8 kernel: [250981.461794] [<ffffffff8106cf0a>] ? warn_slowpath_fmt+0x4a/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461796] [<ffffffff8110cb52>] ? watchdog_overflow_callback+0x92/0xc0
Jun 11 11:45:41 debian8 kernel: [250981.461801] [<ffffffff81149cd6>] ? __perf_event_overflow+0x86/0x230
Jun 11 11:45:41 debian8 kernel: [250981.461806] [<ffffffff8102a879>] ? x86_perf_event_set_period+0xc9/0x170
Jun 11 11:45:41 debian8 kernel: [250981.461809] [<ffffffff81031bba>] ? intel_pmu_handle_irq+0x1ba/0x3a0
Jun 11 11:45:41 debian8 kernel: [250981.461816] [<ffffffff812d431a>] ? ioremap_page_range+0x21a/0x320
Jun 11 11:45:41 debian8 kernel: [250981.461821] [<ffffffff8137d409>] ? ghes_copy_tofrom_phys+0x129/0x2c0
Jun 11 11:45:41 debian8 kernel: [250981.461822] [<ffffffff8137d611>] ? ghes_read_estatus+0x71/0x150
Jun 11 11:45:41 debian8 kernel: [250981.461824] [<ffffffff8102982a>] ? perf_event_nmi_handler+0x2a/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461829] [<ffffffff8101d166>] ? native_sched_clock+0x26/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461830] [<ffffffff81017e76>] ? nmi_handle+0x86/0x120
Jun 11 11:45:41 debian8 kernel: [250981.461832] [<ffffffff810183d4>] ? default_do_nmi+0x44/0x130
Jun 11 11:45:41 debian8 kernel: [250981.461833] [<ffffffff81018548>] ? do_nmi+0x88/0xd0
Jun 11 11:45:41 debian8 kernel: [250981.461838] [<ffffffff81566681>] ? end_repeat_nmi+0x1e/0x2e
Jun 11 11:45:41 debian8 kernel: [250981.461841] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461842] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461843] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461844] <<EOE>> <IRQ> [<ffffffffa03a8532>] ? qlt_fc_port_deleted+0x52/0xb0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461861] [<ffffffffa034da63>] ? qla2x00_mark_device_lost+0x153/0x2e0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461867] [<ffffffffa036bd2a>] ? qla2x00_async_event+0xd6a/0x17f0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461873] [<ffffffffa036d0ca>] ? qla24xx_msix_default+0x1ea/0x250 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461878] [<ffffffff810be885>] ? handle_irq_event_percpu+0x75/0x190
Jun 11 11:45:41 debian8 kernel: [250981.461880] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:45:41 debian8 kernel: [250981.461881] [<ffffffff810be9de>] ? handle_irq_event+0x3e/0x60
Jun 11 11:45:41 debian8 kernel: [250981.461884] [<ffffffff810c1ae1>] ? handle_edge_irq+0x81/0x140
Jun 11 11:45:41 debian8 kernel: [250981.461887] [<ffffffff8101649d>] ? handle_irq+0x1d/0x30
Jun 11 11:45:41 debian8 kernel: [250981.461891] [<ffffffff815670a6>] ? do_IRQ+0x46/0xe0
Jun 11 11:45:41 debian8 kernel: [250981.461893] [<ffffffff81564ead>] ? common_interrupt+0x6d/0x6d
Jun 11 11:45:41 debian8 kernel: [250981.461894] <EOI> [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:45:41 debian8 kernel: [250981.461902] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:45:41 debian8 kernel: [250981.461905] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:45:41 debian8 kernel: [250981.461908] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:45:41 debian8 kernel: [250981.461911] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:45:41 debian8 kernel: [250981.461912] ---[ end trace 82cffa7a20869247 ]---
Jun 11 11:45:41 debian8 kernel: [250991.901597] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 12, t=5252 jiffies, g=3003262, c=3003261, q=18101)
Jun 11 11:45:41 debian8 kernel: [250991.901674] Task dump for CPU 10:
Jun 11 11:45:41 debian8 kernel: [250991.901676] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:45:41 debian8 kernel: [250991.901678] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:45:41 debian8 kernel: [250991.901679] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:45:41 debian8 kernel: [250991.901681] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:45:41 debian8 kernel: [250991.901683] Call Trace:
Jun 11 11:45:41 debian8 kernel: [250991.901687] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:45:41 debian8 kernel: [250991.901689] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:45:41 debian8 kernel: [250991.901691] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:45:41 debian8 kernel: [250991.901693] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:45:41 debian8 kernel: [250991.901695] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:45:53 debian8 kernel: [251002.954745] sd 14:0:47:0: attempting task abort! scmd(ffff88146bfee540)
Jun 11 11:45:53 debian8 kernel: [251002.954751] sd 14:0:47:0: [sdau] CDB: Write(16) 8a 00 00 00 00 00 3a 81 c5 4b 00 00 00 09 00 00
Jun 11 11:45:53 debian8 kernel: [251002.954758] scsi target14:0:47: handle(0x003d), sas_address(0x5000cca027576842), phy(30)
Jun 11 11:45:53 debian8 kernel: [251002.954760] scsi target14:0:47: enclosure_logical_id(0x50030480015ce63f), slot(18)
Jun 11 11:45:53 debian8 kernel: [251002.954866] sd 15:0:36:0: attempting task abort! scmd(ffff8814b3274200)
Jun 11 11:45:53 debian8 kernel: [251002.954870] sd 15:0:36:0: [sdck] CDB: Write(16) 8a 00 00 00 00 00 20 11 e9 b4 00 00 00 04 00 00
Jun 11 11:45:53 debian8 kernel: [251002.954875] scsi target15:0:36: handle(0x0033), sas_address(0x5000cca0273ab5a9), phy(15)
Jun 11 11:45:53 debian8 kernel: [251002.954876] scsi target15:0:36: enclosure_logical_id(0x50030480015ce63f), slot(3)
Jun 11 11:45:53 debian8 kernel: [251002.956319] sd 14:0:47:0: task abort: SUCCESS scmd(ffff88146bfee540)
Jun 11 11:45:53 debian8 kernel: [251002.956667] sd 15:0:36:0: task abort: SUCCESS scmd(ffff8814b3274200)
Jun 11 11:46:01 debian8 CRON[15759]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:46:01 debian8 CRON[15760]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:46:45 debian8 kernel: [251054.859700] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 6, t=21007 jiffies, g=3003262, c=3003261, q=59964)
Jun 11 11:46:45 debian8 kernel: [251054.859787] Task dump for CPU 10:
Jun 11 11:46:45 debian8 kernel: [251054.859791] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:46:45 debian8 kernel: [251054.859796] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:46:45 debian8 kernel: [251054.859800] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:46:45 debian8 kernel: [251054.859804] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:46:45 debian8 kernel: [251054.859808] Call Trace:
Jun 11 11:46:45 debian8 kernel: [251054.859821] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:46:45 debian8 kernel: [251054.859826] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:46:45 debian8 kernel: [251054.859832] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:46:45 debian8 kernel: [251054.859837] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:46:45 debian8 kernel: [251054.859843] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:47:01 debian8 CRON[16368]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:47:01 debian8 CRON[16369]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:47:48 debian8 kernel: [251117.817778] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=36762 jiffies, g=3003262, c=3003261, q=95499)
Jun 11 11:47:48 debian8 kernel: [251117.817859] Task dump for CPU 10:
Jun 11 11:47:48 debian8 kernel: [251117.817862] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:47:48 debian8 kernel: [251117.817867] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:47:48 debian8 kernel: [251117.817870] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:47:48 debian8 kernel: [251117.817872] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:47:48 debian8 kernel: [251117.817875] Call Trace:
Jun 11 11:47:48 debian8 kernel: [251117.817888] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:47:48 debian8 kernel: [251117.817892] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:47:48 debian8 kernel: [251117.817897] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:47:48 debian8 kernel: [251117.817902] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:47:48 debian8 kernel: [251117.817908] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:48:02 debian8 CRON[16877]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:48:02 debian8 CRON[16878]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:48:24 debian8 kernel: [251154.054372] INFO: task txg_sync:32359 blocked for more than 120 seconds.
Jun 11 11:48:24 debian8 kernel: [251154.054409] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:48:24 debian8 kernel: [251154.054436] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:48:24 debian8 kernel: [251154.054482] txg_sync D ffff88203f214440 0 32359 2 0x00000000
Jun 11 11:48:24 debian8 kernel: [251154.054488] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0
Jun 11 11:48:24 debian8 kernel: [251154.054492] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10
Jun 11 11:48:24 debian8 kernel: [251154.054495] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:48:24 debian8 kernel: [251154.054499] Call Trace:
Jun 11 11:48:24 debian8 kernel: [251154.054512] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:48:24 debian8 kernel: [251154.054518] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:48:24 debian8 kernel: [251154.054526] [<ffffffff810a9824>] ? __wake_up_common+0x54/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054533] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054537] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:48:24 debian8 kernel: [251154.054556] [<ffffffffa02b5bd5>] ? cv_wait_common+0xa5/0x120 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054561] [<ffffffff810a9e80>] ? wait_woken+0x90/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054589] [<ffffffffa08fd35b>] ? zio_wait+0x11b/0x1f0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054612] [<ffffffffa088aa42>] ? dsl_pool_sync+0xb2/0x460 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054635] [<ffffffffa08a4236>] ? spa_sync+0x466/0xbb0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054640] [<ffffffff810a9a74>] ? __wake_up+0x34/0x50
Jun 11 11:48:24 debian8 kernel: [251154.054664] [<ffffffffa08b56a3>] ? txg_sync_thread+0x373/0x5c0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054668] [<ffffffff8109d5f7>] ? update_curr+0xd7/0x120
Jun 11 11:48:24 debian8 kernel: [251154.054690] [<ffffffffa08b5330>] ? txg_fini+0x270/0x270 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054697] [<ffffffffa02b1379>] ? thread_generic_wrapper+0x79/0x90 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054703] [<ffffffffa02b1300>] ? __thread_exit+0x20/0x20 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054707] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:48:24 debian8 kernel: [251154.054711] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054715] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054719] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054728] INFO: task kworker/10:1:5969 blocked for more than 120 seconds.
Jun 11 11:48:24 debian8 kernel: [251154.054758] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:48:24 debian8 kernel: [251154.054785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:48:24 debian8 kernel: [251154.054829] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000
Jun 11 11:48:24 debian8 kernel: [251154.054839] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx]
Jun 11 11:48:24 debian8 kernel: [251154.054841] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003
Jun 11 11:48:24 debian8 kernel: [251154.054844] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430
Jun 11 11:48:24 debian8 kernel: [251154.054847] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:48:24 debian8 kernel: [251154.054851] Call Trace:
Jun 11 11:48:24 debian8 kernel: [251154.054855] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:48:24 debian8 kernel: [251154.054859] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:48:24 debian8 kernel: [251154.054866] [<ffffffff812a288f>] ? generic_make_request_checks+0x1df/0x3c0
Jun 11 11:48:24 debian8 kernel: [251154.054870] [<ffffffff8101d1d5>] ? sched_clock+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054874] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054878] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:48:24 debian8 kernel: [251154.054882] [<ffffffff8156050d>] ? wait_for_completion_io+0xad/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054888] [<ffffffff81097450>] ? wake_up_state+0x20/0x20
Jun 11 11:48:24 debian8 kernel: [251154.054892] [<ffffffff812ad93b>] ? blkdev_issue_discard+0x20b/0x270
Jun 11 11:48:24 debian8 kernel: [251154.054899] [<ffffffffa05cd270>] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock]
Jun 11 11:48:24 debian8 kernel: [251154.054903] [<ffffffffa05cd28f>] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock]
Jun 11 11:48:24 debian8 kernel: [251154.054914] [<ffffffffa07cf46c>] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod]
Jun 11 11:48:24 debian8 kernel: [251154.054919] [<ffffffff810125cb>] ? __switch_to+0x14b/0x5d0
Jun 11 11:48:24 debian8 kernel: [251154.054927] [<ffffffffa07cc37a>] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod]
Jun 11 11:48:24 debian8 kernel: [251154.054931] [<ffffffff81084f82>] ? process_one_work+0x152/0x420
Jun 11 11:48:24 debian8 kernel: [251154.054934] [<ffffffff81085ab3>] ? worker_thread+0x53/0x540
Jun 11 11:48:24 debian8 kernel: [251154.054937] [<ffffffff81085a60>] ? rescuer_thread+0x3a0/0x3a0
Jun 11 11:48:24 debian8 kernel: [251154.054940] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:48:24 debian8 kernel: [251154.054944] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054948] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054951] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:51 debian8 kernel: [251180.787862] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 31, t=52520 jiffies, g=3003262, c=3003261, q=130454)
Jun 11 11:48:51 debian8 kernel: [251180.787941] Task dump for CPU 10:
Jun 11 11:48:51 debian8 kernel: [251180.787943] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:48:51 debian8 kernel: [251180.787946] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:48:51 debian8 kernel: [251180.787948] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:48:51 debian8 kernel: [251180.787949] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:48:51 debian8 kernel: [251180.787951] Call Trace:
Jun 11 11:48:51 debian8 kernel: [251180.787962] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:48:51 debian8 kernel: [251180.787964] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:48:51 debian8 kernel: [251180.787969] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:48:51 debian8 kernel: [251180.787972] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:48:51 debian8 kernel: [251180.787977] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
root@debian8:/sys/block/zd0/queue# grep -A 500 '11:45' /var/log/syslog
Jun 11 11:45:01 debian8 CRON[15171]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:45:01 debian8 CRON[15170]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:45:01 debian8 CRON[15172]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 11 11:45:41 debian8 kernel: [250981.461697] ------------[ cut here ]------------
Jun 11 11:45:41 debian8 kernel: [250981.461709] WARNING: CPU: 10 PID: 0 at /build/linux-Xbe5gu/linux-4.0.2/kernel/watchdog.c:290 watchdog_overflow_callback+0x92/0xc0()
Jun 11 11:45:41 debian8 kernel: [250981.461710] Watchdog detected hard LOCKUP on cpu 10
Jun 11 11:45:41 debian8 kernel: [250981.461712] Modules linked in: tcm_fc libfc ib_srpt ib_cm ib_sa ib_mad ib_core ib_addr iscsi_target_mod tcm_qla2xxx tcm_loop vhost_scsi vhost target_core_file target_core_iblock target_core_pscsi target_core_mod configfs ses enclosure mpt2sas dm_round_robin dm_multipath scsi_dh nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc bonding zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 iTCO_wdt lrw gf128mul drm iTCO_vendor_support glue_helper dm_mod joydev evdev ablk_helper cryptd pcspkr sb_edac edac_core mei_me lpc_ich mei i2c_i801 ipmi_si mfd_core ioatdma 8250_fintek ipmi_msghandler tpm_tis processor shpchp tpm thermal_sys wmi button autofs4 ext4 crc16 mbcache jbd2 sg sd_mod hid_generic usbhid hid isci crc32c_intel ahci ehci_pci libsas libahci qla2xxx ehci_hcd mpt3sas igb raid_class scsi_transport_fc libata i2c_algo_bit scsi_transport_sas i2c_core usbcore dca ptp usb_common pps_core scsi_mod
Jun 11 11:45:41 debian8 kernel: [250981.461772] CPU: 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.0-1-amd64 #1 Debian 4.0.2-1
Jun 11 11:45:41 debian8 kernel: [250981.461773] Hardware name: Supermicro SSG-2027R-AR24NV/X9DRH-IF-NV, BIOS 3.0 12/11/2013
Jun 11 11:45:41 debian8 kernel: [250981.461775] 0000000000000000 ffffffff8172ddf0 ffffffff8155df6b ffff88203f005b20
Jun 11 11:45:41 debian8 kernel: [250981.461777] ffffffff8106ce91 ffff882038741800 0000000000000000 ffff88203f005c40
Jun 11 11:45:41 debian8 kernel: [250981.461778] ffff88203f005ef8 0000000000000000 ffffffff8106cf0a ffffffff8172ddc8
Jun 11 11:45:41 debian8 kernel: [250981.461780] Call Trace:
Jun 11 11:45:41 debian8 kernel: [250981.461782] <NMI> [<ffffffff8155df6b>] ? dump_stack+0x40/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461793] [<ffffffff8106ce91>] ? warn_slowpath_common+0x81/0xb0
Jun 11 11:45:41 debian8 kernel: [250981.461794] [<ffffffff8106cf0a>] ? warn_slowpath_fmt+0x4a/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461796] [<ffffffff8110cb52>] ? watchdog_overflow_callback+0x92/0xc0
Jun 11 11:45:41 debian8 kernel: [250981.461801] [<ffffffff81149cd6>] ? __perf_event_overflow+0x86/0x230
Jun 11 11:45:41 debian8 kernel: [250981.461806] [<ffffffff8102a879>] ? x86_perf_event_set_period+0xc9/0x170
Jun 11 11:45:41 debian8 kernel: [250981.461809] [<ffffffff81031bba>] ? intel_pmu_handle_irq+0x1ba/0x3a0
Jun 11 11:45:41 debian8 kernel: [250981.461816] [<ffffffff812d431a>] ? ioremap_page_range+0x21a/0x320
Jun 11 11:45:41 debian8 kernel: [250981.461821] [<ffffffff8137d409>] ? ghes_copy_tofrom_phys+0x129/0x2c0
Jun 11 11:45:41 debian8 kernel: [250981.461822] [<ffffffff8137d611>] ? ghes_read_estatus+0x71/0x150
Jun 11 11:45:41 debian8 kernel: [250981.461824] [<ffffffff8102982a>] ? perf_event_nmi_handler+0x2a/0x50
Jun 11 11:45:41 debian8 kernel: [250981.461829] [<ffffffff8101d166>] ? native_sched_clock+0x26/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461830] [<ffffffff81017e76>] ? nmi_handle+0x86/0x120
Jun 11 11:45:41 debian8 kernel: [250981.461832] [<ffffffff810183d4>] ? default_do_nmi+0x44/0x130
Jun 11 11:45:41 debian8 kernel: [250981.461833] [<ffffffff81018548>] ? do_nmi+0x88/0xd0
Jun 11 11:45:41 debian8 kernel: [250981.461838] [<ffffffff81566681>] ? end_repeat_nmi+0x1e/0x2e
Jun 11 11:45:41 debian8 kernel: [250981.461841] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461842] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461843] [<ffffffff81563aa0>] ? _raw_spin_lock_irqsave+0x40/0x90
Jun 11 11:45:41 debian8 kernel: [250981.461844] <<EOE>> <IRQ> [<ffffffffa03a8532>] ? qlt_fc_port_deleted+0x52/0xb0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461861] [<ffffffffa034da63>] ? qla2x00_mark_device_lost+0x153/0x2e0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461867] [<ffffffffa036bd2a>] ? qla2x00_async_event+0xd6a/0x17f0 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461873] [<ffffffffa036d0ca>] ? qla24xx_msix_default+0x1ea/0x250 [qla2xxx]
Jun 11 11:45:41 debian8 kernel: [250981.461878] [<ffffffff810be885>] ? handle_irq_event_percpu+0x75/0x190
Jun 11 11:45:41 debian8 kernel: [250981.461880] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:45:41 debian8 kernel: [250981.461881] [<ffffffff810be9de>] ? handle_irq_event+0x3e/0x60
Jun 11 11:45:41 debian8 kernel: [250981.461884] [<ffffffff810c1ae1>] ? handle_edge_irq+0x81/0x140
Jun 11 11:45:41 debian8 kernel: [250981.461887] [<ffffffff8101649d>] ? handle_irq+0x1d/0x30
Jun 11 11:45:41 debian8 kernel: [250981.461891] [<ffffffff815670a6>] ? do_IRQ+0x46/0xe0
Jun 11 11:45:41 debian8 kernel: [250981.461893] [<ffffffff81564ead>] ? common_interrupt+0x6d/0x6d
Jun 11 11:45:41 debian8 kernel: [250981.461894] <EOI> [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:45:41 debian8 kernel: [250981.461902] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:45:41 debian8 kernel: [250981.461905] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:45:41 debian8 kernel: [250981.461908] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:45:41 debian8 kernel: [250981.461911] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:45:41 debian8 kernel: [250981.461912] ---[ end trace 82cffa7a20869247 ]---
Jun 11 11:45:41 debian8 kernel: [250991.901597] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 12, t=5252 jiffies, g=3003262, c=3003261, q=18101)
Jun 11 11:45:41 debian8 kernel: [250991.901674] Task dump for CPU 10:
Jun 11 11:45:41 debian8 kernel: [250991.901676] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:45:41 debian8 kernel: [250991.901678] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:45:41 debian8 kernel: [250991.901679] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:45:41 debian8 kernel: [250991.901681] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:45:41 debian8 kernel: [250991.901683] Call Trace:
Jun 11 11:45:41 debian8 kernel: [250991.901687] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:45:41 debian8 kernel: [250991.901689] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:45:41 debian8 kernel: [250991.901691] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:45:41 debian8 kernel: [250991.901693] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:45:41 debian8 kernel: [250991.901695] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:45:53 debian8 kernel: [251002.954745] sd 14:0:47:0: attempting task abort! scmd(ffff88146bfee540)
Jun 11 11:45:53 debian8 kernel: [251002.954751] sd 14:0:47:0: [sdau] CDB: Write(16) 8a 00 00 00 00 00 3a 81 c5 4b 00 00 00 09 00 00
Jun 11 11:45:53 debian8 kernel: [251002.954758] scsi target14:0:47: handle(0x003d), sas_address(0x5000cca027576842), phy(30)
Jun 11 11:45:53 debian8 kernel: [251002.954760] scsi target14:0:47: enclosure_logical_id(0x50030480015ce63f), slot(18)
Jun 11 11:45:53 debian8 kernel: [251002.954866] sd 15:0:36:0: attempting task abort! scmd(ffff8814b3274200)
Jun 11 11:45:53 debian8 kernel: [251002.954870] sd 15:0:36:0: [sdck] CDB: Write(16) 8a 00 00 00 00 00 20 11 e9 b4 00 00 00 04 00 00
Jun 11 11:45:53 debian8 kernel: [251002.954875] scsi target15:0:36: handle(0x0033), sas_address(0x5000cca0273ab5a9), phy(15)
Jun 11 11:45:53 debian8 kernel: [251002.954876] scsi target15:0:36: enclosure_logical_id(0x50030480015ce63f), slot(3)
Jun 11 11:45:53 debian8 kernel: [251002.956319] sd 14:0:47:0: task abort: SUCCESS scmd(ffff88146bfee540)
Jun 11 11:45:53 debian8 kernel: [251002.956667] sd 15:0:36:0: task abort: SUCCESS scmd(ffff8814b3274200)
Jun 11 11:46:01 debian8 CRON[15759]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:46:01 debian8 CRON[15760]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:46:45 debian8 kernel: [251054.859700] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 6, t=21007 jiffies, g=3003262, c=3003261, q=59964)
Jun 11 11:46:45 debian8 kernel: [251054.859787] Task dump for CPU 10:
Jun 11 11:46:45 debian8 kernel: [251054.859791] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:46:45 debian8 kernel: [251054.859796] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:46:45 debian8 kernel: [251054.859800] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:46:45 debian8 kernel: [251054.859804] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:46:45 debian8 kernel: [251054.859808] Call Trace:
Jun 11 11:46:45 debian8 kernel: [251054.859821] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:46:45 debian8 kernel: [251054.859826] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:46:45 debian8 kernel: [251054.859832] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:46:45 debian8 kernel: [251054.859837] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:46:45 debian8 kernel: [251054.859843] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:47:01 debian8 CRON[16368]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:47:01 debian8 CRON[16369]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:47:48 debian8 kernel: [251117.817778] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=36762 jiffies, g=3003262, c=3003261, q=95499)
Jun 11 11:47:48 debian8 kernel: [251117.817859] Task dump for CPU 10:
Jun 11 11:47:48 debian8 kernel: [251117.817862] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:47:48 debian8 kernel: [251117.817867] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:47:48 debian8 kernel: [251117.817870] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:47:48 debian8 kernel: [251117.817872] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:47:48 debian8 kernel: [251117.817875] Call Trace:
Jun 11 11:47:48 debian8 kernel: [251117.817888] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:47:48 debian8 kernel: [251117.817892] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:47:48 debian8 kernel: [251117.817897] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:47:48 debian8 kernel: [251117.817902] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:47:48 debian8 kernel: [251117.817908] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:48:02 debian8 CRON[16877]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:48:02 debian8 CRON[16878]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:48:24 debian8 kernel: [251154.054372] INFO: task txg_sync:32359 blocked for more than 120 seconds.
Jun 11 11:48:24 debian8 kernel: [251154.054409] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:48:24 debian8 kernel: [251154.054436] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:48:24 debian8 kernel: [251154.054482] txg_sync D ffff88203f214440 0 32359 2 0x00000000
Jun 11 11:48:24 debian8 kernel: [251154.054488] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0
Jun 11 11:48:24 debian8 kernel: [251154.054492] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10
Jun 11 11:48:24 debian8 kernel: [251154.054495] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:48:24 debian8 kernel: [251154.054499] Call Trace:
Jun 11 11:48:24 debian8 kernel: [251154.054512] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:48:24 debian8 kernel: [251154.054518] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:48:24 debian8 kernel: [251154.054526] [<ffffffff810a9824>] ? __wake_up_common+0x54/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054533] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054537] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:48:24 debian8 kernel: [251154.054556] [<ffffffffa02b5bd5>] ? cv_wait_common+0xa5/0x120 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054561] [<ffffffff810a9e80>] ? wait_woken+0x90/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054589] [<ffffffffa08fd35b>] ? zio_wait+0x11b/0x1f0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054612] [<ffffffffa088aa42>] ? dsl_pool_sync+0xb2/0x460 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054635] [<ffffffffa08a4236>] ? spa_sync+0x466/0xbb0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054640] [<ffffffff810a9a74>] ? __wake_up+0x34/0x50
Jun 11 11:48:24 debian8 kernel: [251154.054664] [<ffffffffa08b56a3>] ? txg_sync_thread+0x373/0x5c0 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054668] [<ffffffff8109d5f7>] ? update_curr+0xd7/0x120
Jun 11 11:48:24 debian8 kernel: [251154.054690] [<ffffffffa08b5330>] ? txg_fini+0x270/0x270 [zfs]
Jun 11 11:48:24 debian8 kernel: [251154.054697] [<ffffffffa02b1379>] ? thread_generic_wrapper+0x79/0x90 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054703] [<ffffffffa02b1300>] ? __thread_exit+0x20/0x20 [spl]
Jun 11 11:48:24 debian8 kernel: [251154.054707] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:48:24 debian8 kernel: [251154.054711] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054715] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054719] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054728] INFO: task kworker/10:1:5969 blocked for more than 120 seconds.
Jun 11 11:48:24 debian8 kernel: [251154.054758] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:48:24 debian8 kernel: [251154.054785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:48:24 debian8 kernel: [251154.054829] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000
Jun 11 11:48:24 debian8 kernel: [251154.054839] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx]
Jun 11 11:48:24 debian8 kernel: [251154.054841] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003
Jun 11 11:48:24 debian8 kernel: [251154.054844] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430
Jun 11 11:48:24 debian8 kernel: [251154.054847] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:48:24 debian8 kernel: [251154.054851] Call Trace:
Jun 11 11:48:24 debian8 kernel: [251154.054855] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:48:24 debian8 kernel: [251154.054859] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:48:24 debian8 kernel: [251154.054866] [<ffffffff812a288f>] ? generic_make_request_checks+0x1df/0x3c0
Jun 11 11:48:24 debian8 kernel: [251154.054870] [<ffffffff8101d1d5>] ? sched_clock+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054874] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:48:24 debian8 kernel: [251154.054878] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:48:24 debian8 kernel: [251154.054882] [<ffffffff8156050d>] ? wait_for_completion_io+0xad/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054888] [<ffffffff81097450>] ? wake_up_state+0x20/0x20
Jun 11 11:48:24 debian8 kernel: [251154.054892] [<ffffffff812ad93b>] ? blkdev_issue_discard+0x20b/0x270
Jun 11 11:48:24 debian8 kernel: [251154.054899] [<ffffffffa05cd270>] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock]
Jun 11 11:48:24 debian8 kernel: [251154.054903] [<ffffffffa05cd28f>] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock]
Jun 11 11:48:24 debian8 kernel: [251154.054914] [<ffffffffa07cf46c>] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod]
Jun 11 11:48:24 debian8 kernel: [251154.054919] [<ffffffff810125cb>] ? __switch_to+0x14b/0x5d0
Jun 11 11:48:24 debian8 kernel: [251154.054927] [<ffffffffa07cc37a>] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod]
Jun 11 11:48:24 debian8 kernel: [251154.054931] [<ffffffff81084f82>] ? process_one_work+0x152/0x420
Jun 11 11:48:24 debian8 kernel: [251154.054934] [<ffffffff81085ab3>] ? worker_thread+0x53/0x540
Jun 11 11:48:24 debian8 kernel: [251154.054937] [<ffffffff81085a60>] ? rescuer_thread+0x3a0/0x3a0
Jun 11 11:48:24 debian8 kernel: [251154.054940] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:48:24 debian8 kernel: [251154.054944] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:24 debian8 kernel: [251154.054948] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:48:24 debian8 kernel: [251154.054951] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:48:51 debian8 kernel: [251180.787862] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 31, t=52520 jiffies, g=3003262, c=3003261, q=130454)
Jun 11 11:48:51 debian8 kernel: [251180.787941] Task dump for CPU 10:
Jun 11 11:48:51 debian8 kernel: [251180.787943] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:48:51 debian8 kernel: [251180.787946] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:48:51 debian8 kernel: [251180.787948] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:48:51 debian8 kernel: [251180.787949] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:48:51 debian8 kernel: [251180.787951] Call Trace:
Jun 11 11:48:51 debian8 kernel: [251180.787962] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:48:51 debian8 kernel: [251180.787964] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:48:51 debian8 kernel: [251180.787969] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:48:51 debian8 kernel: [251180.787972] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:48:51 debian8 kernel: [251180.787977] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:49:01 debian8 CRON[17352]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:49:01 debian8 CRON[17353]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:49:54 debian8 kernel: [251243.749946] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 14, t=68276 jiffies, g=3003262, c=3003261, q=171658)
Jun 11 11:49:54 debian8 kernel: [251243.750025] Task dump for CPU 10:
Jun 11 11:49:54 debian8 kernel: [251243.750028] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:49:54 debian8 kernel: [251243.750031] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:49:54 debian8 kernel: [251243.750033] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:49:54 debian8 kernel: [251243.750034] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:49:54 debian8 kernel: [251243.750036] Call Trace:
Jun 11 11:49:54 debian8 kernel: [251243.750047] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:49:54 debian8 kernel: [251243.750049] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:49:54 debian8 kernel: [251243.750053] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:49:54 debian8 kernel: [251243.750057] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:49:54 debian8 kernel: [251243.750061] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:50:01 debian8 CRON[17909]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:50:01 debian8 CRON[17910]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:50:24 debian8 kernel: [251273.936547] INFO: task txg_sync:32359 blocked for more than 120 seconds.
Jun 11 11:50:24 debian8 kernel: [251273.936585] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:50:24 debian8 kernel: [251273.936612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:50:24 debian8 kernel: [251273.936659] txg_sync D ffff88203f214440 0 32359 2 0x00000000
Jun 11 11:50:24 debian8 kernel: [251273.936666] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0
Jun 11 11:50:24 debian8 kernel: [251273.936670] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10
Jun 11 11:50:24 debian8 kernel: [251273.936674] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:50:24 debian8 kernel: [251273.936678] Call Trace:
Jun 11 11:50:24 debian8 kernel: [251273.936693] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:50:24 debian8 kernel: [251273.936699] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:50:24 debian8 kernel: [251273.936708] [<ffffffff810a9824>] ? __wake_up_common+0x54/0x90
Jun 11 11:50:24 debian8 kernel: [251273.936716] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:50:24 debian8 kernel: [251273.936721] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:50:24 debian8 kernel: [251273.936743] [<ffffffffa02b5bd5>] ? cv_wait_common+0xa5/0x120 [spl]
Jun 11 11:50:24 debian8 kernel: [251273.936749] [<ffffffff810a9e80>] ? wait_woken+0x90/0x90
Jun 11 11:50:24 debian8 kernel: [251273.936780] [<ffffffffa08fd35b>] ? zio_wait+0x11b/0x1f0 [zfs]
Jun 11 11:50:24 debian8 kernel: [251273.936806] [<ffffffffa088aa42>] ? dsl_pool_sync+0xb2/0x460 [zfs]
Jun 11 11:50:24 debian8 kernel: [251273.936834] [<ffffffffa08a4236>] ? spa_sync+0x466/0xbb0 [zfs]
Jun 11 11:50:24 debian8 kernel: [251273.936840] [<ffffffff810a9a74>] ? __wake_up+0x34/0x50
Jun 11 11:50:24 debian8 kernel: [251273.936867] [<ffffffffa08b56a3>] ? txg_sync_thread+0x373/0x5c0 [zfs]
Jun 11 11:50:24 debian8 kernel: [251273.936872] [<ffffffff8109d5f7>] ? update_curr+0xd7/0x120
Jun 11 11:50:24 debian8 kernel: [251273.936898] [<ffffffffa08b5330>] ? txg_fini+0x270/0x270 [zfs]
Jun 11 11:50:24 debian8 kernel: [251273.936906] [<ffffffffa02b1379>] ? thread_generic_wrapper+0x79/0x90 [spl]
Jun 11 11:50:24 debian8 kernel: [251273.936913] [<ffffffffa02b1300>] ? __thread_exit+0x20/0x20 [spl]
Jun 11 11:50:24 debian8 kernel: [251273.936919] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:50:24 debian8 kernel: [251273.936924] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:50:24 debian8 kernel: [251273.936928] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:50:24 debian8 kernel: [251273.936932] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:50:24 debian8 kernel: [251273.936942] INFO: task kworker/10:1:5969 blocked for more than 120 seconds.
Jun 11 11:50:24 debian8 kernel: [251273.936973] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:50:24 debian8 kernel: [251273.937000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:50:24 debian8 kernel: [251273.937045] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000
Jun 11 11:50:24 debian8 kernel: [251273.937056] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx]
Jun 11 11:50:24 debian8 kernel: [251273.937058] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003
Jun 11 11:50:24 debian8 kernel: [251273.937062] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430
Jun 11 11:50:24 debian8 kernel: [251273.937066] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:50:24 debian8 kernel: [251273.937069] Call Trace:
Jun 11 11:50:24 debian8 kernel: [251273.937075] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:50:24 debian8 kernel: [251273.937080] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:50:24 debian8 kernel: [251273.937087] [<ffffffff812a288f>] ? generic_make_request_checks+0x1df/0x3c0
Jun 11 11:50:24 debian8 kernel: [251273.937092] [<ffffffff8101d1d5>] ? sched_clock+0x5/0x10
Jun 11 11:50:24 debian8 kernel: [251273.937096] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:50:24 debian8 kernel: [251273.937101] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:50:24 debian8 kernel: [251273.937106] [<ffffffff8156050d>] ? wait_for_completion_io+0xad/0x180
Jun 11 11:50:24 debian8 kernel: [251273.937113] [<ffffffff81097450>] ? wake_up_state+0x20/0x20
Jun 11 11:50:24 debian8 kernel: [251273.937118] [<ffffffff812ad93b>] ? blkdev_issue_discard+0x20b/0x270
Jun 11 11:50:24 debian8 kernel: [251273.937126] [<ffffffffa05cd270>] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock]
Jun 11 11:50:24 debian8 kernel: [251273.937131] [<ffffffffa05cd28f>] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock]
Jun 11 11:50:24 debian8 kernel: [251273.937143] [<ffffffffa07cf46c>] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod]
Jun 11 11:50:24 debian8 kernel: [251273.937149] [<ffffffff810125cb>] ? __switch_to+0x14b/0x5d0
Jun 11 11:50:24 debian8 kernel: [251273.937158] [<ffffffffa07cc37a>] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod]
Jun 11 11:50:24 debian8 kernel: [251273.937162] [<ffffffff81084f82>] ? process_one_work+0x152/0x420
Jun 11 11:50:24 debian8 kernel: [251273.937166] [<ffffffff81085ab3>] ? worker_thread+0x53/0x540
Jun 11 11:50:24 debian8 kernel: [251273.937170] [<ffffffff81085a60>] ? rescuer_thread+0x3a0/0x3a0
Jun 11 11:50:24 debian8 kernel: [251273.937174] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:50:24 debian8 kernel: [251273.937178] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:50:24 debian8 kernel: [251273.937182] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:50:24 debian8 kernel: [251273.937186] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:50:34 debian8 kernel: [251283.830751] sd 9:1:0:0: attempting task abort! scmd(ffff8820356ed540)
Jun 11 11:50:34 debian8 kernel: [251283.830762] sd 9:1:0:0: [sda] CDB: Write(10) 2a 00 05 c7 f5 20 00 00 98 00
Jun 11 11:50:34 debian8 kernel: [251283.830767] scsi target9:1:0: volume handle(0x0143), volume wwid(0x0e964964d5e3b6fa)
Jun 11 11:50:34 debian8 kernel: [251283.830772] sd 9:1:0:0: task abort: FAILED scmd(ffff8820356ed540)
Jun 11 11:50:34 debian8 kernel: [251283.830832] sd 9:1:0:0: attempting device reset! scmd(ffff8820356ed540)
Jun 11 11:50:34 debian8 kernel: [251283.830839] sd 9:1:0:0: [sda] CDB: Write(10) 2a 00 05 c7 f5 20 00 00 98 00
Jun 11 11:50:34 debian8 kernel: [251283.830843] scsi target9:1:0: volume handle(0x0143), volume wwid(0x0e964964d5e3b6fa)
Jun 11 11:50:34 debian8 kernel: [251283.880685] sd 9:1:0:0: device reset: SUCCESS scmd(ffff8820356ed540)
Jun 11 11:50:57 debian8 kernel: [251306.712033] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 13, t=84032 jiffies, g=3003262, c=3003261, q=200277)
Jun 11 11:50:57 debian8 kernel: [251306.712102] Task dump for CPU 10:
Jun 11 11:50:57 debian8 kernel: [251306.712104] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:50:57 debian8 kernel: [251306.712107] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:50:57 debian8 kernel: [251306.712109] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:50:57 debian8 kernel: [251306.712110] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:50:57 debian8 kernel: [251306.712112] Call Trace:
Jun 11 11:50:57 debian8 kernel: [251306.712123] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:50:57 debian8 kernel: [251306.712125] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:50:57 debian8 kernel: [251306.712129] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:50:57 debian8 kernel: [251306.712133] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:50:57 debian8 kernel: [251306.712138] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:51:01 debian8 CRON[18232]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:51:01 debian8 CRON[18233]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:52:00 debian8 kernel: [251369.674117] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=99788 jiffies, g=3003262, c=3003261, q=229197)
Jun 11 11:52:00 debian8 kernel: [251369.674196] Task dump for CPU 10:
Jun 11 11:52:00 debian8 kernel: [251369.674197] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:52:00 debian8 kernel: [251369.674201] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:52:00 debian8 kernel: [251369.674204] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:52:00 debian8 kernel: [251369.674207] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:52:00 debian8 kernel: [251369.674209] Call Trace:
Jun 11 11:52:00 debian8 kernel: [251369.674219] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:52:00 debian8 kernel: [251369.674222] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:52:00 debian8 kernel: [251369.674227] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:52:00 debian8 kernel: [251369.674231] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:52:00 debian8 kernel: [251369.674236] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Jun 11 11:52:01 debian8 CRON[18549]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:52:01 debian8 CRON[18550]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:52:24 debian8 kernel: [251393.818642] INFO: task txg_sync:32359 blocked for more than 120 seconds.
Jun 11 11:52:24 debian8 kernel: [251393.818680] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:52:24 debian8 kernel: [251393.818708] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:52:24 debian8 kernel: [251393.818754] txg_sync D ffff88203f214440 0 32359 2 0x00000000
Jun 11 11:52:24 debian8 kernel: [251393.818761] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0
Jun 11 11:52:24 debian8 kernel: [251393.818766] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10
Jun 11 11:52:24 debian8 kernel: [251393.818769] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:52:24 debian8 kernel: [251393.818773] Call Trace:
Jun 11 11:52:24 debian8 kernel: [251393.818788] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:52:24 debian8 kernel: [251393.818795] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:52:24 debian8 kernel: [251393.818804] [<ffffffff810a9824>] ? __wake_up_common+0x54/0x90
Jun 11 11:52:24 debian8 kernel: [251393.818812] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:52:24 debian8 kernel: [251393.818817] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:52:24 debian8 kernel: [251393.818838] [<ffffffffa02b5bd5>] ? cv_wait_common+0xa5/0x120 [spl]
Jun 11 11:52:24 debian8 kernel: [251393.818844] [<ffffffff810a9e80>] ? wait_woken+0x90/0x90
Jun 11 11:52:24 debian8 kernel: [251393.818874] [<ffffffffa08fd35b>] ? zio_wait+0x11b/0x1f0 [zfs]
Jun 11 11:52:24 debian8 kernel: [251393.818900] [<ffffffffa088aa42>] ? dsl_pool_sync+0xb2/0x460 [zfs]
Jun 11 11:52:24 debian8 kernel: [251393.818928] [<ffffffffa08a4236>] ? spa_sync+0x466/0xbb0 [zfs]
Jun 11 11:52:24 debian8 kernel: [251393.818934] [<ffffffff810a9a74>] ? __wake_up+0x34/0x50
Jun 11 11:52:24 debian8 kernel: [251393.818962] [<ffffffffa08b56a3>] ? txg_sync_thread+0x373/0x5c0 [zfs]
Jun 11 11:52:24 debian8 kernel: [251393.818966] [<ffffffff8109d5f7>] ? update_curr+0xd7/0x120
Jun 11 11:52:24 debian8 kernel: [251393.818992] [<ffffffffa08b5330>] ? txg_fini+0x270/0x270 [zfs]
Jun 11 11:52:24 debian8 kernel: [251393.819001] [<ffffffffa02b1379>] ? thread_generic_wrapper+0x79/0x90 [spl]
Jun 11 11:52:24 debian8 kernel: [251393.819007] [<ffffffffa02b1300>] ? __thread_exit+0x20/0x20 [spl]
Jun 11 11:52:24 debian8 kernel: [251393.819013] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:52:24 debian8 kernel: [251393.819018] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:52:24 debian8 kernel: [251393.819022] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:52:24 debian8 kernel: [251393.819026] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:52:24 debian8 kernel: [251393.819035] INFO: task kworker/10:1:5969 blocked for more than 120 seconds.
Jun 11 11:52:24 debian8 kernel: [251393.819066] Tainted: P W O 4.0.0-1-amd64 #1
Jun 11 11:52:24 debian8 kernel: [251393.819093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 11 11:52:24 debian8 kernel: [251393.819138] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000
Jun 11 11:52:24 debian8 kernel: [251393.819148] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx]
Jun 11 11:52:24 debian8 kernel: [251393.819150] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003
Jun 11 11:52:24 debian8 kernel: [251393.819154] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430
Jun 11 11:52:24 debian8 kernel: [251393.819158] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000
Jun 11 11:52:24 debian8 kernel: [251393.819162] Call Trace:
Jun 11 11:52:24 debian8 kernel: [251393.819167] [<ffffffff8155fb5f>] ? schedule+0x2f/0x80
Jun 11 11:52:24 debian8 kernel: [251393.819172] [<ffffffff81562dca>] ? schedule_timeout+0x21a/0x290
Jun 11 11:52:24 debian8 kernel: [251393.819180] [<ffffffff812a288f>] ? generic_make_request_checks+0x1df/0x3c0
Jun 11 11:52:24 debian8 kernel: [251393.819184] [<ffffffff8101d1d5>] ? sched_clock+0x5/0x10
Jun 11 11:52:24 debian8 kernel: [251393.819188] [<ffffffff8101cc45>] ? read_tsc+0x5/0x10
Jun 11 11:52:24 debian8 kernel: [251393.819193] [<ffffffff8155f183>] ? io_schedule_timeout+0xa3/0x130
Jun 11 11:52:24 debian8 kernel: [251393.819199] [<ffffffff8156050d>] ? wait_for_completion_io+0xad/0x180
Jun 11 11:52:24 debian8 kernel: [251393.819205] [<ffffffff81097450>] ? wake_up_state+0x20/0x20
Jun 11 11:52:24 debian8 kernel: [251393.819210] [<ffffffff812ad93b>] ? blkdev_issue_discard+0x20b/0x270
Jun 11 11:52:24 debian8 kernel: [251393.819219] [<ffffffffa05cd270>] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock]
Jun 11 11:52:24 debian8 kernel: [251393.819223] [<ffffffffa05cd28f>] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock]
Jun 11 11:52:24 debian8 kernel: [251393.819235] [<ffffffffa07cf46c>] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod]
Jun 11 11:52:24 debian8 kernel: [251393.819241] [<ffffffff810125cb>] ? __switch_to+0x14b/0x5d0
Jun 11 11:52:24 debian8 kernel: [251393.819251] [<ffffffffa07cc37a>] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod]
Jun 11 11:52:24 debian8 kernel: [251393.819255] [<ffffffff81084f82>] ? process_one_work+0x152/0x420
Jun 11 11:52:24 debian8 kernel: [251393.819258] [<ffffffff81085ab3>] ? worker_thread+0x53/0x540
Jun 11 11:52:24 debian8 kernel: [251393.819262] [<ffffffff81085a60>] ? rescuer_thread+0x3a0/0x3a0
Jun 11 11:52:24 debian8 kernel: [251393.819266] [<ffffffff8108aa41>] ? kthread+0xc1/0xe0
Jun 11 11:52:24 debian8 kernel: [251393.819271] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:52:24 debian8 kernel: [251393.819274] [<ffffffff815640d8>] ? ret_from_fork+0x58/0x90
Jun 11 11:52:24 debian8 kernel: [251393.819279] [<ffffffff8108a980>] ? kthread_create_on_node+0x180/0x180
Jun 11 11:53:01 debian8 CRON[18898]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out)
Jun 11 11:53:01 debian8 CRON[18899]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out)
Jun 11 11:53:03 debian8 kernel: [251432.636209] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 36, t=115544 jiffies, g=3003262, c=3003261, q=267550)
Jun 11 11:53:03 debian8 kernel: [251432.636287] Task dump for CPU 10:
Jun 11 11:53:03 debian8 kernel: [251432.636289] swapper/10 R running task 0 0 1 0x00000008
Jun 11 11:53:03 debian8 kernel: [251432.636292] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207
Jun 11 11:53:03 debian8 kernel: [251432.636294] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092
Jun 11 11:53:03 debian8 kernel: [251432.636295] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000
Jun 11 11:53:03 debian8 kernel: [251432.636297] Call Trace:
Jun 11 11:53:03 debian8 kernel: [251432.636307] [<ffffffff81432f51>] ? cpuidle_enter_state+0x61/0x160
Jun 11 11:53:03 debian8 kernel: [251432.636309] [<ffffffff81432f3e>] ? cpuidle_enter_state+0x4e/0x160
Jun 11 11:53:03 debian8 kernel: [251432.636313] [<ffffffff810aa57f>] ? cpu_startup_entry+0x36f/0x420
Jun 11 11:53:03 debian8 kernel: [251432.636317] [<ffffffff810dc910>] ? tick_check_new_device+0xe0/0x110
Jun 11 11:53:03 debian8 kernel: [251432.636322] [<ffffffff810466f9>] ? start_secondary+0x199/0x1c0
Attachment:
smime.p7s
Description: Kryptograficzna sygnatura S/MIME
Reply to: