Bug#617666: Acknowledgement (nfs-kernel-server: Periodic nfsd failure - single nfsd process with high CPU and no mounts working)
On 10/03/11 12:54, Debian Bug Tracking System wrote:
Hi all,
I have some extra information about this problem - the syslog contains
some kernel error messages related to nfs and xfs (the filesystem of the
/export partition). I have attached the relevant log section...
It could be this is a problem with xfs or even with our hardware raid
controller. I have rebooted the machine with /export unmounted and am
currently running xfs_repair over it to see if that picks up any problems.
Thanks,
Dan
--
Systems Administrator
Cambridge Systems Biology Centre (CSBC)
Tennis Court Road
Cambridge
CB2 1QR
01223 760252
Mar 10 08:30:56 fileserver2 kernel: [157603.877011] INFO: task nfsd:20040 blocked for more than 120 seconds.
Mar 10 08:30:56 fileserver2 kernel: [157603.877065] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:30:56 fileserver2 kernel: [157603.877149] nfsd D ffff88000544fb30 0 20040 2 0x00000000
Mar 10 08:30:56 fileserver2 kernel: [157603.877208] ffff88016fab8e20 0000000000000046 0000000000000000 ffff880167f1b110
Mar 10 08:30:56 fileserver2 kernel: [157603.877300] ffff880138b3ed00 0000000000000000 000000000000f9e0 ffff88016e269fd8
Mar 10 08:30:56 fileserver2 kernel: [157603.877392] 0000000000015780 0000000000015780 ffff88016d0c2a60 ffff88016d0c2d58
Mar 10 08:30:56 fileserver2 kernel: [157603.877485] Call Trace:
Mar 10 08:30:56 fileserver2 kernel: [157603.877547] [<ffffffffa020a895>] ? xfs_mod_incore_sb_batch+0x6f/0x101 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.877605] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:30:56 fileserver2 kernel: [157603.877675] [<ffffffffa021769f>] ? xfs_buf_rele+0x37/0xab [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.877728] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:30:56 fileserver2 kernel: [157603.877786] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:30:56 fileserver2 kernel: [157603.877846] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.877911] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.877976] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878042] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878109] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878174] [<ffffffffa01feed5>] ? xfs_iunlink_remove+0x4b/0x2a9 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878240] [<ffffffffa020fa53>] ? xfs_trans_log_inode+0x1a/0x42 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878307] [<ffffffffa01fe157>] ? xfs_itruncate_finish+0x284/0x295 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878375] [<ffffffffa01ff14b>] ? xfs_ifree+0x18/0xf2 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878438] [<ffffffffa02130a1>] ? xfs_inactive+0x30c/0x3f0 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.878492] [<ffffffff810fff23>] ? clear_inode+0x79/0xd0
Mar 10 08:30:56 fileserver2 kernel: [157603.878542] [<ffffffff8110066c>] ? generic_delete_inode+0xf4/0x168
Mar 10 08:30:56 fileserver2 kernel: [157603.878596] [<ffffffff810fe688>] ? d_delete+0x47/0xb2
Mar 10 08:30:56 fileserver2 kernel: [157603.878645] [<ffffffff810f7a81>] ? vfs_unlink+0x9f/0xa6
Mar 10 08:30:56 fileserver2 kernel: [157603.878700] [<ffffffffa031f02b>] ? nfsd_unlink+0x105/0x16c [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.878758] [<ffffffffa03262d1>] ? nfsd3_proc_remove+0x74/0x81 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.878816] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.878875] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:30:56 fileserver2 kernel: [157603.878932] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.878985] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.879035] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.879085] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.879134] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.879183] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.879231] INFO: task nfsd:20041 blocked for more than 120 seconds.
Mar 10 08:30:56 fileserver2 kernel: [157603.879283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:30:56 fileserver2 kernel: [157603.879366] nfsd D ffff88000548fa20 0 20041 2 0x00000000
Mar 10 08:30:56 fileserver2 kernel: [157603.879425] ffff88016fb08000 0000000000000046 0000000000000000 ffffffff810e4493
Mar 10 08:30:56 fileserver2 kernel: [157603.879517] ffffea0004635b78 ffffffff810e6870 000000000000f9e0 ffff88016e0dffd8
Mar 10 08:30:56 fileserver2 kernel: [157603.879610] 0000000000015780 0000000000015780 ffff88016d0c0e20 ffff88016d0c1118
Mar 10 08:30:56 fileserver2 kernel: [157603.879702] Call Trace:
Mar 10 08:30:56 fileserver2 kernel: [157603.879743] [<ffffffff810e4493>] ? get_partial_node+0x15/0x85
Mar 10 08:30:56 fileserver2 kernel: [157603.879795] [<ffffffff810e6870>] ? __slab_alloc+0xd3/0x39b
Mar 10 08:30:56 fileserver2 kernel: [157603.879846] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:30:56 fileserver2 kernel: [157603.879899] [<ffffffff811971bb>] ? __bitmap_weight+0x3a/0x7e
Mar 10 08:30:56 fileserver2 kernel: [157603.879952] [<ffffffff8118ddf2>] ? cpumask_next_and+0x2a/0x3a
Mar 10 08:30:56 fileserver2 kernel: [157603.880005] [<ffffffff8103a6e1>] ? scale_rt_power+0x1f/0x65
Mar 10 08:30:56 fileserver2 kernel: [157603.880056] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:30:56 fileserver2 kernel: [157603.880107] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:30:56 fileserver2 kernel: [157603.880167] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880231] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880296] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880362] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880429] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880494] [<ffffffffa01f9349>] ? xfs_ialloc_read_agi+0x11/0x50 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880560] [<ffffffffa01f961c>] ? xfs_ialloc_ag_select+0x1e4/0x281 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880617] [<ffffffff8126296f>] ? sch_direct_xmit+0x7f/0x14c
Mar 10 08:30:56 fileserver2 kernel: [157603.880681] [<ffffffffa01fa611>] ? xfs_dialloc+0x33/0x866 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.880733] [<ffffffff810e5513>] ? __slab_free+0x7f/0x27a
Mar 10 08:30:56 fileserver2 kernel: [157603.880784] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:30:56 fileserver2 kernel: [157603.880835] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:30:56 fileserver2 kernel: [157603.880886] [<ffffffff8127e4f7>] ? tcp_recvmsg+0x98b/0xa9e
Mar 10 08:30:56 fileserver2 kernel: [157603.880950] [<ffffffffa01fe321>] ? xfs_ialloc+0x61/0x562 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.881014] [<ffffffffa02103b8>] ? xfs_dir_ialloc+0x84/0x2a8 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.881080] [<ffffffffa020e737>] ? xfs_trans_reserve+0xc8/0x19d [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.881146] [<ffffffffa021246a>] ? xfs_create+0x30f/0x539 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.881199] [<ffffffff810fe556>] ? __d_lookup+0xc3/0x107
Mar 10 08:30:56 fileserver2 kernel: [157603.881260] [<ffffffffa021a99b>] ? xfs_vn_mknod+0xc3/0x15a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.881313] [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Mar 10 08:30:56 fileserver2 kernel: [157603.881367] [<ffffffffa0320b27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.881426] [<ffffffffa03266a0>] ? nfsd3_proc_create+0x116/0x127 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.881484] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.881542] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:30:56 fileserver2 kernel: [157603.881599] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.881658] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.881710] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.881768] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.881823] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.881879] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.881933] INFO: task nfsd:20044 blocked for more than 120 seconds.
Mar 10 08:30:56 fileserver2 kernel: [157603.881985] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:30:56 fileserver2 kernel: [157603.882068] nfsd D 0000000000000000 0 20044 2 0x00000000
Mar 10 08:30:56 fileserver2 kernel: [157603.882127] ffff88016fa7d4c0 0000000000000046 0000000000000000 0000000000000007
Mar 10 08:30:56 fileserver2 kernel: [157603.882219] ffff8801a3e54c78 ffff88026d85c000 000000000000f9e0 ffff88016e44dfd8
Mar 10 08:30:56 fileserver2 kernel: [157603.882311] 0000000000015780 0000000000015780 ffff88016d0c3170 ffff88016d0c3468
Mar 10 08:30:56 fileserver2 kernel: [157603.882403] Call Trace:
Mar 10 08:30:56 fileserver2 kernel: [157603.882455] [<ffffffffa020fc14>] ? xfs_trans_unlock_chunk+0x77/0xb6 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.882511] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:30:56 fileserver2 kernel: [157603.882563] [<ffffffff810b5908>] ? mempool_alloc+0x55/0x106
Mar 10 08:30:56 fileserver2 kernel: [157603.882614] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:30:56 fileserver2 kernel: [157603.882665] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:30:56 fileserver2 kernel: [157603.882724] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.882789] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.882854] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.882920] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.882987] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883051] [<ffffffffa01f9349>] ? xfs_ialloc_read_agi+0x11/0x50 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883118] [<ffffffffa01f961c>] ? xfs_ialloc_ag_select+0x1e4/0x281 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883174] [<ffffffff8126296f>] ? sch_direct_xmit+0x7f/0x14c
Mar 10 08:30:56 fileserver2 kernel: [157603.883237] [<ffffffffa01fa611>] ? xfs_dialloc+0x33/0x866 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883290] [<ffffffff810e5513>] ? __slab_free+0x7f/0x27a
Mar 10 08:30:56 fileserver2 kernel: [157603.883340] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:30:56 fileserver2 kernel: [157603.883391] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:30:56 fileserver2 kernel: [157603.883441] [<ffffffff8127e4f7>] ? tcp_recvmsg+0x98b/0xa9e
Mar 10 08:30:56 fileserver2 kernel: [157603.883505] [<ffffffffa01fe321>] ? xfs_ialloc+0x61/0x562 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883568] [<ffffffffa02103b8>] ? xfs_dir_ialloc+0x84/0x2a8 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883634] [<ffffffffa020e737>] ? xfs_trans_reserve+0xc8/0x19d [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883700] [<ffffffffa021246a>] ? xfs_create+0x30f/0x539 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883752] [<ffffffff810fe556>] ? __d_lookup+0xc3/0x107
Mar 10 08:30:56 fileserver2 kernel: [157603.883814] [<ffffffffa021a99b>] ? xfs_vn_mknod+0xc3/0x15a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.883866] [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Mar 10 08:30:56 fileserver2 kernel: [157603.883920] [<ffffffffa0320b27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.883979] [<ffffffffa03266a0>] ? nfsd3_proc_create+0x116/0x127 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.884037] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.884095] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:30:56 fileserver2 kernel: [157603.884152] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.884205] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.884255] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.884304] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.884353] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.884402] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.884450] INFO: task nfsd:20046 blocked for more than 120 seconds.
Mar 10 08:30:56 fileserver2 kernel: [157603.884502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:30:56 fileserver2 kernel: [157603.884585] nfsd D ffff88000548fa20 0 20046 2 0x00000000
Mar 10 08:30:56 fileserver2 kernel: [157603.884644] ffff88016fb08000 0000000000000046 0000000000000000 ffff88026dc94d00
Mar 10 08:30:56 fileserver2 kernel: [157603.884736] ffff88026cace69c ffff88026df28000 000000000000f9e0 ffff88015db8dfd8
Mar 10 08:30:56 fileserver2 kernel: [157603.892874] 0000000000015780 0000000000015780 ffff88016d0c3f90 ffff88016d0c4288
Mar 10 08:30:56 fileserver2 kernel: [157603.892971] Call Trace:
Mar 10 08:30:56 fileserver2 kernel: [157603.893017] [<ffffffff812495d0>] ? memcpy_toiovec+0x34/0x63
Mar 10 08:30:56 fileserver2 kernel: [157603.893074] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:30:56 fileserver2 kernel: [157603.893125] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:30:56 fileserver2 kernel: [157603.893176] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:30:56 fileserver2 kernel: [157603.893227] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:30:56 fileserver2 kernel: [157603.893278] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:30:56 fileserver2 kernel: [157603.893337] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893402] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893467] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893533] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893600] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893669] [<ffffffffa01fecfd>] ? xfs_iunlink+0x3b/0x127 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893735] [<ffffffffa021b0f7>] ? xfs_ichgtime+0x17/0x9c [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893806] [<ffffffffa020fa53>] ? xfs_trans_log_inode+0x1a/0x42 [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893872] [<ffffffffa0211f38>] ? xfs_remove+0x259/0x33e [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.893925] [<ffffffff810f74d5>] ? inode_permission+0x84/0xa1
Mar 10 08:30:56 fileserver2 kernel: [157603.893976] [<ffffffff810f5db8>] ? generic_permission+0xe/0x8a
Mar 10 08:30:56 fileserver2 kernel: [157603.894039] [<ffffffffa021ab31>] ? xfs_vn_unlink+0x37/0x6d [xfs]
Mar 10 08:30:56 fileserver2 kernel: [157603.894092] [<ffffffff810f7a54>] ? vfs_unlink+0x72/0xa6
Mar 10 08:30:56 fileserver2 kernel: [157603.894145] [<ffffffffa031f02b>] ? nfsd_unlink+0x105/0x16c [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.894203] [<ffffffffa03262d1>] ? nfsd3_proc_remove+0x74/0x81 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.894260] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.894318] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:30:56 fileserver2 kernel: [157603.894375] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.894428] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:30:56 fileserver2 kernel: [157603.894479] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.894528] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:30:56 fileserver2 kernel: [157603.894577] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:30:56 fileserver2 kernel: [157603.894625] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.833197] INFO: task nfsd:20040 blocked for more than 120 seconds.
Mar 10 08:32:56 fileserver2 kernel: [157723.833250] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:32:56 fileserver2 kernel: [157723.833333] nfsd D ffff88000544fb30 0 20040 2 0x00000000
Mar 10 08:32:56 fileserver2 kernel: [157723.833392] ffff88016fab8e20 0000000000000046 0000000000000000 ffff880167f1b110
Mar 10 08:32:56 fileserver2 kernel: [157723.833485] ffff880138b3ed00 0000000000000000 000000000000f9e0 ffff88016e269fd8
Mar 10 08:32:56 fileserver2 kernel: [157723.833578] 0000000000015780 0000000000015780 ffff88016d0c2a60 ffff88016d0c2d58
Mar 10 08:32:56 fileserver2 kernel: [157723.833670] Call Trace:
Mar 10 08:32:56 fileserver2 kernel: [157723.833726] [<ffffffffa020a895>] ? xfs_mod_incore_sb_batch+0x6f/0x101 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.833782] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:32:56 fileserver2 kernel: [157723.833852] [<ffffffffa021769f>] ? xfs_buf_rele+0x37/0xab [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.833904] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:32:56 fileserver2 kernel: [157723.833956] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:32:56 fileserver2 kernel: [157723.834015] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834079] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834145] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834211] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834278] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834343] [<ffffffffa01feed5>] ? xfs_iunlink_remove+0x4b/0x2a9 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834409] [<ffffffffa020fa53>] ? xfs_trans_log_inode+0x1a/0x42 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834475] [<ffffffffa01fe157>] ? xfs_itruncate_finish+0x284/0x295 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834543] [<ffffffffa01ff14b>] ? xfs_ifree+0x18/0xf2 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834606] [<ffffffffa02130a1>] ? xfs_inactive+0x30c/0x3f0 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.834660] [<ffffffff810fff23>] ? clear_inode+0x79/0xd0
Mar 10 08:32:56 fileserver2 kernel: [157723.834710] [<ffffffff8110066c>] ? generic_delete_inode+0xf4/0x168
Mar 10 08:32:56 fileserver2 kernel: [157723.834763] [<ffffffff810fe688>] ? d_delete+0x47/0xb2
Mar 10 08:32:56 fileserver2 kernel: [157723.834812] [<ffffffff810f7a81>] ? vfs_unlink+0x9f/0xa6
Mar 10 08:32:56 fileserver2 kernel: [157723.834866] [<ffffffffa031f02b>] ? nfsd_unlink+0x105/0x16c [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.834924] [<ffffffffa03262d1>] ? nfsd3_proc_remove+0x74/0x81 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.834981] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.835041] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:32:56 fileserver2 kernel: [157723.835097] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.835151] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.835201] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.835250] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.835300] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.835348] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.835396] INFO: task nfsd:20041 blocked for more than 120 seconds.
Mar 10 08:32:56 fileserver2 kernel: [157723.835448] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:32:56 fileserver2 kernel: [157723.835531] nfsd D ffff88000548fa20 0 20041 2 0x00000000
Mar 10 08:32:56 fileserver2 kernel: [157723.835590] ffff88016fb08000 0000000000000046 0000000000000000 ffffffff810e4493
Mar 10 08:32:56 fileserver2 kernel: [157723.835682] ffffea0004635b78 ffffffff810e6870 000000000000f9e0 ffff88016e0dffd8
Mar 10 08:32:56 fileserver2 kernel: [157723.835774] 0000000000015780 0000000000015780 ffff88016d0c0e20 ffff88016d0c1118
Mar 10 08:32:56 fileserver2 kernel: [157723.835867] Call Trace:
Mar 10 08:32:56 fileserver2 kernel: [157723.835908] [<ffffffff810e4493>] ? get_partial_node+0x15/0x85
Mar 10 08:32:56 fileserver2 kernel: [157723.835959] [<ffffffff810e6870>] ? __slab_alloc+0xd3/0x39b
Mar 10 08:32:56 fileserver2 kernel: [157723.836011] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:32:56 fileserver2 kernel: [157723.836062] [<ffffffff811971bb>] ? __bitmap_weight+0x3a/0x7e
Mar 10 08:32:56 fileserver2 kernel: [157723.836115] [<ffffffff8118ddf2>] ? cpumask_next_and+0x2a/0x3a
Mar 10 08:32:56 fileserver2 kernel: [157723.836167] [<ffffffff8103a6e1>] ? scale_rt_power+0x1f/0x65
Mar 10 08:32:56 fileserver2 kernel: [157723.836218] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:32:56 fileserver2 kernel: [157723.836269] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:32:56 fileserver2 kernel: [157723.836328] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836393] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836458] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836524] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836591] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836655] [<ffffffffa01f9349>] ? xfs_ialloc_read_agi+0x11/0x50 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836722] [<ffffffffa01f961c>] ? xfs_ialloc_ag_select+0x1e4/0x281 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836777] [<ffffffff8126296f>] ? sch_direct_xmit+0x7f/0x14c
Mar 10 08:32:56 fileserver2 kernel: [157723.836841] [<ffffffffa01fa611>] ? xfs_dialloc+0x33/0x866 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.836893] [<ffffffff810e5513>] ? __slab_free+0x7f/0x27a
Mar 10 08:32:56 fileserver2 kernel: [157723.836944] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:32:56 fileserver2 kernel: [157723.836995] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:32:56 fileserver2 kernel: [157723.837045] [<ffffffff8127e4f7>] ? tcp_recvmsg+0x98b/0xa9e
Mar 10 08:32:56 fileserver2 kernel: [157723.837109] [<ffffffffa01fe321>] ? xfs_ialloc+0x61/0x562 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.837172] [<ffffffffa02103b8>] ? xfs_dir_ialloc+0x84/0x2a8 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.837238] [<ffffffffa020e737>] ? xfs_trans_reserve+0xc8/0x19d [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.837303] [<ffffffffa021246a>] ? xfs_create+0x30f/0x539 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.837356] [<ffffffff810fe556>] ? __d_lookup+0xc3/0x107
Mar 10 08:32:56 fileserver2 kernel: [157723.837417] [<ffffffffa021a99b>] ? xfs_vn_mknod+0xc3/0x15a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.837470] [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Mar 10 08:32:56 fileserver2 kernel: [157723.837524] [<ffffffffa0320b27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.837583] [<ffffffffa03266a0>] ? nfsd3_proc_create+0x116/0x127 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.837643] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.837703] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:32:56 fileserver2 kernel: [157723.837761] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.837814] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.837868] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.837918] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.837967] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.838016] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.838064] INFO: task nfsd:20044 blocked for more than 120 seconds.
Mar 10 08:32:56 fileserver2 kernel: [157723.838116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:32:56 fileserver2 kernel: [157723.838199] nfsd D 0000000000000000 0 20044 2 0x00000000
Mar 10 08:32:56 fileserver2 kernel: [157723.838258] ffff88016fa7d4c0 0000000000000046 0000000000000000 0000000000000007
Mar 10 08:32:56 fileserver2 kernel: [157723.838350] ffff8801a3e54c78 ffff88026d85c000 000000000000f9e0 ffff88016e44dfd8
Mar 10 08:32:56 fileserver2 kernel: [157723.838443] 0000000000015780 0000000000015780 ffff88016d0c3170 ffff88016d0c3468
Mar 10 08:32:56 fileserver2 kernel: [157723.838534] Call Trace:
Mar 10 08:32:56 fileserver2 kernel: [157723.838587] [<ffffffffa020fc14>] ? xfs_trans_unlock_chunk+0x77/0xb6 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.838642] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:32:56 fileserver2 kernel: [157723.838694] [<ffffffff810b5908>] ? mempool_alloc+0x55/0x106
Mar 10 08:32:56 fileserver2 kernel: [157723.838745] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:32:56 fileserver2 kernel: [157723.838796] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:32:56 fileserver2 kernel: [157723.838855] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.838919] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.838984] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839050] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839117] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839182] [<ffffffffa01f9349>] ? xfs_ialloc_read_agi+0x11/0x50 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839248] [<ffffffffa01f961c>] ? xfs_ialloc_ag_select+0x1e4/0x281 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839304] [<ffffffff8126296f>] ? sch_direct_xmit+0x7f/0x14c
Mar 10 08:32:56 fileserver2 kernel: [157723.839367] [<ffffffffa01fa611>] ? xfs_dialloc+0x33/0x866 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839420] [<ffffffff810e5513>] ? __slab_free+0x7f/0x27a
Mar 10 08:32:56 fileserver2 kernel: [157723.839471] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:32:56 fileserver2 kernel: [157723.839521] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:32:56 fileserver2 kernel: [157723.839572] [<ffffffff8127e4f7>] ? tcp_recvmsg+0x98b/0xa9e
Mar 10 08:32:56 fileserver2 kernel: [157723.839635] [<ffffffffa01fe321>] ? xfs_ialloc+0x61/0x562 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839699] [<ffffffffa02103b8>] ? xfs_dir_ialloc+0x84/0x2a8 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839764] [<ffffffffa020e737>] ? xfs_trans_reserve+0xc8/0x19d [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839829] [<ffffffffa021246a>] ? xfs_create+0x30f/0x539 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839883] [<ffffffff810fe556>] ? __d_lookup+0xc3/0x107
Mar 10 08:32:56 fileserver2 kernel: [157723.839944] [<ffffffffa021a99b>] ? xfs_vn_mknod+0xc3/0x15a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.839997] [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Mar 10 08:32:56 fileserver2 kernel: [157723.840051] [<ffffffffa0320b27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.840109] [<ffffffffa03266a0>] ? nfsd3_proc_create+0x116/0x127 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.840167] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.840225] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:32:56 fileserver2 kernel: [157723.840282] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.840335] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.840385] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.840434] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.840483] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.840532] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.840580] INFO: task nfsd:20046 blocked for more than 120 seconds.
Mar 10 08:32:56 fileserver2 kernel: [157723.840632] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:32:56 fileserver2 kernel: [157723.840715] nfsd D ffff88000548fa20 0 20046 2 0x00000000
Mar 10 08:32:56 fileserver2 kernel: [157723.840774] ffff88016fb08000 0000000000000046 0000000000000000 ffff88026dc94d00
Mar 10 08:32:56 fileserver2 kernel: [157723.840866] ffff88026cace69c ffff88026df28000 000000000000f9e0 ffff88015db8dfd8
Mar 10 08:32:56 fileserver2 kernel: [157723.840958] 0000000000015780 0000000000015780 ffff88016d0c3f90 ffff88016d0c4288
Mar 10 08:32:56 fileserver2 kernel: [157723.841050] Call Trace:
Mar 10 08:32:56 fileserver2 kernel: [157723.841091] [<ffffffff812495d0>] ? memcpy_toiovec+0x34/0x63
Mar 10 08:32:56 fileserver2 kernel: [157723.849168] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:32:56 fileserver2 kernel: [157723.849221] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:32:56 fileserver2 kernel: [157723.849273] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:32:56 fileserver2 kernel: [157723.849326] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:32:56 fileserver2 kernel: [157723.849378] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:32:56 fileserver2 kernel: [157723.849438] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849502] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849563] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849624] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849686] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849746] [<ffffffffa01fecfd>] ? xfs_iunlink+0x3b/0x127 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849804] [<ffffffffa021b0f7>] ? xfs_ichgtime+0x17/0x9c [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849866] [<ffffffffa020fa53>] ? xfs_trans_log_inode+0x1a/0x42 [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849928] [<ffffffffa0211f38>] ? xfs_remove+0x259/0x33e [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.849980] [<ffffffff810f74d5>] ? inode_permission+0x84/0xa1
Mar 10 08:32:56 fileserver2 kernel: [157723.850030] [<ffffffff810f5db8>] ? generic_permission+0xe/0x8a
Mar 10 08:32:56 fileserver2 kernel: [157723.850088] [<ffffffffa021ab31>] ? xfs_vn_unlink+0x37/0x6d [xfs]
Mar 10 08:32:56 fileserver2 kernel: [157723.850139] [<ffffffff810f7a54>] ? vfs_unlink+0x72/0xa6
Mar 10 08:32:56 fileserver2 kernel: [157723.850190] [<ffffffffa031f02b>] ? nfsd_unlink+0x105/0x16c [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.850245] [<ffffffffa03262d1>] ? nfsd3_proc_remove+0x74/0x81 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.850300] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.850355] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:32:56 fileserver2 kernel: [157723.850409] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.850460] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:32:56 fileserver2 kernel: [157723.850509] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.850557] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:32:56 fileserver2 kernel: [157723.850605] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:32:56 fileserver2 kernel: [157723.850652] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:34:56 fileserver2 kernel: [157843.789394] INFO: task nfsd:20040 blocked for more than 120 seconds.
Mar 10 08:34:56 fileserver2 kernel: [157843.789446] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:34:56 fileserver2 kernel: [157843.789528] nfsd D ffff88000544fb30 0 20040 2 0x00000000
Mar 10 08:34:56 fileserver2 kernel: [157843.789586] ffff88016fab8e20 0000000000000046 0000000000000000 ffff880167f1b110
Mar 10 08:34:56 fileserver2 kernel: [157843.789677] ffff880138b3ed00 0000000000000000 000000000000f9e0 ffff88016e269fd8
Mar 10 08:34:56 fileserver2 kernel: [157843.789768] 0000000000015780 0000000000015780 ffff88016d0c2a60 ffff88016d0c2d58
Mar 10 08:34:56 fileserver2 kernel: [157843.789859] Call Trace:
Mar 10 08:34:56 fileserver2 kernel: [157843.789910] [<ffffffffa020a895>] ? xfs_mod_incore_sb_batch+0x6f/0x101 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.789965] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:34:56 fileserver2 kernel: [157843.790023] [<ffffffffa021769f>] ? xfs_buf_rele+0x37/0xab [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790082] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:34:56 fileserver2 kernel: [157843.790133] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:34:56 fileserver2 kernel: [157843.790192] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790253] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790313] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790374] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790436] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790496] [<ffffffffa01feed5>] ? xfs_iunlink_remove+0x4b/0x2a9 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790557] [<ffffffffa020fa53>] ? xfs_trans_log_inode+0x1a/0x42 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790619] [<ffffffffa01fe157>] ? xfs_itruncate_finish+0x284/0x295 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790681] [<ffffffffa01ff14b>] ? xfs_ifree+0x18/0xf2 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790739] [<ffffffffa02130a1>] ? xfs_inactive+0x30c/0x3f0 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.790791] [<ffffffff810fff23>] ? clear_inode+0x79/0xd0
Mar 10 08:34:56 fileserver2 kernel: [157843.790840] [<ffffffff8110066c>] ? generic_delete_inode+0xf4/0x168
Mar 10 08:34:56 fileserver2 kernel: [157843.790892] [<ffffffff810fe688>] ? d_delete+0x47/0xb2
Mar 10 08:34:56 fileserver2 kernel: [157843.790940] [<ffffffff810f7a81>] ? vfs_unlink+0x9f/0xa6
Mar 10 08:34:56 fileserver2 kernel: [157843.790991] [<ffffffffa031f02b>] ? nfsd_unlink+0x105/0x16c [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.791046] [<ffffffffa03262d1>] ? nfsd3_proc_remove+0x74/0x81 [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.791101] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.791157] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:34:56 fileserver2 kernel: [157843.791211] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.791262] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.791311] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:34:56 fileserver2 kernel: [157843.791359] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:34:56 fileserver2 kernel: [157843.791406] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:34:56 fileserver2 kernel: [157843.791454] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Mar 10 08:34:56 fileserver2 kernel: [157843.791501] INFO: task nfsd:20041 blocked for more than 120 seconds.
Mar 10 08:34:56 fileserver2 kernel: [157843.791552] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 08:34:56 fileserver2 kernel: [157843.791634] nfsd D ffff88000548fa20 0 20041 2 0x00000000
Mar 10 08:34:56 fileserver2 kernel: [157843.791692] ffff88016fb08000 0000000000000046 0000000000000000 ffffffff810e4493
Mar 10 08:34:56 fileserver2 kernel: [157843.791783] ffffea0004635b78 ffffffff810e6870 000000000000f9e0 ffff88016e0dffd8
Mar 10 08:34:56 fileserver2 kernel: [157843.791874] 0000000000015780 0000000000015780 ffff88016d0c0e20 ffff88016d0c1118
Mar 10 08:34:56 fileserver2 kernel: [157843.791965] Call Trace:
Mar 10 08:34:56 fileserver2 kernel: [157843.792004] [<ffffffff810e4493>] ? get_partial_node+0x15/0x85
Mar 10 08:34:56 fileserver2 kernel: [157843.792055] [<ffffffff810e6870>] ? __slab_alloc+0xd3/0x39b
Mar 10 08:34:56 fileserver2 kernel: [157843.792104] [<ffffffff812fabfc>] ? schedule_timeout+0x2e/0xdd
Mar 10 08:34:56 fileserver2 kernel: [157843.792155] [<ffffffff811971bb>] ? __bitmap_weight+0x3a/0x7e
Mar 10 08:34:56 fileserver2 kernel: [157843.792205] [<ffffffff8118ddf2>] ? cpumask_next_and+0x2a/0x3a
Mar 10 08:34:56 fileserver2 kernel: [157843.792256] [<ffffffff8103a6e1>] ? scale_rt_power+0x1f/0x65
Mar 10 08:34:56 fileserver2 kernel: [157843.792305] [<ffffffff812fb4e0>] ? __down_common+0x8d/0xde
Mar 10 08:34:56 fileserver2 kernel: [157843.792355] [<ffffffff81068547>] ? down+0x27/0x38
Mar 10 08:34:56 fileserver2 kernel: [157843.792409] [<ffffffffa0217df8>] ? _xfs_buf_find+0x162/0x1e0 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792469] [<ffffffffa0217ed0>] ? xfs_buf_get_flags+0x5a/0x13b [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792528] [<ffffffffa0217fc3>] ? xfs_buf_read_flags+0x12/0x7a [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792589] [<ffffffffa020f6d5>] ? xfs_trans_read_buf+0x189/0x27e [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792651] [<ffffffffa01f92c2>] ? xfs_read_agi+0x64/0xda [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792711] [<ffffffffa01f9349>] ? xfs_ialloc_read_agi+0x11/0x50 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792772] [<ffffffffa01f961c>] ? xfs_ialloc_ag_select+0x1e4/0x281 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792827] [<ffffffff8126296f>] ? sch_direct_xmit+0x7f/0x14c
Mar 10 08:34:56 fileserver2 kernel: [157843.792885] [<ffffffffa01fa611>] ? xfs_dialloc+0x33/0x866 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.792936] [<ffffffff810e5513>] ? __slab_free+0x7f/0x27a
Mar 10 08:34:56 fileserver2 kernel: [157843.792986] [<ffffffff812fbc39>] ? _spin_lock_bh+0x9/0x25
Mar 10 08:34:56 fileserver2 kernel: [157843.793035] [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Mar 10 08:34:56 fileserver2 kernel: [157843.793084] [<ffffffff8127e4f7>] ? tcp_recvmsg+0x98b/0xa9e
Mar 10 08:34:56 fileserver2 kernel: [157843.793142] [<ffffffffa01fe321>] ? xfs_ialloc+0x61/0x562 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.793201] [<ffffffffa02103b8>] ? xfs_dir_ialloc+0x84/0x2a8 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.793261] [<ffffffffa020e737>] ? xfs_trans_reserve+0xc8/0x19d [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.793322] [<ffffffffa021246a>] ? xfs_create+0x30f/0x539 [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.793373] [<ffffffff810fe556>] ? __d_lookup+0xc3/0x107
Mar 10 08:34:56 fileserver2 kernel: [157843.793429] [<ffffffffa021a99b>] ? xfs_vn_mknod+0xc3/0x15a [xfs]
Mar 10 08:34:56 fileserver2 kernel: [157843.793480] [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Mar 10 08:34:56 fileserver2 kernel: [157843.793532] [<ffffffffa0320b27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.793587] [<ffffffffa03266a0>] ? nfsd3_proc_create+0x116/0x127 [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.793643] [<ffffffffa031b329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.793698] [<ffffffffa02584ff>] ? svc_process+0x403/0x627 [sunrpc]
Mar 10 08:34:56 fileserver2 kernel: [157843.793752] [<ffffffffa031b772>] ? nfsd+0x0/0x12e [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.793803] [<ffffffffa031b857>] ? nfsd+0xe5/0x12e [nfsd]
Mar 10 08:34:56 fileserver2 kernel: [157843.793852] [<ffffffff81064a5d>] ? kthread+0x79/0x81
Mar 10 08:34:56 fileserver2 kernel: [157843.793900] [<ffffffff81011baa>] ? child_rip+0xa/0x20
Mar 10 08:34:56 fileserver2 kernel: [157843.793948] [<ffffffff810649e4>] ? kthread+0x0/0x81
Mar 10 08:34:56 fileserver2 kernel: [157843.793995] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Reply to: