Bug#861821: Rare general protection faults in Jessie when mylvmbackup is trying to mount snapshotted LV with XFS filesystem
Package: linux-image-3.16.0-4-amd64
Version: 3.16.39-1+deb8u2
Hi,
We've noticed Jessie (Jessie as KVM guest) kernel general protection fault when mylvmbackup was trying to mount snapshotted LV with XFS filesystem; syslog caught:
2017-05-03 00:03:02 mylvmbackup[17592]: Mounting snapshot...
2017-05-03 00:03:02 mylvmbackup[17592]: Running: /bin/mount -o rw,nouuid /dev/vg1/db_snapshot /var/cache/mylvmbackup/mnt/bkp-mysql
2017-05-03 00:03:02 kernel: [283314.227591] XFS (dm-15): Mounting V4 Filesystem
2017-05-03 00:03:03 kernel: [283314.411970] general protection fault: 0000 [#1] SMP
2017-05-03 00:03:03 kernel: [283314.412921] Modules linked in: dm_snapshot dm_bufio binfmt_misc xt_tcpudp xt_LOG xt_limit xt_recent xt_comment iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_addrtype bridge stp llc xt_mark iptable_mangle xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables ppdev joydev evdev serio_raw pcspkr ttm drm_kms_helper i2c_piix4 drm virtio_balloon pvpanic i2c_core parport_pc parport processor thermal_sys button autofs4 xfs libcrc32c crc32c_generic btrfs xor raid6_pq hid_generic usbhid hid dm_mod ata_generic virtio_net virtio_blk ata_piix uhci_hcd ehci_hcd floppy libata scsi_mod psmouse virtio_pci virtio_ring virtio usbcore usb_common
2017-05-03 00:03:03 kernel: [283314.415398] CPU: 1 PID: 17458 Comm: kworker/1:1 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.39-1+deb8u2
2017-05-03 00:03:03 kernel: [283314.415398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org 04/01/2014
2017-05-03 00:03:03 kernel: [283314.415398] Workqueue: kcopyd do_work [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] task: ffff8800ba8d2150 ti: ffff880018b20000 task.ti: ffff880018b20000
2017-05-03 00:03:03 kernel: [283314.415398] RIP: 0010:[<ffffffff812882ee>] [<ffffffff812882ee>] __blk_bios_map_sg+0x1de/0x400
2017-05-03 00:03:03 kernel: [283314.415398] RSP: 0018:ffff880018b238a0 EFLAGS: 00010286
2017-05-03 00:03:03 kernel: [283314.415398] RAX: 80fffffffffffffc RBX: 0000000000000600 RCX: 0000000000000000
2017-05-03 00:03:03 kernel: [283314.415398] RDX: 80ffffffffffffff RSI: ffffea00028bcef0 RDI: 0000000000000000
2017-05-03 00:03:03 kernel: [283314.415398] RBP: ffff880018b23920 R08: ffff8800822cc320 R09: 0000000000000080
2017-05-03 00:03:03 kernel: [283314.415398] R10: 0000000000000600 R11: 00000000ba3d1000 R12: 0000000000000000
2017-05-03 00:03:03 kernel: [283314.415398] R13: 0000000000000600 R14: 0000000000000000 R15: ffff880018b23930
2017-05-03 00:03:03 kernel: [283314.415398] FS: 0000000000000000(0000) GS:ffff8800bfb00000(0000) knlGS:0000000000000000
2017-05-03 00:03:03 kernel: [283314.415398] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2017-05-03 00:03:03 kernel: [283314.415398] CR2: 000000000221f048 CR3: 000000005eb01000 CR4: 00000000000006e0
2017-05-03 00:03:03 kernel: [283314.415398] Stack:
2017-05-03 00:03:03 kernel: [283314.415398] 0000000000000046 ffff8800373137a0 ffff880036b00008 01ffffff81095bb5
2017-05-03 00:03:03 kernel: [283314.415398] ffff8800822cc320 ffffffff00000080 0000000000000000 ffffea00028bcef0
2017-05-03 00:03:03 kernel: [283314.415398] 0000000000000600 ffffea00028bd580 0000060000000a00 ffff880037313600
2017-05-03 00:03:03 kernel: [283314.415398] Call Trace:
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8128853f>] ? blk_rq_map_sg+0x2f/0x170
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa00444db>] ? virtio_queue_rq+0x9b/0x240 [virtio_blk]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8128c172>] ? __blk_mq_run_hw_queue+0x1c2/0x320
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8128d52d>] ? blk_mq_insert_requests+0xcd/0x150
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8128e059>] ? blk_mq_flush_plug_list+0x129/0x140
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81284663>] ? blk_flush_plug_list+0xe3/0x230
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81517c1d>] ? io_schedule_timeout+0x7d/0xe0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff815188f8>] ? wait_for_completion_io+0xa8/0x110
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81098320>] ? wake_up_state+0x10/0x10
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa0150b8b>] ? sync_io+0x9b/0xf0 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8127ff94>] ? generic_make_request_checks+0x264/0x3d0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa0150dce>] ? dm_io+0x1ee/0x220 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa01506b0>] ? dm_io_client_destroy+0x30/0x30 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa0150540>] ? bio_next_page+0x20/0x20 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa04d3630>] ? chunk_io+0xf0/0x100 [dm_snapshot]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff812801b0>] ? generic_make_request+0xb0/0x100
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa04d0070>] ? dm_lookup_exception+0x50/0x50 [dm_snapshot]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa04d3a4d>] ? persistent_commit_exception+0xcd/0x150 [dm_snapshot]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa04d0115>] ? copy_callback+0xa5/0x130 [dm_snapshot]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa015150d>] ? run_complete_job+0x5d/0xb0 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa01514b0>] ? kcopyd_put_pages+0x40/0x40 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa0151042>] ? process_jobs+0x92/0x120 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffffa0151416>] ? do_work+0x36/0x70 [dm_mod]
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81082bb3>] ? process_one_work+0x143/0x430
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81083333>] ? worker_thread+0x113/0x4f0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81517221>] ? __schedule+0x2b1/0x6f0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81083220>] ? rescuer_thread+0x2d0/0x2d0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff810894fd>] ? kthread+0xbd/0xe0
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81089440>] ? kthread_create_on_node+0x180/0x180
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff8151ad98>] ? ret_from_fork+0x58/0x90
2017-05-03 00:03:03 kernel: [283314.415398] [<ffffffff81089440>] ? kthread_create_on_node+0x180/0x180
2017-05-03 00:03:03 kernel: [283314.415398] Code: 3f 44 89 4c 24 28 48 89 4c 24 30 e8 bd 5a 03 00 8b 7c 24 44 48 8b 74 24 38 4c 8b 44 24 20 44 8b 4c 24 28 48 8b 4c 24 30 49 89 07 <48> 8b 10 83 e2 03 40 f6 c6 03 0f 85 b2 01 00 00 48 09 f2 89 78
2017-05-03 00:03:03 kernel: [283314.415398] RIP [<ffffffff812882ee>] __blk_bios_map_sg+0x1de/0x400
2017-05-03 00:03:03 kernel: [283314.415398] RSP <ffff880018b238a0>
2017-05-03 00:03:03 kernel: [283314.487465] ---[ end trace c61d1f486cfc3a7d ]---
2017-05-03 00:03:03 kernel: [283314.489156] BUG: unable to handle kernel paging request at ffffffffffffffd8
2017-05-03 00:03:03 kernel: [283314.491210] IP: [<ffffffff81089acc>] kthread_data+0xc/0x20
2017-05-03 00:03:03 kernel: [283314.492285] PGD 1816067 PUD 1818067 PMD 0
2017-05-03 00:03:03 kernel: [283314.493012] Oops: 0000 [#2] SMP
This hung guest system (cold reset required).
Problem is rare - two occurences in last 2 months in one system only (mylvmbackup runs once every 24h there; no problems most of time).
No such problems noticed with mylvmbackup on Wheezy.
Please verify and consider fixing in Jessie.
Regards,
Pawel
IB Development Team
https://dev.ib.pl/
Reply to: