Bug#709989: linux-image-3.2.0-0.bpo.4-amd64: General protection fault after I/O on ext3 in KVM
Bastian Blank wrote:
> > ** Tainted: PDO (4225)
> > * Proprietary module has been loaded.
> > * Kernel has oopsed before.
> > * Out-of-tree module has been loaded.
>
> You have unsupported modules loaded. Please remove them and try again.
OK, somehow I could reproduce the problem today simply with iozone and
without loaded OpenAFS module. Machine crashed during creation of
initial work file:
$ iozone -Mce -s256g -r256k -i0 -i1 -i2
May 28 11:58:30 zapat102 kernel: [ 0.000000] Linux version 3.2.0-0.bpo.4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Debian 3.2.41-2+deb7u2~bpo60+1
...
May 28 12:06:04 zapat102 kernel: [ 480.642346] general protection fault: 0000 [#1] SMP
May 28 12:06:04 zapat102 kernel: [ 480.645190] CPU 0
May 28 12:06:04 zapat102 kernel: [ 480.645358] Modules linked in: nls_utf8 isofs des_generic cbc rpcsec_gss_krb5 nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment xt_multiport iptable_filter ip_tables x_tables ext3 jbd ib_ipoib mlx4_ib rdma_ucm rdma_cm ib_cm iw_cm ib_addr ib_sa ib_uverbs mlx4_en mlx4_core ib_umad ib_mad ib_core loop crc32c_intel ghash_clmulni_intel snd_pcm snd_timer aesni_intel snd cryptd tpm_tis i2c_piix4 soundcore aes_x86_64 snd_page_alloc tpm psmouse i6300esb tpm_bios virtio_balloon aes_generic i2c_core evdev serio_raw pcspkr processor thermal_sys button ext4 mbcache jbd2 crc16 dm_mod microcode virtio_net ata_generic virtio_blk uhci_hcd ata_piix ehci_hcd libata usbcore virtio_pci virtio_ring virtio scsi_mod usb_common floppy [last unloaded: scsi_wait_scan]
May 28 12:06:04 zapat102 kernel: [ 480.645358]
May 28 12:06:04 zapat102 kernel: [ 480.645358] Pid: 903, comm: flush-254:64 Not tainted 3.2.0-0.bpo.4-amd64 #1 Debian 3.2.41-2+deb7u2~bpo60+1 Bochs Bochs
May 28 12:06:04 zapat102 kernel: [ 480.645358] RIP: 0010:[<ffffffffa02719be>] [<ffffffffa02719be>] walk_page_buffers+0x22/0x97 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.645358] RSP: 0018:ffff881e28779a00 EFLAGS: 00010246
May 28 12:06:04 zapat102 kernel: [ 480.645358] RAX: 0000000000000000 RBX: ffff8800bdf7f130 RCX: fff4fbb100000325
May 28 12:06:04 zapat102 kernel: [ 480.645358] RDX: 0000000000000000 RSI: fff4fbb100000325 RDI: 0000000000000000
May 28 12:06:04 zapat102 kernel: [ 480.645358] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffffa0271a41
May 28 12:06:04 zapat102 kernel: [ 480.645358] R10: 000000000000000f R11: 0000000000000064 R12: 0000000000001000
May 28 12:06:04 zapat102 kernel: [ 480.645358] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
May 28 12:06:04 zapat102 kernel: [ 480.645358] FS: 0000000000000000(0000) GS:ffff880f42c00000(0000) knlGS:0000000000000000
May 28 12:06:04 zapat102 kernel: [ 480.645358] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 28 12:06:04 zapat102 kernel: [ 480.645358] CR2: 00007fb78cab10e0 CR3: 0000000f055ff000 CR4: 00000000000406f0
May 28 12:06:04 zapat102 kernel: [ 480.645358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 28 12:06:04 zapat102 kernel: [ 480.645358] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 28 12:06:04 zapat102 kernel: [ 480.645358] Process flush-254:64 (pid: 903, threadinfo ffff881e28778000, task ffff881e27eae2c0)
May 28 12:06:04 zapat102 kernel: [ 480.645358] Stack:
May 28 12:06:04 zapat102 kernel: [ 480.645358] ffff881e28779b78 0000000000000000 000000000000000e fff4fbb100000325
May 28 12:06:04 zapat102 kernel: [ 480.645358] 0000000000000000 ffffea0059909110 ffff880eeb273760 ffff881e28779c90
May 28 12:06:04 zapat102 kernel: [ 480.645358] 0000000000000000 ffff880f056ed1b0 ffffea0059909110 ffffffffa0273286
May 28 12:06:04 zapat102 kernel: [ 480.645358] Call Trace:
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffffa0273286>] ? ext3_writeback_writepage+0xd3/0x157 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff810c478e>] ? __writepage+0xa/0x21
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff810c5881>] ? write_cache_pages+0x226/0x31e
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff810c4784>] ? set_page_dirty+0x61/0x61
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff810c59b7>] ? generic_writepages+0x3e/0x55
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff811254fc>] ? writeback_single_inode+0x178/0x35e
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff811259ae>] ? writeback_sb_inodes+0x169/0x1ff
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81125ab1>] ? __writeback_inodes_wb+0x6d/0xab
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81125cd7>] ? wb_writeback+0x128/0x222
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff8136761c>] ? __schedule+0x5a0/0x5cd
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81125f4a>] ? wb_do_writeback+0x179/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81055ccb>] ? del_timer_sync+0x34/0x3e
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81126072>] ? bdi_writeback_thread+0xc3/0x1fe
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81125faf>] ? wb_do_writeback+0x1de/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81125faf>] ? wb_do_writeback+0x1de/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff8106371d>] ? kthread+0x7a/0x82
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81370274>] ? kernel_thread_helper+0x4/0x10
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff810636a3>] ? kthread_worker_fn+0x147/0x147
May 28 12:06:04 zapat102 kernel: [ 480.645358] [<ffffffff81370270>] ? gs_change+0x13/0x13
May 28 12:06:04 zapat102 kernel: [ 480.645358] Code: 5e 41 5f c3 48 89 dd eb 93 41 57 31 c0 41 56 41 55 41 54 41 89 cc 55 48 89 fd 53 48 89 f3 48 83 ec 28 4c 8b 6e 20 45 89 ee eb 5b <48> 8b 4e 08 44 39 e0 46 8d 3c 30 48 89 4c 24 18 73 05 41 39 d5
May 28 12:06:04 zapat102 kernel: [ 480.645358] RIP [<ffffffffa02719be>] walk_page_buffers+0x22/0x97 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.645358] RSP <ffff881e28779a00>
May 28 12:06:04 zapat102 kernel: [ 480.838258] ---[ end trace 929b8282c06bf79a ]---
May 28 12:06:04 zapat102 kernel: [ 480.840967] ------------[ cut here ]------------
May 28 12:06:04 zapat102 kernel: [ 480.843449] WARNING: at /build/buildd-linux_3.2.41-2+deb7u2~bpo60+1-amd64-mnypfK/linux-3.2.41/kernel/exit.c:879 do_exit+0x5f/0x799()
May 28 12:06:04 zapat102 kernel: [ 480.850313] Hardware name: Bochs
May 28 12:06:04 zapat102 kernel: [ 480.852242] Modules linked in: nls_utf8 isofs des_generic cbc rpcsec_gss_krb5 nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment xt_multiport iptable_filter ip_tables x_tables ext3 jbd ib_ipoib mlx4_ib rdma_ucm rdma_cm ib_cm iw_cm ib_addr ib_sa ib_uverbs mlx4_en mlx4_core ib_umad ib_mad ib_core loop crc32c_intel ghash_clmulni_intel snd_pcm snd_timer aesni_intel snd cryptd tpm_tis i2c_piix4 soundcore aes_x86_64 snd_page_alloc tpm psmouse i6300esb tpm_bios virtio_balloon aes_generic i2c_core evdev serio_raw pcspkr processor thermal_sys button ext4 mbcache jbd2 crc16 dm_mod microcode virtio_net ata_generic virtio_blk uhci_hcd ata_piix ehci_hcd libata usbcore virtio_pci virtio_ring virtio scsi_mod usb_common floppy [last unloaded: scsi_wait_scan]
May 28 12:06:04 zapat102 kernel: [ 480.893898] Pid: 903, comm: flush-254:64 Tainted: G D 3.2.0-0.bpo.4-amd64 #1 Debian 3.2.41-2+deb7u2~bpo60+1
May 28 12:06:04 zapat102 kernel: [ 480.899252] Call Trace:
May 28 12:06:04 zapat102 kernel: [ 480.900612] [<ffffffff8104996b>] ? warn_slowpath_common+0x78/0x8c
May 28 12:06:04 zapat102 kernel: [ 480.903780] [<ffffffff8104cd4d>] ? do_exit+0x5f/0x799
May 28 12:06:04 zapat102 kernel: [ 480.906949] [<ffffffff81366e89>] ? printk+0x40/0x47
May 28 12:06:04 zapat102 kernel: [ 480.909710] [<ffffffff81049ea6>] ? kmsg_dump+0x53/0xef
May 28 12:06:04 zapat102 kernel: [ 480.912582] [<ffffffff813699bd>] ? oops_end+0x65/0xb6
May 28 12:06:04 zapat102 kernel: [ 480.915215] [<ffffffff81369a09>] ? oops_end+0xb1/0xb6
May 28 12:06:04 zapat102 kernel: [ 480.918224] [<ffffffff813690c5>] ? general_protection+0x25/0x30
May 28 12:06:04 zapat102 kernel: [ 480.921470] [<ffffffffa0271a41>] ? bput_one+0x7/0x7 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.924537] [<ffffffffa02719be>] ? walk_page_buffers+0x22/0x97 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.928408] [<ffffffffa0273286>] ? ext3_writeback_writepage+0xd3/0x157 [ext3]
May 28 12:06:04 zapat102 kernel: [ 480.932557] [<ffffffff810c478e>] ? __writepage+0xa/0x21
May 28 12:06:04 zapat102 kernel: [ 480.935502] [<ffffffff810c5881>] ? write_cache_pages+0x226/0x31e
May 28 12:06:04 zapat102 kernel: [ 480.938622] [<ffffffff810c4784>] ? set_page_dirty+0x61/0x61
May 28 12:06:04 zapat102 kernel: [ 480.941815] [<ffffffff810c59b7>] ? generic_writepages+0x3e/0x55
May 28 12:06:04 zapat102 kernel: [ 480.945223] [<ffffffff811254fc>] ? writeback_single_inode+0x178/0x35e
May 28 12:06:04 zapat102 kernel: [ 480.948802] [<ffffffff811259ae>] ? writeback_sb_inodes+0x169/0x1ff
May 28 12:06:04 zapat102 kernel: [ 480.952123] [<ffffffff81125ab1>] ? __writeback_inodes_wb+0x6d/0xab
May 28 12:06:04 zapat102 kernel: [ 480.955687] [<ffffffff81125cd7>] ? wb_writeback+0x128/0x222
May 28 12:06:04 zapat102 kernel: [ 480.958824] [<ffffffff8136761c>] ? __schedule+0x5a0/0x5cd
May 28 12:06:04 zapat102 kernel: [ 480.961884] [<ffffffff81125f4a>] ? wb_do_writeback+0x179/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.965552] [<ffffffff81055ccb>] ? del_timer_sync+0x34/0x3e
May 28 12:06:04 zapat102 kernel: [ 480.968765] [<ffffffff81126072>] ? bdi_writeback_thread+0xc3/0x1fe
May 28 12:06:04 zapat102 kernel: [ 480.972315] [<ffffffff81125faf>] ? wb_do_writeback+0x1de/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.975563] [<ffffffff81125faf>] ? wb_do_writeback+0x1de/0x1de
May 28 12:06:04 zapat102 kernel: [ 480.979044] [<ffffffff8106371d>] ? kthread+0x7a/0x82
May 28 12:06:04 zapat102 kernel: [ 480.981865] [<ffffffff81370274>] ? kernel_thread_helper+0x4/0x10
May 28 12:06:04 zapat102 kernel: [ 480.985114] [<ffffffff810636a3>] ? kthread_worker_fn+0x147/0x147
May 28 12:06:04 zapat102 kernel: [ 480.988509] [<ffffffff81370270>] ? gs_change+0x13/0x13
May 28 12:06:04 zapat102 kernel: [ 480.991259] ---[ end trace 929b8282c06bf79b ]---
We'll do further investigation in this problem today and let you know.
--
Vlastimil Holer phone: +420-549 49 5349
CERIT Scientific Cloud e-mail: holer@ics.muni.cz
Institute of Computer Science MU twitter: @vholer
Reply to: