Bug#661558: kernel messages
First things first: I can confirm that linux-image-3.2.0-1-amd64 3.2.6-1
crashes too.
I've been able to capture the kernel messages using netconsole. The
messages start, accidentally, just in time to see the kernel assemble the
two "real" arrays I have here:
[ 6.119851] md: raid1 personality registered for level 1
[ 6.127270] mdadm: sending ioctl 800c0910 to a partition!
[ 6.127355] mdadm: sending ioctl 800c0910 to a partition!
[ 6.127420] mdadm: sending ioctl 1261 to a partition!
[ 6.127487] mdadm: sending ioctl 1261 to a partition!
[ 6.128099] mdadm: sending ioctl 800c0910 to a partition!
[ 6.128170] mdadm: sending ioctl 800c0910 to a partition!
[ 6.128232] mdadm: sending ioctl 1261 to a partition!
[ 6.128299] mdadm: sending ioctl 1261 to a partition!
[ 6.144287] mdadm: sending ioctl 800c0910 to a partition!
[ 6.144358] mdadm: sending ioctl 800c0910 to a partition!
[ 6.218941] md: md1 stopped.
[ 6.225242] md: bind<sdc1>
[ 6.225652] md: bind<sdb1>
[ 6.227112] bio: create slab <bio-1> at 1
[ 6.227274] md/raid1:md1: active with 2 out of 2 mirrors
[ 6.227659] created bitmap (1 pages) for device md1
[ 6.227996] md1: bitmap initialized from disk: read 1/1 pages, set 0 of
251 bits
[ 6.249696] md1: detected capacity change from 0 to 263127040
[ 6.250354] md1:
[ 6.292819] md: md2 stopped.
[ 6.294717] md: bind<sdc2>
[ 6.295058] md: bind<sdb2>
[ 6.296711] md/raid1:md2: active with 2 out of 2 mirrors
[ 6.304329] created bitmap (3 pages) for device md2
[ 6.304714] md2: bitmap initialized from disk: read 1/1 pages, set 0 of
5021 bits
[ 6.330160] md2: detected capacity change from 0 to 336907403264
[ 6.332079] md2: unknown partition table
[ 6.925562] EXT4-fs (dm-0): mounted filesystem with ordered data mode.
Opts: (null)
Here I create a new array on spare partitions:
# mdadm --create /dev/md3 --metadata=0.90 --assume-clean -l1 -n2 /dev/sdb3
/dev/sdc1
[ 37.530284] scsi_verify_blk_ioctl: 74 callbacks suppressed
[ 37.530354] mdadm: sending ioctl 1261 to a partition!
[ 37.530415] mdadm: sending ioctl 1261 to a partition!
[ 37.566744] mdadm: sending ioctl 1261 to a partition!
[ 37.566814] mdadm: sending ioctl 1261 to a partition!
[ 37.568463] mdadm: sending ioctl 1261 to a partition!
[ 37.568534] mdadm: sending ioctl 1261 to a partition!
[ 37.570387] mdadm: sending ioctl 1261 to a partition!
[ 37.570457] mdadm: sending ioctl 1261 to a partition!
[ 37.574025] mdadm: sending ioctl 1261 to a partition!
[ 37.574095] mdadm: sending ioctl 1261 to a partition!
[ 38.038824] md: bind<sdd1>
[ 38.039093] md: bind<sdc3>
[ 38.308214] md/raid1:md3: active with 2 out of 2 mirrors
[ 38.308311] md3: detected capacity change from 0 to 100029104128
[ 38.309864] md3: unknown partition table
Here I add the internal bitmap:
# mdadm --grow /dev/md3 --bitmap=internal
[ 38.360808] md3: bitmap file is out of date (0 < 1) -- forcing full recovery
[ 38.360883] created bitmap (1 pages) for device md3
(at this point the array was still "auto-read-only")
Here I write something to the array:
# dd if=/dev/zero of=/dev/md3 bs=1M count=10
[ 90.957478] BUG: unable to handle kernel NULL pointer dereference at
0000000000000010
[ 90.957645] IP: [<ffffffffa01dd2c1>] bitmap_endwrite+0x131/0x18f [md_mod]
[ 90.957752] PGD 2226c3067 PUD 223ed0067 PMD 0
[ 90.957927] Oops: 0000 [#1] SMP
[ 90.958061] CPU 1
[ 90.958103] Modules linked in: ext4 mbcache jbd2 crc16 raid1 md_mod
netconsole configfs nbd dm_mirror dm_region_hash dm_log dm_mod btrfs
zlib_deflate crc32c libcrc32c usbhid hid sr_mod cdrom sd_mod crc_t10dif
ata_generic ohci_hcd ahci libahci pata_atiixp ehci_hcd r8169 mii libata
usbcore usb_common scsi_mod [last unloaded: scsi_wait_scan]
[ 90.959794]
[ 90.959849] Pid: 0, comm: swapper/1 Not tainted 3.2.0-1-amd64 #1 System
manufacturer System Product Name/M5A78L
[ 90.960046] RIP: 0010:[<ffffffffa01dd2c1>] [<ffffffffa01dd2c1>]
bitmap_endwrite+0x131/0x18f [md_mod]
[ 90.960167] RSP: 0018:ffff88022fc43cb0 EFLAGS: 00010046
[ 90.960291] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8802236b70c0
[ 90.960355] RBP: 0000000000004ff8 R08: 0000000000000000 R09:
00000000000163a8
[ 90.960418] R10: 00000000000163a8 R11: 00000000000163a8 R12:
0000000000000008
[ 90.960482] R13: ffff8802236b70fc R14: 0000000000000202 R15:
0000000000000001
[ 90.961330] FS: 00007f1170c90700(0000) GS:ffff88022fc40000(0000)
knlGS:0000000000000000
[ 90.961330] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 90.961330] CR2: 0000000000000010 CR3: 0000000223e5b000 CR4:
00000000000006e0
[ 90.961330] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 90.961330] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 90.961330] Process swapper/1 (pid: 0, threadinfo ffff880226ce4000, task
ffff880226cd60c0)
[ 90.961330] Stack:
[ 90.961330] 000000000000000a ffff8802236b7178 0000000000000000
000000000001b008
[ 90.961330] 0000000000000286 ffff880224f66340 ffff880223702e40
ffff880224f6a988
[ 90.961330] 000000000000000a ffff880222770a80 0000000000000000
ffffffffa01a6808
[ 90.961330] Call Trace:
[ 90.961330] <IRQ>
[ 90.961330] [<ffffffffa01a6808>] ? close_write+0x71/0x7d [raid1]
[ 90.961330] [<ffffffffa01a9677>] ? r1_bio_write_done+0x1e/0x37 [raid1]
[ 90.961330] [<ffffffffa01a97a8>] ? raid1_end_write_request+0x118/0x134
[raid1]
[ 90.961330] [<ffffffff81195904>] ? blk_update_request+0x18c/0x30a
[ 90.961330] [<ffffffff8104c15c>] ? raise_softirq_irqoff+0x6/0x25
[ 90.961330] [<ffffffff81195a9a>] ? blk_update_bidi_request+0x18/0x63
[ 90.961330] [<ffffffff811967ed>] ? blk_end_bidi_request+0x17/0x52
[ 90.961330] [<ffffffffa00086bc>] ? scsi_io_completion+0x1c5/0x4ca
[scsi_mod]
[ 90.961330] [<ffffffff81340824>] ? _raw_spin_lock_irqsave+0x9/0x25
[ 90.961330] [<ffffffff8119a508>] ? blk_done_softirq+0x65/0x74
[ 90.961330] [<ffffffff8104be90>] ? __do_softirq+0xb9/0x177
[ 90.961330] [<ffffffff81347a2c>] ? call_softirq+0x1c/0x30
[ 90.961330] [<ffffffff8100f875>] ? do_softirq+0x3c/0x7b
[ 90.961330] [<ffffffff8104c0f8>] ? irq_exit+0x3c/0x9a
[ 90.961330] [<ffffffff81027601>] ? x2apic_cluster_probe+0x25/0x7f
[ 90.961330] [<ffffffff8100f5a5>] ? do_IRQ+0x82/0x98
[ 90.961330] [<ffffffff81340aee>] ? common_interrupt+0x6e/0x6e
[ 90.961330] <EOI>
[ 90.961330] [<ffffffff8102b2c4>] ? native_safe_halt+0x2/0x3
[ 90.961330] [<ffffffff8101448c>] ? default_idle+0x47/0x7f
[ 90.961330] [<ffffffff81014583>] ? amd_e400_idle+0xbf/0xe4
[ 90.961330] [<ffffffff8100d25f>] ? cpu_idle+0xaf/0xf2
[ 90.961330] [<ffffffff81332d86>] ? start_secondary+0x1d5/0x1db
[ 90.961330] Code: 77 aa e5 e0 48 8b 04 24 66 8b 10 ff ca 66 83 fa 02 66
89 10 77 2e 48 8b 4b 20 48 89 ee 48 89 df 83 e9 09 48 d3 ee e8 f9 f4 ff ff
<48> 8b 40 10 48 8b 53 58 8d 04 85 01 00 00 00 0f ab 02 c7 43 78
[ 90.961330] RIP [<ffffffffa01dd2c1>] bitmap_endwrite+0x131/0x18f [md_mod]
[ 90.961330] RSP <ffff88022fc43cb0>
[ 90.961330] CR2: 0000000000000010
[ 90.961330] ---[ end trace ab37057796e30798 ]---
[ 90.961330] Kernel panic - not syncing: Fatal exception in interrupt
[ 90.961330] Pid: 0, comm: swapper/1 Tainted: G D 3.2.0-1-amd64 #1
[ 90.961330] Call Trace:
[ 90.961330] <IRQ> [<ffffffff8133a15d>] ? panic+0x95/0x1a5
[ 90.961330] [<ffffffff81341646>] ? oops_end+0xa9/0xb6
[ 90.961330] [<ffffffff81339aa9>] ? no_context+0x1ff/0x20e
[ 90.961330] [<ffffffff81343659>] ? do_page_fault+0x1a8/0x337
[ 90.961330] [<ffffffff8103f457>] ? try_to_wake_up+0x187/0x197
[ 90.961330] [<ffffffff810c8c52>] ? zone_page_state_add+0x14/0x23
[ 90.961330] [<ffffffff810c946c>] ? __mod_zone_page_state+0x3c/0x45
[ 90.961330] [<ffffffff810b9277>] ? free_pcppages_bulk+0xf9/0x10c
[ 90.961330] [<ffffffff81340db5>] ? page_fault+0x25/0x30
[ 90.961330] [<ffffffffa01dd2c1>] ? bitmap_endwrite+0x131/0x18f [md_mod]
[ 90.961330] [<ffffffffa01a6808>] ? close_write+0x71/0x7d [raid1]
[ 90.961330] [<ffffffffa01a9677>] ? r1_bio_write_done+0x1e/0x37 [raid1]
[ 90.961330] [<ffffffffa01a97a8>] ? raid1_end_write_request+0x118/0x134
[raid1]
[ 90.961330] [<ffffffff81195904>] ? blk_update_request+0x18c/0x30a
[ 90.961330] [<ffffffff8104c15c>] ? raise_softirq_irqoff+0x6/0x25
[ 90.961330] [<ffffffff81195a9a>] ? blk_update_bidi_request+0x18/0x63
[ 90.961330] [<ffffffff811967ed>] ? blk_end_bidi_request+0x17/0x52
[ 90.961330] [<ffffffffa00086bc>] ? scsi_io_completion+0x1c5/0x4ca
[scsi_mod]
[ 90.961330] [<ffffffff81340824>] ? _raw_spin_lock_irqsave+0x9/0x25
[ 90.961330] [<ffffffff8119a508>] ? blk_done_softirq+0x65/0x74
[ 90.961330] [<ffffffff8104be90>] ? __do_softirq+0xb9/0x177
[ 90.961330] [<ffffffff81347a2c>] ? call_softirq+0x1c/0x30
[ 90.961330] [<ffffffff8100f875>] ? do_softirq+0x3c/0x7b
[ 90.961330] [<ffffffff8104c0f8>] ? irq_exit+0x3c/0x9a
[ 90.961330] [<ffffffff81027601>] ? x2apic_cluster_probe+0x25/0x7f
[ 90.961330] [<ffffffff8100f5a5>] ? do_IRQ+0x82/0x98
[ 90.961330] [<ffffffff81340aee>] ? common_interrupt+0x6e/0x6e
[ 90.961330] <EOI> [<ffffffff8102b2c4>] ? native_safe_halt+0x2/0x3
[ 90.961330] [<ffffffff8101448c>] ? default_idle+0x47/0x7f
[ 90.961330] [<ffffffff81014583>] ? amd_e400_idle+0xbf/0xe4
[ 90.961330] [<ffffffff8100d25f>] ? cpu_idle+0xaf/0xf2
[ 90.961330] [<ffffffff81332d86>] ? start_secondary+0x1d5/0x1db
--
Ciao, Flavio
Those who do not understand Unix are condemned to reinvent it, poorly.
-- Henry Spencer
Reply to: