Bug#946354: linux-image-4.19.0-6-amd64: Filesystem access gets stuck when LVM snapshot creation interrupted
Package: src:linux
Version: 4.19.67-2+deb10u2
Severity: critical
Justification: breaks the whole system
Dear Maintainer,
I'm using LVM snapshots for creating consistent backups of ext4 filesystems.
Since a few months I'm experiencing severe problems causing the whole system
to get stuck. First I thought this was a hardware issue, but I found some
ways to reproduce this or a similar issue (even on a different system in a VM).
Steps to reproduce this issue:
Have a LVM set up with one LV (test, size: 2G) formatted with ext4 and
mounted. Do some work on the mounted FS (e.g., using fio) and create a
snapshot while the FS is busy using
lvcreate --snapshot -p r -L 1G --name snap /dev/vg/test
and issue Ctrl+C immediately before the command finishes. Now the system
will get into a broken state (this is especially a problem if you snapshot
/var): All further writes to the mounted LV will be stalled forever as well
as LVM commands such as pvdisplay and lvs.
After a few minutes the are the following errors logged in the kernel ring
buffer:
[ 605.032827] INFO: task pvdisplay:878 blocked for more than 120 seconds.
[ 605.032926] Not tainted 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[ 605.033000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 605.033088] pvdisplay D 0 878 543 0x80000004
[ 605.033094] Call Trace:
[ 605.033113] ? __schedule+0x2a2/0x870
[ 605.033119] ? _cond_resched+0x15/0x30
[ 605.033124] schedule+0x28/0x80
[ 605.033130] schedule_timeout+0x26d/0x390
[ 605.033139] ? __kfifo_from_user_r+0xb0/0xb0
[ 605.033144] ? __percpu_ref_switch_mode+0xd4/0x180
[ 605.033149] wait_for_completion+0x11f/0x190
[ 605.033159] ? wake_up_q+0x70/0x70
[ 605.033166] exit_aio+0xdc/0xe3
[ 605.033177] mmput+0x28/0x130
[ 605.033183] do_exit+0x290/0xb90
[ 605.033190] ? __switch_to_asm+0x41/0x70
[ 605.033195] ? __switch_to_asm+0x41/0x70
[ 605.033200] ? __switch_to_asm+0x35/0x70
[ 605.033204] ? __switch_to_asm+0x41/0x70
[ 605.033209] ? __switch_to+0x115/0x440
[ 605.033214] ? __switch_to_asm+0x41/0x70
[ 605.033219] do_group_exit+0x3a/0xa0
[ 605.033226] get_signal+0x137/0x790
[ 605.033231] do_signal+0x36/0x6c0
[ 605.033239] ? finish_wait+0x80/0x80
[ 605.033246] ? __hrtimer_init+0xb0/0xb0
[ 605.033250] ? do_io_getevents+0x7c/0xc0
[ 605.033260] exit_to_usermode_loop+0x89/0xf0
[ 605.033266] do_syscall_64+0x10d/0x110
[ 605.033272] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 605.033278] RIP: 0033:0x7f2ad18a9f59
[ 605.033290] Code: Bad RIP value.
[ 605.033293] RSP: 002b:00007ffca85d4628 EFLAGS: 00000246 ORIG_RAX: 00000000000000d0
[ 605.033298] RAX: fffffffffffffffc RBX: 00007f2ad154a700 RCX: 00007f2ad18a9f59
[ 605.033299] RDX: 0000000000000040 RSI: 0000000000000001 RDI: 00007f2ad1255000
[ 605.033301] RBP: 00007f2ad1255000 R08: 0000000000000000 R09: 00000002a85d46b0
[ 605.033303] R10: 00007ffca85d46b0 R11: 0000000000000246 R12: 0000000000000001
[ 605.033305] R13: 0000000000000000 R14: 0000000000000040 R15: 00007ffca85d46b0
[ 605.033314] INFO: task fio:1010 blocked for more than 120 seconds.
[ 605.033385] Not tainted 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[ 605.033458] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 605.033542] fio D 0 1010 1006 0x00000000
[ 605.033546] Call Trace:
[ 605.033553] ? __schedule+0x2a2/0x870
[ 605.033558] schedule+0x28/0x80
[ 605.033564] rwsem_down_read_failed+0x111/0x180
[ 605.033572] ? kmem_cache_alloc_trace+0x155/0x1d0
[ 605.033581] call_rwsem_down_read_failed+0x14/0x30
[ 605.033587] __percpu_down_read+0x4f/0x80
[ 605.033596] __sb_start_write+0x5f/0x70
[ 605.033603] mnt_want_write+0x20/0x50
[ 605.033609] path_openat+0xc88/0x1480
[ 605.033616] ? alloc_set_pte+0xf2/0x560
[ 605.033624] ? filemap_map_pages+0x360/0x3a0
[ 605.033629] do_filp_open+0x93/0x100
[ 605.033635] ? __handle_mm_fault+0x1090/0x1270
[ 605.033641] ? __check_object_size+0x15d/0x189
[ 605.033647] do_sys_open+0x186/0x210
[ 605.033655] do_syscall_64+0x53/0x110
[ 605.033661] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 605.033664] RIP: 0033:0x7f06d6ea4d0e
[ 605.033669] Code: Bad RIP value.
[ 605.033671] RSP: 002b:00007ffdb6c3df30 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 605.033675] RAX: ffffffffffffffda RBX: 00007f06ca1e9510 RCX: 00007f06d6ea4d0e
[ 605.033677] RDX: 0000000000004042 RSI: 00007f06ca1e9690 RDI: 00000000ffffff9c
[ 605.033678] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000560c2b42dc20
[ 605.033680] R10: 0000000000000180 R11: 0000000000000293 R12: 0000000000004042
[ 605.033682] R13: 0000000000004042 R14: 00007f06a9c1d000 R15: 0000000000000000
[ 725.866073] INFO: task pvdisplay:878 blocked for more than 120 seconds.
[ 725.866161] Not tainted 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[ 725.866235] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 725.866321] pvdisplay D 0 878 543 0x80000004
[ 725.866327] Call Trace:
[ 725.866344] ? __schedule+0x2a2/0x870
[ 725.866349] ? _cond_resched+0x15/0x30
[ 725.866354] schedule+0x28/0x80
[ 725.866360] schedule_timeout+0x26d/0x390
[ 725.866368] ? __kfifo_from_user_r+0xb0/0xb0
[ 725.866372] ? __percpu_ref_switch_mode+0xd4/0x180
[ 725.866378] wait_for_completion+0x11f/0x190
[ 725.866386] ? wake_up_q+0x70/0x70
[ 725.866392] exit_aio+0xdc/0xe3
[ 725.866401] mmput+0x28/0x130
[ 725.866406] do_exit+0x290/0xb90
[ 725.866412] ? __switch_to_asm+0x41/0x70
[ 725.866417] ? __switch_to_asm+0x41/0x70
[ 725.866422] ? __switch_to_asm+0x35/0x70
[ 725.866426] ? __switch_to_asm+0x41/0x70
[ 725.866431] ? __switch_to+0x115/0x440
[ 725.866436] ? __switch_to_asm+0x41/0x70
[ 725.866440] do_group_exit+0x3a/0xa0
[ 725.866447] get_signal+0x137/0x790
[ 725.866452] do_signal+0x36/0x6c0
[ 725.866458] ? finish_wait+0x80/0x80
[ 725.866464] ? __hrtimer_init+0xb0/0xb0
[ 725.866468] ? do_io_getevents+0x7c/0xc0
[ 725.866476] exit_to_usermode_loop+0x89/0xf0
[ 725.866482] do_syscall_64+0x10d/0x110
[ 725.866488] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 725.866493] RIP: 0033:0x7f2ad18a9f59
[ 725.866503] Code: Bad RIP value.
[ 725.866506] RSP: 002b:00007ffca85d4628 EFLAGS: 00000246 ORIG_RAX: 00000000000000d0
[ 725.866510] RAX: fffffffffffffffc RBX: 00007f2ad154a700 RCX: 00007f2ad18a9f59
[ 725.866512] RDX: 0000000000000040 RSI: 0000000000000001 RDI: 00007f2ad1255000
[ 725.866514] RBP: 00007f2ad1255000 R08: 0000000000000000 R09: 00000002a85d46b0
[ 725.866516] R10: 00007ffca85d46b0 R11: 0000000000000246 R12: 0000000000000001
[ 725.866518] R13: 0000000000000000 R14: 0000000000000040 R15: 00007ffca85d46b0
[ 725.866526] INFO: task fio:1010 blocked for more than 120 seconds.
[ 725.866597] Not tainted 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[ 725.866670] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 725.866771] fio D 0 1010 1006 0x00000000
[ 725.866776] Call Trace:
[ 725.866783] ? __schedule+0x2a2/0x870
[ 725.866788] schedule+0x28/0x80
[ 725.866793] rwsem_down_read_failed+0x111/0x180
[ 725.866801] ? kmem_cache_alloc_trace+0x155/0x1d0
[ 725.866808] call_rwsem_down_read_failed+0x14/0x30
[ 725.866814] __percpu_down_read+0x4f/0x80
[ 725.866821] __sb_start_write+0x5f/0x70
[ 725.866827] mnt_want_write+0x20/0x50
[ 725.866833] path_openat+0xc88/0x1480
[ 725.866839] ? alloc_set_pte+0xf2/0x560
[ 725.866845] ? filemap_map_pages+0x360/0x3a0
[ 725.866851] do_filp_open+0x93/0x100
[ 725.866856] ? __handle_mm_fault+0x1090/0x1270
[ 725.866861] ? __check_object_size+0x15d/0x189
[ 725.866868] do_sys_open+0x186/0x210
[ 725.866875] do_syscall_64+0x53/0x110
[ 725.866881] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 725.866884] RIP: 0033:0x7f06d6ea4d0e
[ 725.866889] Code: Bad RIP value.
[ 725.866891] RSP: 002b:00007ffdb6c3df30 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 725.866895] RAX: ffffffffffffffda RBX: 00007f06ca1e9510 RCX: 00007f06d6ea4d0e
[ 725.866897] RDX: 0000000000004042 RSI: 00007f06ca1e9690 RDI: 00000000ffffff9c
[ 725.866899] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000560c2b42dc20
[ 725.866900] R10: 0000000000000180 R11: 0000000000000293 R12: 0000000000004042
[ 725.866902] R13: 0000000000004042 R14: 00007f06a9c1d000 R15: 0000000000000000
[ 846.697015] INFO: task pvdisplay:878 blocked for more than 120 seconds.
[ 846.697193] Not tainted 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[ 846.697272] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.697358] pvdisplay D 0 878 543 0x80000004
[ 846.697365] Call Trace:
[ 846.697384] ? __schedule+0x2a2/0x870
[ 846.697388] ? _cond_resched+0x15/0x30
[ 846.697393] schedule+0x28/0x80
[ 846.697399] schedule_timeout+0x26d/0x390
[ 846.697459] ? __kfifo_from_user_r+0xb0/0xb0
[ 846.697466] ? __percpu_ref_switch_mode+0xd4/0x180
[ 846.697471] wait_for_completion+0x11f/0x190
[ 846.697481] ? wake_up_q+0x70/0x70
[ 846.697487] exit_aio+0xdc/0xe3
[ 846.697497] mmput+0x28/0x130
[ 846.697502] do_exit+0x290/0xb90
[ 846.697509] ? __switch_to_asm+0x41/0x70
[ 846.697514] ? __switch_to_asm+0x41/0x70
[ 846.697518] ? __switch_to_asm+0x35/0x70
[ 846.697523] ? __switch_to_asm+0x41/0x70
[ 846.697528] ? __switch_to+0x115/0x440
[ 846.697533] ? __switch_to_asm+0x41/0x70
[ 846.697537] do_group_exit+0x3a/0xa0
[ 846.697544] get_signal+0x137/0x790
[ 846.697549] do_signal+0x36/0x6c0
[ 846.697555] ? finish_wait+0x80/0x80
[ 846.697562] ? __hrtimer_init+0xb0/0xb0
[ 846.697566] ? do_io_getevents+0x7c/0xc0
[ 846.697574] exit_to_usermode_loop+0x89/0xf0
[ 846.697579] do_syscall_64+0x10d/0x110
[ 846.697586] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 846.697592] RIP: 0033:0x7f2ad18a9f59
[ 846.697604] Code: Bad RIP value.
[ 846.697607] RSP: 002b:00007ffca85d4628 EFLAGS: 00000246 ORIG_RAX: 00000000000000d0
[ 846.697611] RAX: fffffffffffffffc RBX: 00007f2ad154a700 RCX: 00007f2ad18a9f59
[ 846.697613] RDX: 0000000000000040 RSI: 0000000000000001 RDI: 00007f2ad1255000
[ 846.697615] RBP: 00007f2ad1255000 R08: 0000000000000000 R09: 00000002a85d46b0
[ 846.697617] R10: 00007ffca85d46b0 R11: 0000000000000246 R12: 0000000000000001
[ 846.697618] R13: 0000000000000000 R14: 0000000000000040 R15: 00007ffca85d46b0
On the bare metal machine I also get errors like the following:
[ 203.920873] ------------[ cut here ]------------
[ 203.920906] generic_make_request: Trying to write to read-only block-device dm-15 (partno 0)
[ 203.920950] WARNING: CPU: 5 PID: 452 at /build/linux-3Qtbv4/linux-4.19.67/block/blk-core.c:2179 generic_make_request_checks+0x31d/0x710
[ 203.920989] Modules linked in: cpufreq_conservative cpufreq_powersave cpufreq_userspace bridge stp llc nft_masq_ipv4 nft_masq nft_nat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv6 nf_log_ipv4 nf_log_common intel_rapl nft_reject_in
[ 203.921178] crc16 mbcache jbd2 fscrypto ecb raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multipath linear dm_snapshot dm_bufio dm_mod raid1 md_mod sd_mod crc32c
[ 203.921308] CPU: 5 PID: 452 Comm: kworker/5:2 Not tainted 4.19.0-0.bpo.6-amd64 #1 Debian 4.19.67-2+deb10u2~bpo9+1
[ 203.921374] Hardware name: System manufacturer System Product Name/P8B WS, BIOS 2106 07/16/2012
[ 203.921433] Workqueue: ksnaphd do_metadata [dm_snapshot]
[ 203.921475] RIP: 0010:generic_make_request_checks+0x31d/0x710
[ 203.921515] Code: 5c 03 00 00 48 8d 74 24 10 48 89 df c6 05 41 d8 d6 00 01 e8 95 6f 01 00 44 89 e2 48 89 c6 48 c7 c7 38 55 a8 98 e8 83 08 cf ff <0f> 0b 4c 8b 63 08 8b 43 30 c1 e8 09 85 c0 49 8b 74 24 50 74 1e 48
[ 203.921596] RSP: 0018:ffffa5dd03a9bb70 EFLAGS: 00010286
[ 203.921647] RAX: 0000000000000000 RBX: ffff93667ecad900 RCX: 0000000000000006
[ 203.921693] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff93667f1566b0
[ 203.921739] RBP: ffff936672c49360 R08: 0000000000000000 R09: 0000000000000301
[ 203.921785] R10: ffffa5dd03a9bc58 R11: ffffffff991f16ed R12: 0000000000000000
[ 203.921831] R13: 0000000000000000 R14: 0000000000001000 R15: 0000000000000000
[ 203.921878] FS: 0000000000000000(0000) GS:ffff93667f140000(0000) knlGS:0000000000000000
[ 203.921938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 203.921982] CR2: 00007fff9c1fda60 CR3: 00000000a9c0a006 CR4: 00000000001606e0
[ 203.922027] Call Trace:
[ 203.922072] ? kmem_cache_alloc+0x15b/0x1c0
[ 203.922115] generic_make_request+0x66/0x410
[ 203.922159] ? submit_bio+0x6c/0x140
[ 203.922199] submit_bio+0x6c/0x140
[ 203.922241] ? bio_add_page+0x48/0x60
[ 203.922289] dispatch_io+0x362/0x460 [dm_mod]
[ 203.922334] ? bio_next_page+0xb0/0xb0 [dm_mod]
[ 203.922380] ? km_next_page+0x20/0x20 [dm_mod]
[ 203.922426] sync_io+0xc2/0x120 [dm_mod]
[ 203.922469] ? __switch_to_asm+0x41/0x70
[ 203.922514] dm_io+0x207/0x240 [dm_mod]
[ 203.922555] ? __switch_to_asm+0x41/0x70
[ 203.922599] ? bio_next_page+0xb0/0xb0 [dm_mod]
[ 203.922645] ? km_next_page+0x20/0x20 [dm_mod]
[ 203.922689] do_metadata+0x1d/0x30 [dm_snapshot]
[ 203.922733] process_one_work+0x191/0x370
[ 203.922776] worker_thread+0x4f/0x3b0
[ 203.922818] kthread+0xf8/0x130
[ 203.922859] ? rescuer_thread+0x340/0x340
[ 203.922901] ? kthread_create_worker_on_cpu+0x70/0x70
[ 203.922944] ret_from_fork+0x35/0x40
[ 203.922986] ---[ end trace 694f9987d4209aea ]---
and:
[1804870.466049] INFO: task jbd2/dm-3-8:638 blocked for more than 120 seconds.
[1804870.466104] Tainted: G W 4.19.0-0.bpo.6-amd64 #1 Debian 4.19.67-2+deb10u1~bpo9+1
[1804870.466168] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1804870.466230] jbd2/dm-3-8 D 0 638 2 0x80000000
[1804870.466277] Call Trace:
[1804870.466328] ? __schedule+0x3f5/0x880
[1804870.466373] schedule+0x32/0x80
[1804870.466415] schedule_timeout+0x1e5/0x350
[1804870.466460] __down+0x9e/0x100
[1804870.466503] ? down+0x3b/0x50
[1804870.466545] down+0x3b/0x50
[1804870.466590] start_copy+0x8a/0xd0 [dm_snapshot]
[1804870.466636] __origin_write+0x152/0x270 [dm_snapshot]
[1804870.466683] do_origin.isra.20+0x7e/0x90 [dm_snapshot]
[1804870.466734] __map_bio+0x42/0x170 [dm_mod]
[1804870.466781] __split_and_process_non_flush+0x12c/0x220 [dm_mod]
[1804870.466832] ? __process_bio+0x170/0x170 [dm_mod]
[1804870.466878] __split_and_process_bio+0xb2/0x1a0 [dm_mod]
[1804870.466928] __dm_make_request.isra.33+0x3f/0xa0 [dm_mod]
[1804870.466975] generic_make_request+0x1e7/0x410
[1804870.467019] ? submit_bio+0x6c/0x140
[1804870.467061] submit_bio+0x6c/0x140
[1804870.467104] ? guard_bio_eod+0x36/0x100
[1804870.467147] submit_bh_wbc+0x163/0x190
[1804870.467195] ? jbd2_journal_begin_ordered_truncate+0xa0/0xa0 [jbd2]
[1804870.467245] jbd2_journal_commit_transaction+0x5f5/0x18b0 [jbd2]
[1804870.467294] ? __switch_to_asm+0x35/0x70
[1804870.467336] ? __switch_to_asm+0x41/0x70
[1804870.467379] ? __switch_to_asm+0x35/0x70
[1804870.467426] ? kjournald2+0xc1/0x260 [jbd2]
[1804870.467472] kjournald2+0xc1/0x260 [jbd2]
[1804870.467516] ? remove_wait_queue+0x60/0x60
[1804870.467560] kthread+0xf8/0x130
[1804870.467604] ? commit_timeout+0x10/0x10 [jbd2]
[1804870.467648] ? kthread_create_worker_on_cpu+0x70/0x70
[1804870.467693] ret_from_fork+0x35/0x40
On the bare metal machine, these lock ups came out of the sudden from an
automated backup script which runs several times a day creating and deleting
several snapshots and occur every few weeks/months. Then also the WA went up
to 913.25 (nearly all CUP cores are waiting state wheras iotop shows 0.0 MB
read/write). The Ctrl+C case produces an issue which looks quite the same.
Also affected is the latest stretch kernel (vmlinuz-4.9.0-11-amd64).
-- Package-specific info:
** Version:
Linux version 4.19.0-6-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
** Command line:
BOOT_IMAGE=/boot/vmlinuz-4.19.0-6-amd64 root=UUID=6dd675ec-8c34-4f10-bcba-10a014c0f46a ro quiet
** Not tainted
** Kernel log:
Unable to read kernel log; any relevant messages should be attached
** Model information
sys_vendor: VMware, Inc.
product_name: VMware Virtual Platform
product_version: None
chassis_vendor: No Enclosure
chassis_version: N/A
bios_vendor: Phoenix Technologies LTD
bios_version: 6.00
board_vendor: Intel Corporation
board_name: 440BX Desktop Reference Platform
board_version: None
** Loaded modules:
dm_snapshot
dm_bufio
crct10dif_pclmul
crc32_pclmul
ghash_clmulni_intel
vmw_balloon
intel_rapl_perf
joydev
evdev
serio_raw
pcspkr
vmwgfx
ttm
drm_kms_helper
drm
sg
vmw_vmci
ac
button
dm_mod
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
crc32c_generic
fscrypto
ecb
sd_mod
sr_mod
cdrom
ata_generic
crc32c_intel
aesni_intel
ahci
libahci
ata_piix
aes_x86_64
crypto_simd
cryptd
glue_helper
libata
psmouse
scsi_mod
e1000
i2c_piix4
-- System Information:
Debian Release: 10.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages linux-image-4.19.0-6-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.133+deb10u1
ii kmod 26-1
ii linux-base 4.6
Versions of packages linux-image-4.19.0-6-amd64 recommends:
ii apparmor 2.13.2-10
ii firmware-linux-free 3.4
Versions of packages linux-image-4.19.0-6-amd64 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 2.02+dfsg1-20
pn linux-doc-4.19 <none>
Versions of packages linux-image-4.19.0-6-amd64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>
-- no debconf information
Reply to: