Re: [PATCH] nbd: handle device refs for DESTROY_ON_DISCONNECT properly
On 2/22/21 1:09 PM, Josef Bacik wrote:
> There exists a race where we can be attempting to create a new nbd
> configuration while a previous configuration is going down, both
> configured with DESTROY_ON_DISCONNECT. Normally devices all have a
> reference of 1, as they won't be cleaned up until the module is torn
> down. However with DESTROY_ON_DISCONNECT we'll make sure that there is
> only 1 reference (generally) on the device for the config itself, and
> then once the config is dropped, the device is torn down.
>
> The race that exists looks like this
>
> TASK1 TASK2
> nbd_genl_connect()
> idr_find()
> refcount_inc_not_zero(nbd)
> * count is 2 here ^^
> nbd_config_put()
> nbd_put(nbd) (count is 1)
> setup new config
> check DESTROY_ON_DISCONNECT
> put_dev = true
> if (put_dev) nbd_put(nbd)
> * free'd here ^^
>
> In nbd_genl_connect() we assume that the nbd ref count will be 2,
> however clearly that won't be true if the nbd device had been setup as
> DESTROY_ON_DISCONNECT with its prior configuration. Fix this by getting
> rid of the runtime flag to check if we need to mess with the nbd device
> refcount, and use the device NBD_DESTROY_ON_DISCONNECT flag to check if
> we need to adjust the ref counts. This was reported by syzkaller with
> the following kasan dump
>
> BUG: KASAN: use-after-free in instrument_atomic_read include/linux/instrumented.h:71 [inline]
> BUG: KASAN: use-after-free in atomic_read include/asm-generic/atomic-instrumented.h:27 [inline]
> BUG: KASAN: use-after-free in refcount_dec_not_one+0x71/0x1e0 lib/refcount.c:76
> Read of size 4 at addr ffff888143bf71a0 by task systemd-udevd/8451
>
> CPU: 0 PID: 8451 Comm: systemd-udevd Not tainted 5.11.0-rc7-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:79 [inline]
> dump_stack+0x107/0x163 lib/dump_stack.c:120
> print_address_description.constprop.0.cold+0x5b/0x2f8 mm/kasan/report.c:230
> __kasan_report mm/kasan/report.c:396 [inline]
> kasan_report.cold+0x79/0xd5 mm/kasan/report.c:413
> check_memory_region_inline mm/kasan/generic.c:179 [inline]
> check_memory_region+0x13d/0x180 mm/kasan/generic.c:185
> instrument_atomic_read include/linux/instrumented.h:71 [inline]
> atomic_read include/asm-generic/atomic-instrumented.h:27 [inline]
> refcount_dec_not_one+0x71/0x1e0 lib/refcount.c:76
> refcount_dec_and_mutex_lock+0x19/0x140 lib/refcount.c:115
> nbd_put drivers/block/nbd.c:248 [inline]
> nbd_release+0x116/0x190 drivers/block/nbd.c:1508
> __blkdev_put+0x548/0x800 fs/block_dev.c:1579
> blkdev_put+0x92/0x570 fs/block_dev.c:1632
> blkdev_close+0x8c/0xb0 fs/block_dev.c:1640
> __fput+0x283/0x920 fs/file_table.c:280
> task_work_run+0xdd/0x190 kernel/task_work.c:140
> tracehook_notify_resume include/linux/tracehook.h:189 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:174 [inline]
> exit_to_user_mode_prepare+0x249/0x250 kernel/entry/common.c:201
> __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7fc1e92b5270
> Code: 73 01 c3 48 8b 0d 38 7d 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 59 c1 20 00 00 75 10 b8 03 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 ee fb ff ff 48 89 04 24
> RSP: 002b:00007ffe8beb2d18 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
> RAX: 0000000000000000 RBX: 0000000000000007 RCX: 00007fc1e92b5270
> RDX: 000000000aba9500 RSI: 0000000000000000 RDI: 0000000000000007
> RBP: 00007fc1ea16f710 R08: 000000000000004a R09: 0000000000000008
> R10: 0000562f8cb0b2a8 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000562f8cb0afd0 R14: 0000000000000003 R15: 000000000000000e
>
> Allocated by task 1:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_set_track mm/kasan/common.c:46 [inline]
> set_alloc_info mm/kasan/common.c:401 [inline]
> ____kasan_kmalloc.constprop.0+0x82/0xa0 mm/kasan/common.c:429
> kmalloc include/linux/slab.h:552 [inline]
> kzalloc include/linux/slab.h:682 [inline]
> nbd_dev_add+0x44/0x8e0 drivers/block/nbd.c:1673
> nbd_init+0x250/0x271 drivers/block/nbd.c:2394
> do_one_initcall+0x103/0x650 init/main.c:1223
> do_initcall_level init/main.c:1296 [inline]
> do_initcalls init/main.c:1312 [inline]
> do_basic_setup init/main.c:1332 [inline]
> kernel_init_freeable+0x605/0x689 init/main.c:1533
> kernel_init+0xd/0x1b8 init/main.c:1421
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296
>
> Freed by task 8451:
> kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
> kasan_set_track+0x1c/0x30 mm/kasan/common.c:46
> kasan_set_free_info+0x20/0x30 mm/kasan/generic.c:356
> ____kasan_slab_free+0xe1/0x110 mm/kasan/common.c:362
> kasan_slab_free include/linux/kasan.h:192 [inline]
> slab_free_hook mm/slub.c:1547 [inline]
> slab_free_freelist_hook+0x5d/0x150 mm/slub.c:1580
> slab_free mm/slub.c:3143 [inline]
> kfree+0xdb/0x3b0 mm/slub.c:4139
> nbd_dev_remove drivers/block/nbd.c:243 [inline]
> nbd_put.part.0+0x180/0x1d0 drivers/block/nbd.c:251
> nbd_put drivers/block/nbd.c:295 [inline]
> nbd_config_put+0x6dd/0x8c0 drivers/block/nbd.c:1242
> nbd_release+0x103/0x190 drivers/block/nbd.c:1507
> __blkdev_put+0x548/0x800 fs/block_dev.c:1579
> blkdev_put+0x92/0x570 fs/block_dev.c:1632
> blkdev_close+0x8c/0xb0 fs/block_dev.c:1640
> __fput+0x283/0x920 fs/file_table.c:280
> task_work_run+0xdd/0x190 kernel/task_work.c:140
> tracehook_notify_resume include/linux/tracehook.h:189 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:174 [inline]
> exit_to_user_mode_prepare+0x249/0x250 kernel/entry/common.c:201
> __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> The buggy address belongs to the object at ffff888143bf7000
> which belongs to the cache kmalloc-1k of size 1024
> The buggy address is located 416 bytes inside of
> 1024-byte region [ffff888143bf7000, ffff888143bf7400)
> The buggy address belongs to the page:
> page:000000005238f4ce refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x143bf0
> head:000000005238f4ce order:3 compound_mapcount:0 compound_pincount:0
> flags: 0x57ff00000010200(slab|head)
> raw: 057ff00000010200 ffffea00004b1400 0000000300000003 ffff888010c41140
> raw: 0000000000000000 0000000000100010 00000001ffffffff 0000000000000000
> page dumped because: kasan: bad access detected
>
> Memory state around the buggy address:
> ffff888143bf7080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff888143bf7100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> ffff888143bf7180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ^
> ffff888143bf7200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Applied, thanks.
--
Jens Axboe
Reply to: