[PATCH v2 0/5] nbd: eliminate queue freeze/unfreeze overhead in connection setup
- To: josef@toxicpanda.com, axboe@kernel.dk, hch@lst.de, yukuai@kernel.org
- Cc: yi.zhang@huawei.com, chengzhihao1@huawei.com, echo.chenlin@huawei.com, leo.lilong@huaweicloud.com, wangkefeng.wang@huawei.com, linux-block@vger.kernel.org, nbd@other.debian.org
- Subject: [PATCH v2 0/5] nbd: eliminate queue freeze/unfreeze overhead in connection setup
- From: Yang Erkun <yangerkun@huawei.com>
- Date: Thu, 25 Jun 2026 16:44:53 +0800
- Message-id: <[🔎] 20260625084458.4171890-1-yangerkun@huawei.com>
This series eliminates the blk_mq_freeze_queue()/unfreeze_queue()
overhead that currently occurs during nbd connection setup. On
multi-core systems, each freeze cycle involves waiting for percpu_ref
to drain, and blk_mq_update_nr_hw_queues() further adds RCU grace
period waits (synchronize_rcu/srcu for elevator switch, kfree_rcu on
the old hctx array). These delays are significant when nbd devices
must come online quickly.
The series proceeds in three stages:
1. Cleanups that simplify the sock management code and remove dead
logic (patches 1-3).
2. A structural change that replaces the krealloc-based socks pointer
array with an xarray (patch 2), which is the key prerequisite for
removing the queue freeze from nbd_add_socket(). With xarray,
concurrent readers via xa_load() never observe a partially
initialized socket, and xa_store() is safe under RCU without
freezing the queue.
3. Two patches that target the remaining freeze/unfreeze points:
- Patch 4 removes the freeze from nbd_add_socket(), since the
xarray no longer requires it. This alone reduces connection
setup time from ~4.5s to ~0.26s with 256 connections (-C 256).
- Patch 5 (new) targets nbd_start_device(), which still calls
blk_mq_update_nr_hw_queues() when nr_hw_queues differs from
num_connections. For devices created via the netlink connect
path, the connection count is known upfront from NBD_ATTR_SOCKETS,
so we can set nr_hw_queues correctly at device creation time and
skip the runtime update entirely.
The ioctl path (NBD_SET_SOCK + NBD_DO_IT) remains fully functional:
pre-created devices with nbds_max>0 default to nr_hw_queues=1, and
nbd_start_device() still calls blk_mq_update_nr_hw_queues() when the
hardware queue count needs adjustment.
v1->v2:
1. rewrite cover letter
2. add patch 5
Long Li (4):
nbd: simplify find_fallback() by removing redundant logic
nbd: replace socks pointer array with xarray
nbd: remove redundant num_connections boundary checks
nbd: remove queue freeze in nbd_add_socket
Yang Erkun (1):
nbd: set nr_hw_queues at device creation to skip queue freeze
drivers/block/nbd.c | 236 ++++++++++++++++++++++++++------------------
1 file changed, 139 insertions(+), 97 deletions(-)
--
2.52.0
Reply to: