[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[PATCH v2 0/5] nbd: eliminate queue freeze/unfreeze overhead in connection setup



This series eliminates the blk_mq_freeze_queue()/unfreeze_queue()
overhead that currently occurs during nbd connection setup.  On
multi-core systems, each freeze cycle involves waiting for percpu_ref
to drain, and blk_mq_update_nr_hw_queues() further adds RCU grace
period waits (synchronize_rcu/srcu for elevator switch, kfree_rcu on
the old hctx array).  These delays are significant when nbd devices
must come online quickly.

The series proceeds in three stages:

1. Cleanups that simplify the sock management code and remove dead
   logic (patches 1-3).

2. A structural change that replaces the krealloc-based socks pointer
   array with an xarray (patch 2), which is the key prerequisite for
   removing the queue freeze from nbd_add_socket().  With xarray,
   concurrent readers via xa_load() never observe a partially
   initialized socket, and xa_store() is safe under RCU without
   freezing the queue.

3. Two patches that target the remaining freeze/unfreeze points:
   - Patch 4 removes the freeze from nbd_add_socket(), since the
     xarray no longer requires it.  This alone reduces connection
     setup time from ~4.5s to ~0.26s with 256 connections (-C 256).
   - Patch 5 (new) targets nbd_start_device(), which still calls
     blk_mq_update_nr_hw_queues() when nr_hw_queues differs from
     num_connections.  For devices created via the netlink connect
     path, the connection count is known upfront from NBD_ATTR_SOCKETS,
     so we can set nr_hw_queues correctly at device creation time and
     skip the runtime update entirely.

The ioctl path (NBD_SET_SOCK + NBD_DO_IT) remains fully functional:
pre-created devices with nbds_max>0 default to nr_hw_queues=1, and
nbd_start_device() still calls blk_mq_update_nr_hw_queues() when the
hardware queue count needs adjustment.

v1->v2:
1. rewrite cover letter
2. add patch 5

Long Li (4):
  nbd: simplify find_fallback() by removing redundant logic
  nbd: replace socks pointer array with xarray
  nbd: remove redundant num_connections boundary checks
  nbd: remove queue freeze in nbd_add_socket

Yang Erkun (1):
  nbd: set nr_hw_queues at device creation to skip queue freeze

 drivers/block/nbd.c | 236 ++++++++++++++++++++++++++------------------
 1 file changed, 139 insertions(+), 97 deletions(-)

-- 
2.52.0


Reply to: