We use a modified nbdkit ("cbdkit") in production with kernel nbd to run a loopback device for our CloudBD Disks (
www.cloudbd.io). We have observed kernel deadlocks when using nbdkit due to memory pressure and have had to take steps such as preallocating and mlocking all nbdkit and plugin memory. Although our plugin is a loopback over a unix domain socket, we still need to send data back out over the network to object storage. This creates additional memory pressure on the system. Preallocating and locking the memory in the nbd loopback device write path and reserving enough kernel memory for the outgoing network writes we are able to avoid the kernel deadlock even under extremely heavy load on low memory systems.
I am able to reproduce a kernel deadlock by putting a low memory system under heavy load with our DEBUG level logging to the highest setting. This causes rsyslog to try to allocate memory while we are in the loopback device write path that is trying to clear out buffer cache and this can wedge the system.
- Shaun McDowell