Re: Setting up NBD server / client - block nbd4: Other side returned error (22) issue
On Fri, Aug 27, 2021 at 12:46:40PM +0100, Simon Fernandez wrote:
> Thanks For the swift reply Richard!
>
> Ive just realised it’s you blog I’ve been reading! (Love the 'about the author'
> bit ; ) )
>
> Hopefully get this working this weekend. Swamped this afternoon.
>
> So if I want to export the device instead of separate directories.
>
> Could I change the config file to something like this
>
> [generic]
> # If you want to run everything as root rather than the nbd user, you
> # may either say "root" in the two following lines, or remove them
> # altogether. Do not remove the [generic] section, however.
> user = pi
> group = pi
> includedir = /etc/nbd-server/conf.d
> allowlist = true
>
> # What follows are export definitions. You may create as much of them as
> # you want, but the section header has to be unique.
> [rpi-C3-food]
> exportname = /dev/sda
> transactionlog = /home/pi/media/500G_FAITH/logs/log_food
> readonly = false
This will export /dev/sda (writably) and as long as /dev/sda is not
used by anything else that's fine. However if /dev/sda is something
that might contain your root filesystem or is mounted on the host
anywhere then it's NOT fine, and in fact will certainly lead to
unrecoverable disk corruption.
If you're looking for an NBD export that is safe to play with, can I
suggest the nbdkit memory plugin? It's a sparse RAM disk and can't do
anything worse than cause your machine to run out of memory if you
fill it up too much.
https://libguestfs.org/nbdkit-memory-plugin.1.html
If you want to export some random files, I'd still suggest floppy
plugin or the similar related plugins in nbdkit.
Rich.
> Then on client
> > sudo nbd-client 192.168.1.14 -N rpi-C3-food /dev/nbd1
> Then mount it.
> > mkdir /mnt/food
> > mount /dev/nbd1 /mnt/food
>
> Thanks all really appreciate the help.
> Simon
>
>
> PS I tried the list option - gave me an error : /
> pi@rpi-C1:~ $ nbdinfo --list nbd://192.168.1.14:10809/recipes
> nbdinfo: nbd_set_export_name: invalid state: READY: the handle must be newly
> created, or negotiating: Invalid argument
>
>
>
>
>
> On 26 Aug 2021, at 20:48, Richard W.M. Jones <rjones@redhat.com> wrote:
>
> On Thu, Aug 26, 2021 at 07:49:33PM +0100, Simon Fernandez wrote:
>
> Hi folks,
> thanks for the reply Wouter's reply was in junk only found it when I
> saw
> Richards reply and fished it out.
>
> I'm still getting
> $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> Negotiation: ..size = 0MB
> Connected /dev/nbd0
>
> Here are the changes I've made - first I built nbdinfo - thanks for the
> tip
> Richard.
>
> Build GnuTLS from
> https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.16.tar.xz
>
> Build libnbd from
> https://download.libguestfs.org/libnbd/1.8-stable/libnbd-1.8.3.tar.gz
>
> After installing quite a lot of libraries . . .
> ----------------------------------------------------------------------
> Thank you for downloading libnbd 1.8.3
>
> This is how we have configured the optional components for you today:
>
> Optional library features:
>
> TLS support ............................ yes
> NBD URI support ........................ yes
> FUSE support ........................... yes
> Manual pages ........................... yes
> Bash tab completion .................... yes
>
> Language bindings:
>
> Go ..................................... no
> OCaml .................................. no
> Python ................................. no
>
> nbdkit version note - during ./configure
> checking for nbdkit... nbdkit
> checking for nbdkit >= 1.12... no (1.10)
> configure: WARNING: nbdkit is too old, some tests will be disabled
>
> The 'make check' skipped a lot of tests but there were no FAILS
>
> $ nbdinfo --version
> nbdinfo 1.8.3
> libnbd 1.8.3
>
>
> == rpi-C3 (192.168.1.14) server side
>
> Q: Wouter ( w@uter.be ) - Are they readable and writable by the "nbd"
> user?
> R: I've changed the user & group to pi (the logged in user) so should
> be R/W
>
> Note: Richard ( rjones@redhat.com ) - You don't need to serve each
> export on a
> separate port. The NBD
> protocol can serve multiple exports (disks) on the same port, with the
> client requesting which one it wants.
> R: Have remove the port addresses from config
>
> pi@rpi-C3:~ $ cat /etc/nbd-server/config
> [generic]
> # If you want to run everything as root rather than the nbd user, you
> # may either say "root" in the two following lines, or remove them
> # altogether. Do not remove the [generic] section, however.
> user = pi
> group = pi
> includedir = /etc/nbd-server/conf.d
> allowlist = true
>
> # What follows are export definitions. You may create as much of them
> as
> # you want, but the section header has to be unique.
> [images]
> exportname = /home/pi/media/500G_FAITH/images/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_images
> [recipes]
> exportname = /home/pi/media/500G_FAITH/recipes/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_recipes
> [movie]
> exportname = /home/pi/media/500G_FAITH/movies/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_movie
>
>
> One problem here is that NBD is a block device export protocol, not a
> file serving protocol (like eg NFS or SMB).
>
> You can however turn a directory into an NBD export using
> nbdkit-floppy-plugin, eg:
>
> $ nbdkit floppy /home/pi/media/500G_FAITH/recipes/
>
> (https://libguestfs.org/nbdkit-floppy-plugin.1.html)
>
> That exports a VFAT filesystem as a block device which has to be
> mounted:
>
> # nbd-client 192.168.1.14 10809 /dev/nbd0
> # mkdir /mnt/recipes
> # mount /dev/nbd0 /mnt/recipes
>
>
> Q: Wouter ( w@uter.be ) - Do the paths that you point to in the
> exportname
> parameters exist?
> R: YES its a directory with a bunch of images in it. YES - Non zero
> size.
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/recipes/
> y000_Pork_Belly_Wontan____TOCOST.rtf
> y011_Rack_And_Black.rtf
> y002_Salmon_Parfait_Forest_Rosemary_Bread_LE.rtf
> y021_StoneAge_Beef_Taragon_Sauce____TOCOST.rtf
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/images
> 160g? of dough - 250C - 6m.jpg 20190228_163410_monkfish and red
> pepper
> skewers.jpg
> 20190629_202814_couscous w apricots.jpg
> 672_V_cheddar_gorgeous.jpg
> 20181022_134259.jpg 20200124_152329_aubergine & pesto
> stuffing.jpg
> 25_androll_into_pancakes.jpg 665 Celeriac & Cockscombe Mint Soup.jpg
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/movies
> NA #015 - Bread.mp4
> NA #016 - Fisetin - Mayo Clinic Trials.mp4
> NA #042 - Delta Variant COVID.mp4
> NA #044 - Lex Fridman.mp4
>
> pi@rpi-C3:~ $ ls -la /home/pi/media/500G_FAITH/logs
> -rw------- 1 pi pi 56 Aug 26 17:00 log_images
> -rw------- 1 pi pi 56 Aug 26 17:00 log_movie
> -rw------- 1 pi pi 700 Aug 26 17:35 log_recipes
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_images
> %`?%`?
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_movie
> %`?%`?
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_recipes
> %`?`%`?a%`?b%`?c%`?d%`?e%`?f%`?g%`?h%`?i%`?j%`?k%`?l%`?%`?%`?%`?%`?%`?%
> `?%`?%`%
> `? %`?
>
> pi@rpi-C3:~ $ nbdkit --version
> nbdkit 1.1.12
>
>
> == client side
>
> Note: Richard ( rjones@redhat.com ) - Use the 'nbdinfo' tool to examine
> what is
> being served by an NBD
> server. It can query all sorts of information as described in the
> manual:
> R: Built the tool. - here is what it says : /
>
> pi@rpi-C1:~ $ nbdinfo nbd://192.168.1.14:10809/recipes
> protocol: newstyle-fixed without TLS
> export="recipes":
> export-size: 4096
> uri: nbd://192.168.1.14:10809/recipes
> is_rotational: false
> is_read_only: true
> can_cache: false
> can_df: false
> can_fast_zero: false
> can_flush: false
> can_fua: false
> can_multi_conn: true
> can_trim: false
> can_zero: true
>
>
> This looks like a good "recipes" export, although the size is
> unusually small (4K), which is probably because what you're exporting
> here is the directory inode, not the directory - see above.
>
>
> pi@rpi-C1:~ $ nbdkit --version
> nbdkit 1.10.3
>
> pi@rpi-C1:~ $ nbd-client -l 192.168.1.14
> Negotiation: ..
> images
> recipes
> movie
>
>
> nbdinfo can also list exports using the --list option which will tell
> you a lot of detail about every export.
>
>
> pi@rpi-C1:~ $ sudo modprobe nbd
> pi@rpi-C1:~ $ lsmod | grep nbd
> nbd 49152 0
>
> pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> Negotiation: ..size = 0MB
> Connected /dev/nbd0
>
> pi@rpi-C1:~ $ sudo partx -a /dev/nbd0
> partx: /dev/nbd0: failed to read partition table
>
> pi@rpi-C1:~ $ sudo parted -l
> Error: /dev/nbd0: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd0: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> pi@rpi-C1:~ $ dmesg | tail -n 20
> [ 2138.454532] blk_update_request: I/O error, dev nbd0, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.454547] Buffer I/O error on dev nbd0, logical block 0, async
> page read
> [ 2138.457380] block nbd0: Other side returned error (22)
> [ 2138.457497] blk_update_request: I/O error, dev nbd0, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.457520] Buffer I/O error on dev nbd0, logical block 0, async
> page read
> [ 2138.459238] block nbd0: Other side returned error (22)
> [ 2138.459280] blk_update_request: I/O error, dev nbd0, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.459297] Buffer I/O error on dev nbd0, logical block 0, async
> page read
> [ 2138.460993] block nbd0: Other side returned error (22)
> [ 2138.461023] blk_update_request: I/O error, dev nbd0, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.461038] Buffer I/O error on dev nbd0, logical block 0, async
> page read
> [ 2138.462649] block nbd0: Other side returned error (22)
> [ 2138.462685] Buffer I/O error on dev nbd0, logical block 0, async
> page read
> [ 2138.464344] block nbd0: Other side returned error (22)
> [ 2138.466926] block nbd0: Other side returned error (22)
> [ 2138.468506] block nbd0: Other side returned error (22)
> [ 2138.470066] block nbd0: Other side returned error (22)
> [ 2138.471442] block nbd0: Other side returned error (22)
> [ 2138.472978] block nbd0: Other side returned error (22)
> [ 2138.474448] block nbd0: Other side returned error (22)
>
> pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N movies /dev/nbd1
> Negotiation: ..Error: E: server does not support NBD_OPT_GO and dropped
> connection after sending NBD_OPT_EXPORT_NAME. Try -g.
> Exiting.
>
> I'm attempting to serve non empty directories so I don't know why it's
> saying
> 0Mb size?
> Scratching my head here . . any thoughts? :/
>
>
> Rich.
>
>
> Thanks in advance,
> Simon
>
>
> On 12 Aug 2021, at 22:32, Richard W.M. Jones <rjones@redhat.com>
> wrote:
>
> On Thu, Jul 22, 2021 at 01:41:00PM +0100, Simon Fernandez wrote:
>
> Hi folks,
> I’m trying setup a client that connect to a 500Gb ext4 disk on
> another
> machine
> running nbd server.
> Server: raspberry pi 3: uname -r 4.19.66+
> Client: raspberry pi 3: uname -r 5.10.17-v7+
>
> I’ve followed the information here [https://github.com/
> NetworkBlockDevice/nbd]
> and a lot of other sites but I think I’m doing something wrong
> on the
> server
> setup because I get this on the client:
>
> $ sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
> mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev
> /nbd2
>
> Also it appears to be listening on default port - NOT ones
> specified in
> config
> file.
>
> This is how I’ve set up the server & client, it’s probably
> something
> obvious to
> more experienced folk.
>
>
> Wouter has answered this already, I just wanted to add a couple of
> smaller points:
>
> (1) You don't need to serve each export on a separate port. The NBD
> protocol can serve multiple exports (disks) on the same port, with
> the
> client requesting which one it wants.
>
> (2) Use the 'nbdinfo' tool to examine what is being served by an NBD
> server. It can query all sorts of information as described in the
> manual:
> https://libguestfs.org/nbdinfo.1.html
>
> Rich.
>
>
> == rpi-C3 (192.168.1.14) server setup
>
> sudo apt-get update
> sudo apt-get install nbd-server
> sudo nano /etc/nbd-server/config
>
> [generic]
> # If you want to run everything as root rather than the nbd
> user, you
> # may either say "root" in the two following lines, or remove
> them
> # altogether. Do not remove the [generic] section, however.
> user = nbd
> group = nbd
> includedir = /etc/nbd-server/conf.d
> allowlist = true
>
> # What follows are export definitions. You may create as much of
> them
> as
> # you want, but the section header has to be unique.
> [images]
> exportname = /home/pi/media/500G_FAITH/images
> port = 510029
> [recipes]
> exportname = /home/pi/media/500G_FAITH/recipes
> port = 510030
> [movie]
> exportname = /home/pi/media/500G_FAITH/movies
> port = 510031
>
> sudo modprobe nbd # NOT nbd-server!
> lsmod | grep nbd
>
> nbd 40960 0
>
> sudo service nbd-server force-reload # if already running
>
>
> == client side
>
> sudo apt-get install nbd-client # done
> sudo modprobe nbd
> nbd-client 192.168.1.14 -l
>
> Negotiation: ..
> images
> recipes
> movie
>
> sudo nbd-client 192.168.1.14 -N recipes /dev/nbd2
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd2
>
> sudo nbd-client 192.168.1.14 -N movie /dev/nbd3
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd3
>
> sudo nbd-client 192.168.1.14 -N images /dev/nbd1
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd1
>
>
> dmesg | tail -n 10
>
> [ 1417.883364] Buffer I/O error on dev nbd1, logical block 0,
> async
> page read
> [ 1417.884614] block nbd1: Other side returned error (22)
> [ 1417.884652] blk_update_request: I/O error, dev nbd1, sector 0
> op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 1417.884673] Buffer I/O error on dev nbd1, logical block 0,
> async
> page read
> [ 1417.885977] block nbd1: Other side returned error (22)
> [ 1417.886016] blk_update_request: I/O error, dev nbd1, sector 0
> op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 1417.886036] Buffer I/O error on dev nbd1, logical block 0,
> async
> page read
> [ 1417.887276] block nbd1: Other side returned error (22)
> [ 1417.888492] block nbd1: Other side returned error (22)
> [ 1417.889751] block nbd1: Other side returned error (22)
>
> sudo nbd-client 192.168.1.14 -d /dev/nbd1 # thought it might
> be
> something to
>
> do w/ nbd1
>
> sudo nbd-client 192.168.1.14 -N images /dev/nbd4
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd4
>
> dmesg | tail -n 10
>
> [ 1848.831897] block nbd4: Other side returned error (22)
> [ 1848.833341] block nbd4: Other side returned error (22)
> [ 1848.834651] block nbd4: Other side returned error (22)
> [ 1848.836002] block nbd4: Other side returned error (22)
>
> lsblk -f
>
> NAME FSTYPE LABEL UUID
> FSAVAIL
> FSUSE% MOUNTPOINT
> sda
>
>
> ├─sda1 vfat EFI 67E3-17ED
>
>
> └─sda2 hfsplus time_box_2018
> f6427a3d-363b-3284-88c4-03ce3493aeff
> 115.5G
> 94% /media/pi/time_box_2018
> nbd2
>
>
> nbd3
>
>
> nbd4
>
>
> mmcblk0
>
>
> ├─mmcblk0p1 vfat boot 5DE4-665C
> 203.4M
> 19% /boot
> └─mmcblk0p2 ext4 rootfs
> 7295bbc3-bbc2-4267-9fa0-099e10ef5bf0
> 4.1G
> 37% /
>
> dmesg | tail -n 10
>
> [ 1848.836002] block nbd4: Other side returned error (22)
> [ 1848.837430] block nbd4: Other side returned error (22)
> [ 1848.838860] block nbd4: Other side returned error (22)
> [ 1848.840224] block nbd4: Other side returned error (22)
> [ 1848.841691] block nbd4: Other side returned error (22)
> [ 1848.843077] block nbd4: Other side returned error (22)
> [ 1848.844415] block nbd4: Other side returned error (22)
> [ 2802.615037] block nbd3: Receive control failed (result -32)
> [ 2802.625871] block nbd2: Receive control failed (result -32)
> [ 2802.648444] block nbd4: Receive control failed (result -32)
>
> ls /home/pi/Vols/
>
> nbd1-recipes/ rpi-C2-nbd-500G/ tbx2018/ time_box_2018
> /
>
> sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
>
> mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev
> /nbd2.
>
> sudo parted -l
>
> Model: Seagate BUP Slim SL (scsi)
> Disk /dev/sda: 2000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Number Start End Size File system Name
> Flags
> 1 20.5kB 210MB 210MB fat32 EFI System Partition
> boot,
> esp
> 2 210MB 2000GB 2000GB hfs+ time_box_2018
>
>
> Error: /dev/nbd3: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd3: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Error: /dev/nbd4: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd4: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Error: /dev/nbd2: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd2: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Model: SD SL08G (sd/mmc)
> Disk /dev/mmcblk0: 7948MB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
>
> Number Start End Size Type File system Flags
> 1 4194kB 273MB 268MB primary fat32 lba
> 2 273MB 7948MB 7676MB primary ext4
>
> I’ve been trying to get them to talk for 2 days now
> One thing I did notice is that the port numbers in the config
> file
> don’t seem
> to change the fact it’s listening on he default port??
>
> $ sudo lsof -i -P -n | grep LISTEN
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> cnid_meta 312 root 3u IPv6 11170 0t0 TCP
> [::1]:4700
> (LISTEN)
> afpd 324 root 3u IPv4 11195 0t0 TCP *:548
> (LISTEN)
> sshd 426 root 3u IPv4 12126 0t0 TCP *:22
> (LISTEN)
> sshd 426 root 4u IPv6 12128 0t0 TCP *:22
> (LISTEN)
> dnsmasq 431 dnsmasq 5u IPv4 12046 0t0 TCP *:53
> (LISTEN)
> dnsmasq 431 dnsmasq 7u IPv6 12048 0t0 TCP *:53
> (LISTEN)
> nbd-serve 2026 nbd 3u IPv6 20100 0t0 TCP *:10809
> (LISTEN)
>
> I’d be very grateful for any pointers, or other resources that
> my be of
> help.
> Thanks in advance,
> Simon
>
>
>
>
>
>
> --
> Richard Jones, Virtualization Group, Red Hat http://
> people.redhat.com/
> ~rjones
> Read my programming and virtualization blog: http://
> rwmj.wordpress.com
> virt-top is 'top' for virtual machines. Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>
>
>
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/
> ~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines. Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>
>
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
Reply to: