Re: Setting up NBD server / client - block nbd4: Other side returned error (22) issue
On Thu, Aug 26, 2021 at 07:49:33PM +0100, Simon Fernandez wrote:
> Hi folks,
> thanks for the reply Wouter's reply was in junk only found it when I saw
> Richards reply and fished it out.
>
> I'm still getting
> $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> Negotiation: ..size = 0MB
> Connected /dev/nbd0
>
> Here are the changes I've made - first I built nbdinfo - thanks for the tip
> Richard.
>
> Build GnuTLS from
> https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.16.tar.xz
>
> Build libnbd from
> https://download.libguestfs.org/libnbd/1.8-stable/libnbd-1.8.3.tar.gz
>
> After installing quite a lot of libraries . . .
> ----------------------------------------------------------------------
> Thank you for downloading libnbd 1.8.3
>
> This is how we have configured the optional components for you today:
>
> Optional library features:
>
> TLS support ............................ yes
> NBD URI support ........................ yes
> FUSE support ........................... yes
> Manual pages ........................... yes
> Bash tab completion .................... yes
>
> Language bindings:
>
> Go ..................................... no
> OCaml .................................. no
> Python ................................. no
>
> nbdkit version note - during ./configure
> checking for nbdkit... nbdkit
> checking for nbdkit >= 1.12... no (1.10)
> configure: WARNING: nbdkit is too old, some tests will be disabled
>
> The 'make check' skipped a lot of tests but there were no FAILS
>
> $ nbdinfo --version
> nbdinfo 1.8.3
> libnbd 1.8.3
>
>
> == rpi-C3 (192.168.1.14) server side
>
> Q: Wouter ( w@uter.be ) - Are they readable and writable by the "nbd" user?
> R: I've changed the user & group to pi (the logged in user) so should be R/W
>
> Note: Richard ( rjones@redhat.com ) - You don't need to serve each export on a
> separate port. The NBD
> protocol can serve multiple exports (disks) on the same port, with the
> client requesting which one it wants.
> R: Have remove the port addresses from config
>
> pi@rpi-C3:~ $ cat /etc/nbd-server/config
> [generic]
> # If you want to run everything as root rather than the nbd user, you
> # may either say "root" in the two following lines, or remove them
> # altogether. Do not remove the [generic] section, however.
> user = pi
> group = pi
> includedir = /etc/nbd-server/conf.d
> allowlist = true
>
> # What follows are export definitions. You may create as much of them as
> # you want, but the section header has to be unique.
> [images]
> exportname = /home/pi/media/500G_FAITH/images/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_images
> [recipes]
> exportname = /home/pi/media/500G_FAITH/recipes/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_recipes
> [movie]
> exportname = /home/pi/media/500G_FAITH/movies/
> transactionlog = /home/pi/media/500G_FAITH/logs/log_movie
One problem here is that NBD is a block device export protocol, not a
file serving protocol (like eg NFS or SMB).
You can however turn a directory into an NBD export using
nbdkit-floppy-plugin, eg:
$ nbdkit floppy /home/pi/media/500G_FAITH/recipes/
(https://libguestfs.org/nbdkit-floppy-plugin.1.html)
That exports a VFAT filesystem as a block device which has to be
mounted:
# nbd-client 192.168.1.14 10809 /dev/nbd0
# mkdir /mnt/recipes
# mount /dev/nbd0 /mnt/recipes
> Q: Wouter ( w@uter.be ) - Do the paths that you point to in the exportname
> parameters exist?
> R: YES its a directory with a bunch of images in it. YES - Non zero size.
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/recipes/
> y000_Pork_Belly_Wontan____TOCOST.rtf y011_Rack_And_Black.rtf
> y002_Salmon_Parfait_Forest_Rosemary_Bread_LE.rtf
> y021_StoneAge_Beef_Taragon_Sauce____TOCOST.rtf
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/images
> 160g? of dough - 250C - 6m.jpg 20190228_163410_monkfish and red pepper
> skewers.jpg
> 20190629_202814_couscous w apricots.jpg 672_V_cheddar_gorgeous.jpg
> 20181022_134259.jpg 20200124_152329_aubergine & pesto stuffing.jpg
> 25_androll_into_pancakes.jpg 665 Celeriac & Cockscombe Mint Soup.jpg
>
> pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/movies
> NA #015 - Bread.mp4
> NA #016 - Fisetin - Mayo Clinic Trials.mp4
> NA #042 - Delta Variant COVID.mp4
> NA #044 - Lex Fridman.mp4
>
> pi@rpi-C3:~ $ ls -la /home/pi/media/500G_FAITH/logs
> -rw------- 1 pi pi 56 Aug 26 17:00 log_images
> -rw------- 1 pi pi 56 Aug 26 17:00 log_movie
> -rw------- 1 pi pi 700 Aug 26 17:35 log_recipes
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_images
> %`?%`?
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_movie
> %`?%`?
>
> pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_recipes
> %`?`%`?a%`?b%`?c%`?d%`?e%`?f%`?g%`?h%`?i%`?j%`?k%`?l%`?%`?%`?%`?%`?%`?%`?%`?%`%
> `? %`?
>
> pi@rpi-C3:~ $ nbdkit --version
> nbdkit 1.1.12
>
>
> == client side
>
> Note: Richard ( rjones@redhat.com ) - Use the 'nbdinfo' tool to examine what is
> being served by an NBD
> server. It can query all sorts of information as described in the manual:
> R: Built the tool. - here is what it says : /
>
> pi@rpi-C1:~ $ nbdinfo nbd://192.168.1.14:10809/recipes
> protocol: newstyle-fixed without TLS
> export="recipes":
> export-size: 4096
> uri: nbd://192.168.1.14:10809/recipes
> is_rotational: false
> is_read_only: true
> can_cache: false
> can_df: false
> can_fast_zero: false
> can_flush: false
> can_fua: false
> can_multi_conn: true
> can_trim: false
> can_zero: true
This looks like a good "recipes" export, although the size is
unusually small (4K), which is probably because what you're exporting
here is the directory inode, not the directory - see above.
> pi@rpi-C1:~ $ nbdkit --version
> nbdkit 1.10.3
>
> pi@rpi-C1:~ $ nbd-client -l 192.168.1.14
> Negotiation: ..
> images
> recipes
> movie
nbdinfo can also list exports using the --list option which will tell
you a lot of detail about every export.
> pi@rpi-C1:~ $ sudo modprobe nbd
> pi@rpi-C1:~ $ lsmod | grep nbd
> nbd 49152 0
>
> pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> Negotiation: ..size = 0MB
> Connected /dev/nbd0
>
> pi@rpi-C1:~ $ sudo partx -a /dev/nbd0
> partx: /dev/nbd0: failed to read partition table
>
> pi@rpi-C1:~ $ sudo parted -l
> Error: /dev/nbd0: unrecognised disk label
> Model: Unknown (unknown)
> Disk /dev/nbd0: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> pi@rpi-C1:~ $ dmesg | tail -n 20
> [ 2138.454532] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.454547] Buffer I/O error on dev nbd0, logical block 0, async page read
> [ 2138.457380] block nbd0: Other side returned error (22)
> [ 2138.457497] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.457520] Buffer I/O error on dev nbd0, logical block 0, async page read
> [ 2138.459238] block nbd0: Other side returned error (22)
> [ 2138.459280] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.459297] Buffer I/O error on dev nbd0, logical block 0, async page read
> [ 2138.460993] block nbd0: Other side returned error (22)
> [ 2138.461023] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 2138.461038] Buffer I/O error on dev nbd0, logical block 0, async page read
> [ 2138.462649] block nbd0: Other side returned error (22)
> [ 2138.462685] Buffer I/O error on dev nbd0, logical block 0, async page read
> [ 2138.464344] block nbd0: Other side returned error (22)
> [ 2138.466926] block nbd0: Other side returned error (22)
> [ 2138.468506] block nbd0: Other side returned error (22)
> [ 2138.470066] block nbd0: Other side returned error (22)
> [ 2138.471442] block nbd0: Other side returned error (22)
> [ 2138.472978] block nbd0: Other side returned error (22)
> [ 2138.474448] block nbd0: Other side returned error (22)
>
> pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N movies /dev/nbd1
> Negotiation: ..Error: E: server does not support NBD_OPT_GO and dropped
> connection after sending NBD_OPT_EXPORT_NAME. Try -g.
> Exiting.
>
> I'm attempting to serve non empty directories so I don't know why it's saying
> 0Mb size?
> Scratching my head here . . any thoughts? :/
Rich.
> Thanks in advance,
> Simon
>
>
> On 12 Aug 2021, at 22:32, Richard W.M. Jones <rjones@redhat.com> wrote:
>
> On Thu, Jul 22, 2021 at 01:41:00PM +0100, Simon Fernandez wrote:
>
> Hi folks,
> I’m trying setup a client that connect to a 500Gb ext4 disk on another
> machine
> running nbd server.
> Server: raspberry pi 3: uname -r 4.19.66+
> Client: raspberry pi 3: uname -r 5.10.17-v7+
>
> I’ve followed the information here [https://github.com/
> NetworkBlockDevice/nbd]
> and a lot of other sites but I think I’m doing something wrong on the
> server
> setup because I get this on the client:
>
> $ sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
> mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev/nbd2
>
> Also it appears to be listening on default port - NOT ones specified in
> config
> file.
>
> This is how I’ve set up the server & client, it’s probably something
> obvious to
> more experienced folk.
>
>
> Wouter has answered this already, I just wanted to add a couple of
> smaller points:
>
> (1) You don't need to serve each export on a separate port. The NBD
> protocol can serve multiple exports (disks) on the same port, with the
> client requesting which one it wants.
>
> (2) Use the 'nbdinfo' tool to examine what is being served by an NBD
> server. It can query all sorts of information as described in the
> manual:
> https://libguestfs.org/nbdinfo.1.html
>
> Rich.
>
>
> == rpi-C3 (192.168.1.14) server setup
>
> sudo apt-get update
> sudo apt-get install nbd-server
> sudo nano /etc/nbd-server/config
>
> [generic]
> # If you want to run everything as root rather than the nbd user, you
> # may either say "root" in the two following lines, or remove them
> # altogether. Do not remove the [generic] section, however.
> user = nbd
> group = nbd
> includedir = /etc/nbd-server/conf.d
> allowlist = true
>
> # What follows are export definitions. You may create as much of them
> as
> # you want, but the section header has to be unique.
> [images]
> exportname = /home/pi/media/500G_FAITH/images
> port = 510029
> [recipes]
> exportname = /home/pi/media/500G_FAITH/recipes
> port = 510030
> [movie]
> exportname = /home/pi/media/500G_FAITH/movies
> port = 510031
>
> sudo modprobe nbd # NOT nbd-server!
> lsmod | grep nbd
>
> nbd 40960 0
>
> sudo service nbd-server force-reload # if already running
>
>
> == client side
>
> sudo apt-get install nbd-client # done
> sudo modprobe nbd
> nbd-client 192.168.1.14 -l
>
> Negotiation: ..
> images
> recipes
> movie
>
> sudo nbd-client 192.168.1.14 -N recipes /dev/nbd2
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd2
>
> sudo nbd-client 192.168.1.14 -N movie /dev/nbd3
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd3
>
> sudo nbd-client 192.168.1.14 -N images /dev/nbd1
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd1
>
>
> dmesg | tail -n 10
>
> [ 1417.883364] Buffer I/O error on dev nbd1, logical block 0, async
> page read
> [ 1417.884614] block nbd1: Other side returned error (22)
> [ 1417.884652] blk_update_request: I/O error, dev nbd1, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 1417.884673] Buffer I/O error on dev nbd1, logical block 0, async
> page read
> [ 1417.885977] block nbd1: Other side returned error (22)
> [ 1417.886016] blk_update_request: I/O error, dev nbd1, sector 0 op
> 0x0:(READ)
> flags 0x0 phys_seg 1 prio class 0
> [ 1417.886036] Buffer I/O error on dev nbd1, logical block 0, async
> page read
> [ 1417.887276] block nbd1: Other side returned error (22)
> [ 1417.888492] block nbd1: Other side returned error (22)
> [ 1417.889751] block nbd1: Other side returned error (22)
>
> sudo nbd-client 192.168.1.14 -d /dev/nbd1 # thought it might be
> something to
>
> do w/ nbd1
>
> sudo nbd-client 192.168.1.14 -N images /dev/nbd4
>
> Negotiation: ..size = 0MB
> Connected /dev/nbd4
>
> dmesg | tail -n 10
>
> [ 1848.831897] block nbd4: Other side returned error (22)
> [ 1848.833341] block nbd4: Other side returned error (22)
> [ 1848.834651] block nbd4: Other side returned error (22)
> [ 1848.836002] block nbd4: Other side returned error (22)
>
> lsblk -f
>
> NAME FSTYPE LABEL UUID
> FSAVAIL
> FSUSE% MOUNTPOINT
> sda
>
>
> ├─sda1 vfat EFI 67E3-17ED
>
>
> └─sda2 hfsplus time_box_2018 f6427a3d-363b-3284-88c4-03ce3493aeff
> 115.5G
> 94% /media/pi/time_box_2018
> nbd2
>
>
> nbd3
>
>
> nbd4
>
>
> mmcblk0
>
>
> ├─mmcblk0p1 vfat boot 5DE4-665C
> 203.4M
> 19% /boot
> └─mmcblk0p2 ext4 rootfs 7295bbc3-bbc2-4267-9fa0-099e10ef5bf0
> 4.1G
> 37% /
>
> dmesg | tail -n 10
>
> [ 1848.836002] block nbd4: Other side returned error (22)
> [ 1848.837430] block nbd4: Other side returned error (22)
> [ 1848.838860] block nbd4: Other side returned error (22)
> [ 1848.840224] block nbd4: Other side returned error (22)
> [ 1848.841691] block nbd4: Other side returned error (22)
> [ 1848.843077] block nbd4: Other side returned error (22)
> [ 1848.844415] block nbd4: Other side returned error (22)
> [ 2802.615037] block nbd3: Receive control failed (result -32)
> [ 2802.625871] block nbd2: Receive control failed (result -32)
> [ 2802.648444] block nbd4: Receive control failed (result -32)
>
> ls /home/pi/Vols/
>
> nbd1-recipes/ rpi-C2-nbd-500G/ tbx2018/ time_box_2018/
>
> sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
>
> mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev/nbd2.
>
> sudo parted -l
>
> Model: Seagate BUP Slim SL (scsi)
> Disk /dev/sda: 2000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Number Start End Size File system Name
> Flags
> 1 20.5kB 210MB 210MB fat32 EFI System Partition boot,
> esp
> 2 210MB 2000GB 2000GB hfs+ time_box_2018
>
>
> Error: /dev/nbd3: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd3: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Error: /dev/nbd4: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd4: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Error: /dev/nbd2: unrecognised disk label
> Model: Unknown (unknown)
>
> Disk /dev/nbd2: 4096B
> Sector size (logical/physical): 512B/512B
> Partition Table: unknown
> Disk Flags:
>
> Model: SD SL08G (sd/mmc)
> Disk /dev/mmcblk0: 7948MB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
>
> Number Start End Size Type File system Flags
> 1 4194kB 273MB 268MB primary fat32 lba
> 2 273MB 7948MB 7676MB primary ext4
>
> I’ve been trying to get them to talk for 2 days now
> One thing I did notice is that the port numbers in the config file
> don’t seem
> to change the fact it’s listening on he default port??
>
> $ sudo lsof -i -P -n | grep LISTEN
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> cnid_meta 312 root 3u IPv6 11170 0t0 TCP [::1]:4700
> (LISTEN)
> afpd 324 root 3u IPv4 11195 0t0 TCP *:548 (LISTEN)
> sshd 426 root 3u IPv4 12126 0t0 TCP *:22 (LISTEN)
> sshd 426 root 4u IPv6 12128 0t0 TCP *:22 (LISTEN)
> dnsmasq 431 dnsmasq 5u IPv4 12046 0t0 TCP *:53 (LISTEN)
> dnsmasq 431 dnsmasq 7u IPv6 12048 0t0 TCP *:53 (LISTEN)
> nbd-serve 2026 nbd 3u IPv6 20100 0t0 TCP *:10809
> (LISTEN)
>
> I’d be very grateful for any pointers, or other resources that my be of
> help.
> Thanks in advance,
> Simon
>
>
>
>
>
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/
> ~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines. Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>
>
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
Reply to: