Re: Setting up NBD server / client - block nbd4: Other side returned error (22) issue
On Thu, Aug 26, 2021 at 08:48:35PM +0100, Richard W.M. Jones wrote:
> On Thu, Aug 26, 2021 at 07:49:33PM +0100, Simon Fernandez wrote:
> > Hi folks,
> > thanks for the reply Wouter's reply was in junk only found it when I saw
> > Richards reply and fished it out.
> >
> > I'm still getting
> > $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd0
> >
> > Here are the changes I've made - first I built nbdinfo - thanks for the tip
> > Richard.
> >
> > Build GnuTLS from
> > https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.16.tar.xz
> >
> > Build libnbd from
> > https://download.libguestfs.org/libnbd/1.8-stable/libnbd-1.8.3.tar.gz
> >
> > After installing quite a lot of libraries . . .
> > ----------------------------------------------------------------------
> > Thank you for downloading libnbd 1.8.3
> >
> > This is how we have configured the optional components for you today:
> >
> > Optional library features:
> >
> > TLS support ............................ yes
> > NBD URI support ........................ yes
> > FUSE support ........................... yes
> > Manual pages ........................... yes
> > Bash tab completion .................... yes
> >
> > Language bindings:
> >
> > Go ..................................... no
> > OCaml .................................. no
> > Python ................................. no
> >
> > nbdkit version note - during ./configure
> > checking for nbdkit... nbdkit
> > checking for nbdkit >= 1.12... no (1.10)
> > configure: WARNING: nbdkit is too old, some tests will be disabled
> >
> > The 'make check' skipped a lot of tests but there were no FAILS
> >
> > $ nbdinfo --version
> > nbdinfo 1.8.3
> > libnbd 1.8.3
> >
> >
> > == rpi-C3 (192.168.1.14) server side
> >
> > Q: Wouter ( w@uter.be ) - Are they readable and writable by the "nbd" user?
> > R: I've changed the user & group to pi (the logged in user) so should be R/W
> >
> > Note: Richard ( rjones@redhat.com ) - You don't need to serve each export on a
> > separate port. The NBD
> > protocol can serve multiple exports (disks) on the same port, with the
> > client requesting which one it wants.
> > R: Have remove the port addresses from config
> >
> > pi@rpi-C3:~ $ cat /etc/nbd-server/config
> > [generic]
> > # If you want to run everything as root rather than the nbd user, you
> > # may either say "root" in the two following lines, or remove them
> > # altogether. Do not remove the [generic] section, however.
> > user = pi
> > group = pi
> > includedir = /etc/nbd-server/conf.d
> > allowlist = true
> >
> > # What follows are export definitions. You may create as much of them as
> > # you want, but the section header has to be unique.
> > [images]
> > exportname = /home/pi/media/500G_FAITH/images/
> > transactionlog = /home/pi/media/500G_FAITH/logs/log_images
> > [recipes]
> > exportname = /home/pi/media/500G_FAITH/recipes/
> > transactionlog = /home/pi/media/500G_FAITH/logs/log_recipes
> > [movie]
> > exportname = /home/pi/media/500G_FAITH/movies/
> > transactionlog = /home/pi/media/500G_FAITH/logs/log_movie
>
> One problem here is that NBD is a block device export protocol, not a
> file serving protocol (like eg NFS or SMB).
>
> You can however turn a directory into an NBD export using
> nbdkit-floppy-plugin, eg:
>
> $ nbdkit floppy /home/pi/media/500G_FAITH/recipes/
>
> (https://libguestfs.org/nbdkit-floppy-plugin.1.html)
>
> That exports a VFAT filesystem as a block device which has to be
> mounted:
>
> # nbd-client 192.168.1.14 10809 /dev/nbd0
> # mkdir /mnt/recipes
> # mount /dev/nbd0 /mnt/recipes
Actually I think this would be /dev/nbd0p1 because nbdkit-floppy-plugin
makes a partitioned virtual disk. You don't usually need to run
kpartx because the kernel will sniff partitions on the block device
when it is instantiated.
Rich.
> > Q: Wouter ( w@uter.be ) - Do the paths that you point to in the exportname
> > parameters exist?
> > R: YES its a directory with a bunch of images in it. YES - Non zero size.
> >
> > pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/recipes/
> > y000_Pork_Belly_Wontan____TOCOST.rtf y011_Rack_And_Black.rtf
> > y002_Salmon_Parfait_Forest_Rosemary_Bread_LE.rtf
> > y021_StoneAge_Beef_Taragon_Sauce____TOCOST.rtf
> >
> > pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/images
> > 160g? of dough - 250C - 6m.jpg 20190228_163410_monkfish and red pepper
> > skewers.jpg
> > 20190629_202814_couscous w apricots.jpg 672_V_cheddar_gorgeous.jpg
> > 20181022_134259.jpg 20200124_152329_aubergine & pesto stuffing.jpg
> > 25_androll_into_pancakes.jpg 665 Celeriac & Cockscombe Mint Soup.jpg
> >
> > pi@rpi-C3:~ $ ls /home/pi/media/500G_FAITH/movies
> > NA #015 - Bread.mp4
> > NA #016 - Fisetin - Mayo Clinic Trials.mp4
> > NA #042 - Delta Variant COVID.mp4
> > NA #044 - Lex Fridman.mp4
> >
> > pi@rpi-C3:~ $ ls -la /home/pi/media/500G_FAITH/logs
> > -rw------- 1 pi pi 56 Aug 26 17:00 log_images
> > -rw------- 1 pi pi 56 Aug 26 17:00 log_movie
> > -rw------- 1 pi pi 700 Aug 26 17:35 log_recipes
> >
> > pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_images
> > %`?%`?
> >
> > pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_movie
> > %`?%`?
> >
> > pi@rpi-C3:~ $ cat /home/pi/media/500G_FAITH/logs/log_recipes
> > %`?`%`?a%`?b%`?c%`?d%`?e%`?f%`?g%`?h%`?i%`?j%`?k%`?l%`?%`?%`?%`?%`?%`?%`?%`?%`%
> > `? %`?
> >
> > pi@rpi-C3:~ $ nbdkit --version
> > nbdkit 1.1.12
> >
> >
> > == client side
> >
> > Note: Richard ( rjones@redhat.com ) - Use the 'nbdinfo' tool to examine what is
> > being served by an NBD
> > server. It can query all sorts of information as described in the manual:
> > R: Built the tool. - here is what it says : /
> >
> > pi@rpi-C1:~ $ nbdinfo nbd://192.168.1.14:10809/recipes
> > protocol: newstyle-fixed without TLS
> > export="recipes":
> > export-size: 4096
> > uri: nbd://192.168.1.14:10809/recipes
> > is_rotational: false
> > is_read_only: true
> > can_cache: false
> > can_df: false
> > can_fast_zero: false
> > can_flush: false
> > can_fua: false
> > can_multi_conn: true
> > can_trim: false
> > can_zero: true
>
> This looks like a good "recipes" export, although the size is
> unusually small (4K), which is probably because what you're exporting
> here is the directory inode, not the directory - see above.
>
> > pi@rpi-C1:~ $ nbdkit --version
> > nbdkit 1.10.3
> >
> > pi@rpi-C1:~ $ nbd-client -l 192.168.1.14
> > Negotiation: ..
> > images
> > recipes
> > movie
>
> nbdinfo can also list exports using the --list option which will tell
> you a lot of detail about every export.
>
> > pi@rpi-C1:~ $ sudo modprobe nbd
> > pi@rpi-C1:~ $ lsmod | grep nbd
> > nbd 49152 0
> >
> > pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N recipes /dev/nbd0
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd0
> >
> > pi@rpi-C1:~ $ sudo partx -a /dev/nbd0
> > partx: /dev/nbd0: failed to read partition table
> >
> > pi@rpi-C1:~ $ sudo parted -l
> > Error: /dev/nbd0: unrecognised disk label
> > Model: Unknown (unknown)
> > Disk /dev/nbd0: 4096B
> > Sector size (logical/physical): 512B/512B
> > Partition Table: unknown
> > Disk Flags:
> >
> > pi@rpi-C1:~ $ dmesg | tail -n 20
> > [ 2138.454532] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 2138.454547] Buffer I/O error on dev nbd0, logical block 0, async page read
> > [ 2138.457380] block nbd0: Other side returned error (22)
> > [ 2138.457497] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 2138.457520] Buffer I/O error on dev nbd0, logical block 0, async page read
> > [ 2138.459238] block nbd0: Other side returned error (22)
> > [ 2138.459280] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 2138.459297] Buffer I/O error on dev nbd0, logical block 0, async page read
> > [ 2138.460993] block nbd0: Other side returned error (22)
> > [ 2138.461023] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 2138.461038] Buffer I/O error on dev nbd0, logical block 0, async page read
> > [ 2138.462649] block nbd0: Other side returned error (22)
> > [ 2138.462685] Buffer I/O error on dev nbd0, logical block 0, async page read
> > [ 2138.464344] block nbd0: Other side returned error (22)
> > [ 2138.466926] block nbd0: Other side returned error (22)
> > [ 2138.468506] block nbd0: Other side returned error (22)
> > [ 2138.470066] block nbd0: Other side returned error (22)
> > [ 2138.471442] block nbd0: Other side returned error (22)
> > [ 2138.472978] block nbd0: Other side returned error (22)
> > [ 2138.474448] block nbd0: Other side returned error (22)
> >
> > pi@rpi-C1:~ $ sudo nbd-client 192.168.1.14 10809 -N movies /dev/nbd1
> > Negotiation: ..Error: E: server does not support NBD_OPT_GO and dropped
> > connection after sending NBD_OPT_EXPORT_NAME. Try -g.
> > Exiting.
> >
> > I'm attempting to serve non empty directories so I don't know why it's saying
> > 0Mb size?
> > Scratching my head here . . any thoughts? :/
>
> Rich.
>
> > Thanks in advance,
> > Simon
> >
> >
> > On 12 Aug 2021, at 22:32, Richard W.M. Jones <rjones@redhat.com> wrote:
> >
> > On Thu, Jul 22, 2021 at 01:41:00PM +0100, Simon Fernandez wrote:
> >
> > Hi folks,
> > I’m trying setup a client that connect to a 500Gb ext4 disk on another
> > machine
> > running nbd server.
> > Server: raspberry pi 3: uname -r 4.19.66+
> > Client: raspberry pi 3: uname -r 5.10.17-v7+
> >
> > I’ve followed the information here [https://github.com/
> > NetworkBlockDevice/nbd]
> > and a lot of other sites but I think I’m doing something wrong on the
> > server
> > setup because I get this on the client:
> >
> > $ sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
> > mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev/nbd2
> >
> > Also it appears to be listening on default port - NOT ones specified in
> > config
> > file.
> >
> > This is how I’ve set up the server & client, it’s probably something
> > obvious to
> > more experienced folk.
> >
> >
> > Wouter has answered this already, I just wanted to add a couple of
> > smaller points:
> >
> > (1) You don't need to serve each export on a separate port. The NBD
> > protocol can serve multiple exports (disks) on the same port, with the
> > client requesting which one it wants.
> >
> > (2) Use the 'nbdinfo' tool to examine what is being served by an NBD
> > server. It can query all sorts of information as described in the
> > manual:
> > https://libguestfs.org/nbdinfo.1.html
> >
> > Rich.
> >
> >
> > == rpi-C3 (192.168.1.14) server setup
> >
> > sudo apt-get update
> > sudo apt-get install nbd-server
> > sudo nano /etc/nbd-server/config
> >
> > [generic]
> > # If you want to run everything as root rather than the nbd user, you
> > # may either say "root" in the two following lines, or remove them
> > # altogether. Do not remove the [generic] section, however.
> > user = nbd
> > group = nbd
> > includedir = /etc/nbd-server/conf.d
> > allowlist = true
> >
> > # What follows are export definitions. You may create as much of them
> > as
> > # you want, but the section header has to be unique.
> > [images]
> > exportname = /home/pi/media/500G_FAITH/images
> > port = 510029
> > [recipes]
> > exportname = /home/pi/media/500G_FAITH/recipes
> > port = 510030
> > [movie]
> > exportname = /home/pi/media/500G_FAITH/movies
> > port = 510031
> >
> > sudo modprobe nbd # NOT nbd-server!
> > lsmod | grep nbd
> >
> > nbd 40960 0
> >
> > sudo service nbd-server force-reload # if already running
> >
> >
> > == client side
> >
> > sudo apt-get install nbd-client # done
> > sudo modprobe nbd
> > nbd-client 192.168.1.14 -l
> >
> > Negotiation: ..
> > images
> > recipes
> > movie
> >
> > sudo nbd-client 192.168.1.14 -N recipes /dev/nbd2
> >
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd2
> >
> > sudo nbd-client 192.168.1.14 -N movie /dev/nbd3
> >
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd3
> >
> > sudo nbd-client 192.168.1.14 -N images /dev/nbd1
> >
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd1
> >
> >
> > dmesg | tail -n 10
> >
> > [ 1417.883364] Buffer I/O error on dev nbd1, logical block 0, async
> > page read
> > [ 1417.884614] block nbd1: Other side returned error (22)
> > [ 1417.884652] blk_update_request: I/O error, dev nbd1, sector 0 op
> > 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 1417.884673] Buffer I/O error on dev nbd1, logical block 0, async
> > page read
> > [ 1417.885977] block nbd1: Other side returned error (22)
> > [ 1417.886016] blk_update_request: I/O error, dev nbd1, sector 0 op
> > 0x0:(READ)
> > flags 0x0 phys_seg 1 prio class 0
> > [ 1417.886036] Buffer I/O error on dev nbd1, logical block 0, async
> > page read
> > [ 1417.887276] block nbd1: Other side returned error (22)
> > [ 1417.888492] block nbd1: Other side returned error (22)
> > [ 1417.889751] block nbd1: Other side returned error (22)
> >
> > sudo nbd-client 192.168.1.14 -d /dev/nbd1 # thought it might be
> > something to
> >
> > do w/ nbd1
> >
> > sudo nbd-client 192.168.1.14 -N images /dev/nbd4
> >
> > Negotiation: ..size = 0MB
> > Connected /dev/nbd4
> >
> > dmesg | tail -n 10
> >
> > [ 1848.831897] block nbd4: Other side returned error (22)
> > [ 1848.833341] block nbd4: Other side returned error (22)
> > [ 1848.834651] block nbd4: Other side returned error (22)
> > [ 1848.836002] block nbd4: Other side returned error (22)
> >
> > lsblk -f
> >
> > NAME FSTYPE LABEL UUID
> > FSAVAIL
> > FSUSE% MOUNTPOINT
> > sda
> >
> >
> > ├─sda1 vfat EFI 67E3-17ED
> >
> >
> > └─sda2 hfsplus time_box_2018 f6427a3d-363b-3284-88c4-03ce3493aeff
> > 115.5G
> > 94% /media/pi/time_box_2018
> > nbd2
> >
> >
> > nbd3
> >
> >
> > nbd4
> >
> >
> > mmcblk0
> >
> >
> > ├─mmcblk0p1 vfat boot 5DE4-665C
> > 203.4M
> > 19% /boot
> > └─mmcblk0p2 ext4 rootfs 7295bbc3-bbc2-4267-9fa0-099e10ef5bf0
> > 4.1G
> > 37% /
> >
> > dmesg | tail -n 10
> >
> > [ 1848.836002] block nbd4: Other side returned error (22)
> > [ 1848.837430] block nbd4: Other side returned error (22)
> > [ 1848.838860] block nbd4: Other side returned error (22)
> > [ 1848.840224] block nbd4: Other side returned error (22)
> > [ 1848.841691] block nbd4: Other side returned error (22)
> > [ 1848.843077] block nbd4: Other side returned error (22)
> > [ 1848.844415] block nbd4: Other side returned error (22)
> > [ 2802.615037] block nbd3: Receive control failed (result -32)
> > [ 2802.625871] block nbd2: Receive control failed (result -32)
> > [ 2802.648444] block nbd4: Receive control failed (result -32)
> >
> > ls /home/pi/Vols/
> >
> > nbd1-recipes/ rpi-C2-nbd-500G/ tbx2018/ time_box_2018/
> >
> > sudo mount /dev/nbd2 /home/pi/Vols/nbd1-recipes/
> >
> > mount: /home/pi/Vols/nbd1-recipes: can't read superblock on /dev/nbd2.
> >
> > sudo parted -l
> >
> > Model: Seagate BUP Slim SL (scsi)
> > Disk /dev/sda: 2000GB
> > Sector size (logical/physical): 512B/512B
> > Partition Table: gpt
> > Disk Flags:
> >
> > Number Start End Size File system Name
> > Flags
> > 1 20.5kB 210MB 210MB fat32 EFI System Partition boot,
> > esp
> > 2 210MB 2000GB 2000GB hfs+ time_box_2018
> >
> >
> > Error: /dev/nbd3: unrecognised disk label
> > Model: Unknown (unknown)
> >
> > Disk /dev/nbd3: 4096B
> > Sector size (logical/physical): 512B/512B
> > Partition Table: unknown
> > Disk Flags:
> >
> > Error: /dev/nbd4: unrecognised disk label
> > Model: Unknown (unknown)
> >
> > Disk /dev/nbd4: 4096B
> > Sector size (logical/physical): 512B/512B
> > Partition Table: unknown
> > Disk Flags:
> >
> > Error: /dev/nbd2: unrecognised disk label
> > Model: Unknown (unknown)
> >
> > Disk /dev/nbd2: 4096B
> > Sector size (logical/physical): 512B/512B
> > Partition Table: unknown
> > Disk Flags:
> >
> > Model: SD SL08G (sd/mmc)
> > Disk /dev/mmcblk0: 7948MB
> > Sector size (logical/physical): 512B/512B
> > Partition Table: msdos
> > Disk Flags:
> >
> > Number Start End Size Type File system Flags
> > 1 4194kB 273MB 268MB primary fat32 lba
> > 2 273MB 7948MB 7676MB primary ext4
> >
> > I’ve been trying to get them to talk for 2 days now
> > One thing I did notice is that the port numbers in the config file
> > don’t seem
> > to change the fact it’s listening on he default port??
> >
> > $ sudo lsof -i -P -n | grep LISTEN
> > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> > cnid_meta 312 root 3u IPv6 11170 0t0 TCP [::1]:4700
> > (LISTEN)
> > afpd 324 root 3u IPv4 11195 0t0 TCP *:548 (LISTEN)
> > sshd 426 root 3u IPv4 12126 0t0 TCP *:22 (LISTEN)
> > sshd 426 root 4u IPv6 12128 0t0 TCP *:22 (LISTEN)
> > dnsmasq 431 dnsmasq 5u IPv4 12046 0t0 TCP *:53 (LISTEN)
> > dnsmasq 431 dnsmasq 7u IPv6 12048 0t0 TCP *:53 (LISTEN)
> > nbd-serve 2026 nbd 3u IPv6 20100 0t0 TCP *:10809
> > (LISTEN)
> >
> > I’d be very grateful for any pointers, or other resources that my be of
> > help.
> > Thanks in advance,
> > Simon
> >
> >
> >
> >
> >
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/
> > ~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-top is 'top' for virtual machines. Tiny program with many
> > powerful monitoring features, net stats, disk stats, logging, etc.
> > http://people.redhat.com/~rjones/virt-top
> >
> >
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines. Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
Reply to: