Skip to contentSkip to conversation

libxl: blktap/tapback support

PATCH In review Jason Andryuk 4 participants 8 messages Apr 7, 2024 – May 27, 2024

lore
Share & tools
lore
#0 lore raw

v2 cover libxl: blktap/tapback support

From: Jason Andryuk <[email protected]>

This patch set re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.

blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.

It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.

The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.

This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path. (It works with the kernel blktap driver as well - upstream blktap hasn't removed the blktap driver requirement yet -
https://github.com/xapi-project/blktap/pull/364)

An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'

VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypted VHDs which is not available in QEMU.

blktap's tapback needs minimal changes to work with libxl:
https://github.com/xapi-project/blktap/pull/394

With a stubdom, the block-tap isn't executed a second time meaning tap-ctl destroy isn't called. The stubdom's disk xenstore frontend entry is removed, but tapback does nothing for this case right now. Making FE removal transition the BE to closed lets the hotplug script run properly.

v2 addresses review comments and adds a CHANGELOG entry.

Jason Andryuk (3):
hotplug: Update block-tap
libxl: Support blktap with HVM device model
CHANGELOG: Mention libxl blktap/tapback support

 CHANGELOG.md                  |   1 +
 tools/hotplug/Linux/block-tap | 169 +++++++++++++++++++++++++++++++---
 tools/libs/light/libxl_disk.c |  18 +++-
 tools/libs/light/libxl_dm.c   |   1 -
 4 files changed, 172 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v2 1/3 hotplug: Update block-tap

Implement a sharing check like the regular block script.

Checking tapback inside block-tap is too late since it needs to be running to transition the backend to InitWait before block-tap is run.

tap-ctl check will be removed when the requirement for the blktap kernel driver is removed. Remove it now as it is of limited use.

find_device() needs to be non-fatal allow a sharing check.

Only write physical-device-path because that is all that tapback needs. Also write_dev doesn't handled files and would incorrectly store physical-device as 0:0 which would confuse the minor inside tapback

---
v2:

Update usage string
Use local more
Don't use local for pid/minor
Use tap-ctl destroy
Use 4 space indent in count_using and check_tap_sharing

---
 tools/hotplug/Linux/block-tap | 169 +++++++++++++++++++++++++++++++---
 1 file changed, 157 insertions(+), 12 deletions(-)
tools/hotplug/Linux/block-tap +157 -12
index 89247921b9..126e472786 100755
@@ -18,11 +18,11 @@
 #
 # Usage:
 #
-# Target should be specified using the following syntax:
+# Disks should be specified using the following syntax:
 #
-# script=block-tap,vdev=xvda,target=<type>:<file>
+# vdev=xvda,backendtype=tap,format=vhd,target=/srv/target.vhd
 #
-# Type is either "aio" (for raw files), or "vhd"
+# format is either "aio" (for raw files), or "vhd"
 
 dir=$(dirname "$0")
 . "$dir/block-common.sh"
@@ -37,10 +37,6 @@ check_tools()
     if ! command -v tap-ctl > /dev/null 2>&1; then
         fatal "Unable to find tap-ctl tool"
     fi
-    modprobe blktap
-    if ! tap-ctl check >& /dev/null ; then
-	fatal "Blocktap kernel module not available"
-    fi
 }
 
 # Sets the following global variables based on the params field passed in as
@@ -81,7 +77,109 @@ find_device()
     done
 
     if [ -z "$pid" ] || [ -z "$minor" ]; then
-        fatal "cannot find required parameters"
+        return 1
+    fi
+
+    return 0
+}
+
+count_using()
+{
+    local file="$1"
+    local dom
+    local dev
+    local f
+
+    local i=0
+    local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+    for dom in $(xenstore-list "$base_path")
+    do
+        for dev in $(xenstore-list "$base_path/$dom")
+        do
+            f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
+            f=$(echo "$f" | cut -d ":" -f 2)
+
+            if [ -n "$f" ] && [ "$file" = $f ] ; then
+                i=$(( i + 1 ))
+            fi
+        done
+    done
+
+    echo "$i"
+}
+
+# tap_shared is used to determine if a shared tap can be closed
+# Since a stubdom and a guest both use the same tap, it can only
+# be freed when there is a single one left.
+tap_shared() {
+    [ $( count_using "$file" ) -gt 1 ]
+}
+
+check_tap_sharing()
+{
+    local file="$1"
+    local mode="$2"
+    local dom
+    local dev
+
+    local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+    for dom in $(xenstore-list "$base_path") ; do
+        for dev in $(xenstore-list "$base_path/$dom") ; do
+            local f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
+            f=$(echo "$f" | cut -d ":" -f 2)
+
+            if [ -n "$f" ] && [ "$file" = "$f" ] ; then
+                if [ "$mode" = 'w' ] ; then
+                    if ! same_vm $dom ; then
+                        echo "guest $f"
+                        return
+                    fi
+                else
+                    local m=$(xenstore_read_default "$base_path/$dom/$dev/mode"
+                                                    "")
+                    m=$(canonicalise_mode "$m")
+
+                    if [ "$m" = 'w' ] ; then
+                        if ! same_vm $dom ; then
+                            echo "guest $f"
+                            return
+                        fi
+                    fi
+                fi
+            fi
+        done
+    done
+
+    echo 'ok'
+}
+
+tap_create()
+{
+    if ! minor=$( tap-ctl allocate ) ; then
+        fatal "Could not allocate minor"
+    fi
+
+    # Handle with or without kernel blktap
+    minor=${minor#/run/blktap-control/tapdisk/tapdisk-}
+    minor=${minor#/dev/xen/blktap-2/tapdev}
+
+    # tap-ctl is spawning tapdisk which would hold the _lockfd open.
+    # Ensure it is closed before running tap-ctl spawn, which needs to be
+    # done in a subshell to continue holding the lock in the parent.
+    if ! pid=$( ( eval "exec $_lockfd>&-" ; tap-ctl spawn ) ) ; then
+        tap-ctl free -m "$minor"
+        fatal "Could not spawn tapdisk for $minor"
+    fi
+
+    if ! tap-ctl attach -p "$pid" -m "$minor" ; then
+        tap-ctl free -m "$minor"
+        fatal "Could not attach $pid and $minor"
+    fi
+
+    if ! tap-ctl open -p "$pid" -m "$minor" -a "$target" ; then
+        tap-ctl detach -p "$pid" -m "$minor"
+        tap-ctl free -m "$minor"
+        fatal "Could not open \"$target\""
     fi
 }
 
@@ -89,15 +187,54 @@ find_device()
 # the device
 add()
 {
-    dev=$(tap-ctl create -a $target)
-    write_dev $dev
+    local result
+
+    claim_lock "block"
+
+    if find_device; then
+        result=$( check_tap_sharing "$file" "$mode" )
+        if [ "$result" != "ok" ] ; then
+            do_ebusy "tap $type file $file in use " "$mode" "${result%% *}"
+        fi
+    else
+        tap_create
+    fi
+
+    # Create nbd unix path.  find_device/tap_create set pid & minor
+    dev=$( printf "/run/blktap-control/nbd%ld.%d" "$pid" "$minor" )
+
+    xenstore_write "$XENBUS_PATH/pid" "$pid"
+    xenstore_write "$XENBUS_PATH/minor" "$minor"
+    # dev, as a unix socket, would end up with major:minor 0:0 in
+    # physical-device if write_dev were used.  tapback would be thrown off by
+    # that incorrect minor, so just skip writing physical-device.
+    xenstore_write "$XENBUS_PATH/physical-device-path" "$dev"
+
+    success
+
+    release_lock "block"
 }
 
 # Disconnects the device
 remove()
 {
-    find_device
-    do_or_die tap-ctl destroy -p ${pid} -m ${minor} > /dev/null
+    local minor
+    local pid
+
+    claim_lock "block"
+
+    if tap_shared ; then
+        return
+    fi
+
+    minor=$( xenstore_read "$XENBUS_PATH/minor" )
+    pid=$( xenstore_read "$XENBUS_PATH/pid" )
+
+    [ -n "$minor" ] || fatal "minor missing"
+    [ -n "$pid" ] || fatal "pid missing"
+    do_or_die tap-ctl destroy -p "$pid" -m "$minor" > /dev/null
+
+    release_lock "block"
 }
 
 command=$1
@@ -110,6 +247,14 @@ parse_target "$target"
 
 check_tools || exit 1
 
+mode=$( xenstore_read $XENBUS_PATH/mode )
+mode=$( canonicalise_mode $mode )
+
+# needed for same_vm
+FRONTEND_ID=$(xenstore_read "$XENBUS_PATH/frontend-id")
+FRONTEND_UUID=$(xenstore_read_default \
+                    "/local/domain/$FRONTEND_ID/vm" 'unknown')
+
 case $command in
 add)
     add
Signature · 1 line
2.39.2
Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] hotplug: Update block-tap

tools/hotplug/Linux/block-tap line 23

Quoted 13 lines · Jason Andryuk
tools/hotplug/Linux/block-tap +2 -2
index 89247921b9..126e472786 100755
@@ -18,11 +18,11 @@
 #
 # Usage:
 #
-# Target should be specified using the following syntax:
+# Disks should be specified using the following syntax:
 #
-# script=block-tap,vdev=xvda,target=<type>:<file>
+# vdev=xvda,backendtype=tap,format=vhd,target=/srv/target.vhd

I still have unanswered question from the previous round:
Is block-tap still going to work with the current example given in the script header? That is:
"script=block-tap,vdev=xvda,target=<type>:<file>"
Or maybe, that example is already broken?

If it's not broken, there could be users which rely on it. But maybe it's not really broken, and the new syntax is better anyway.

My guess is that using "script=block-tap,..." might still work, but we should say something in the CHANGELOG to encourage people to move to the new syntax, with "backendtype=tap" to avoid issues.

In any case, the patch looks good:

Thanks,

Signature · 1 line
Anthony PERARD
#3 lore raw

Re: [PATCH v2 1/3] hotplug: Update block-tap

Quoted 13 lines · Anthony PERARD
Quoted 11 lines · Jason Andryuk
tools/hotplug/Linux/block-tap +1 -1
index 89247921b9..126e472786 100755
@@ -18,11 +18,11 @@
 #
 # Usage:
 #
-# Target should be specified using the following syntax:
+# Disks should be specified using the following syntax:
 #
Quoted 8 lines

tools/hotplug/Linux/block-tap line 23

Quoted 2 lines
tools/hotplug/Linux/block-tap +1 -1
-# script=block-tap,vdev=xvda,target=<type>:<file>
+# vdev=xvda,backendtype=tap,format=vhd,target=/srv/target.vhd

I still have unanswered question from the previous round:
Is block-tap still going to work with the current example given in the script header? That is:
"script=block-tap,vdev=xvda,target=<type>:<file>"
Or maybe, that example is already broken?

Oh, right. Sorry about that.

Quoted 6 lines

If it's not broken, there could be users which rely on it. But maybe it's not really broken, and the new syntax is better anyway.

My guess is that using "script=block-tap,..." might still work, but we should say something in the CHANGELOG to encourage people to move to the new syntax, with "backendtype=tap" to avoid issues.

I think the old syntax with "backendtype=phy" would work except for this:
- write_dev $dev
...

+    # dev, as a unix socket, would end up with major:minor 0:0 in
+    # physical-device if write_dev were used.  tapback would be thrown off by
+    # that incorrect minor, so just skip writing physical-device.
+    xenstore_write "$XENBUS_PATH/physical-device-path" "$dev"

write_dev is needed for blkback to see physical-device and set things up properly.

I could create a second script, but that's a little silly for just the single line. Another approach would be to differentiate off the device type, vbd vs. vbd3, and use write_dev or not that way. Should I just do that?

block-tap will only support backendtype=phy as long as blktap uses the kernel driver. In that case tap-ctl create is returning the kernel module's block dev path. Once the kernel drive support is removed, backendtype=tap will be the only option - without it there is no block dev for blkback.

FYI, "backendtype=tap" is more performant because it is a shorter data path:
blkfront -> tapback/tapdisk
vs
blkfront -> blkback -> /dev/xen/blktap-2/tapdevN -> tapdisk

There are extra copies between blktap and the tapdev.

Quoted 2 lines

In any case, the patch looks good:

Thanks,
Jason

#4 lore raw

PATCH v2 2/3 libxl: Support blktap with HVM device model

In reply to Jason Andryuk

blktap exposes disks over UNIX socket Network Block Device (NBD). Modify libxl__device_disk_find_local_path() to provide back the QEMU-formatted NBD path. This allows tapdisk to be used for booting an HVM.

Use the nbd+unix:/// format specified by the protocol at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md

---
v2:
Add Anthony's R-b
Add matching { } to else

---
 tools/libs/light/libxl_disk.c | 18 ++++++++++++++----
 tools/libs/light/libxl_dm.c   |  1 -
 2 files changed, 14 insertions(+), 5 deletions(-)
tools/libs/light/libxl_disk.c +14 -4
index 819d34933b..432970f5a7 100644
@@ -1724,7 +1724,8 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,
      * If the format isn't raw and / or we're using a script, then see
      * if the script has written a path to the "cooked" node
      */
-    if (disk->script && guest_domid != INVALID_DOMID) {
+    if ((disk->script && guest_domid != INVALID_DOMID) ||
+        disk->backend == LIBXL_DISK_BACKEND_TAP) {
         libxl__device device;
         char *be_path, *pdpath;
         int rc;
@@ -1744,10 +1745,19 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,
         LOGD(DEBUG, guest_domid, "Attempting to read node %s", pdpath);
         path = libxl__xs_read(gc, XBT_NULL, pdpath);
 
-        if (path)
+        if (path) {
             LOGD(DEBUG, guest_domid, "Accessing cooked block device %s", path);
-        else
-            LOGD(DEBUG, guest_domid, "No physical-device-path, can't access locally.");
+
+            /* tapdisk exposes disks locally over UNIX socket NBD. */
+            if (disk->backend == LIBXL_DISK_BACKEND_TAP) {
+                path = libxl__sprintf(gc, "nbd+unix:///?socket=%s", path);
+                LOGD(DEBUG, guest_domid,
+                     "Directly accessing local TAP target %s", path);
+            }
+        } else {
+            LOGD(DEBUG, guest_domid,
+                "No physical-device-path, can't access locally.");
+	}
 
         goto out;
     }
tools/libs/light/libxl_dm.c +0 -1
index 0b03a7c747..1a24e7961a 100644
@@ -1915,7 +1915,6 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
                     continue;
                 }
 
-                assert(disks[i].backend != LIBXL_DISK_BACKEND_TAP);
                 target_path = libxl__device_disk_find_local_path(gc,
                                     guest_domid, &disks[i], true);
 
Signature · 1 line
2.39.2
#5 lore raw

Re: [PATCH v2 2/3] libxl: Support blktap with HVM device model

Quoted 11 lines · Jason Andryuk

blktap exposes disks over UNIX socket Network Block Device (NBD). Modify libxl__device_disk_find_local_path() to provide back the QEMU-formatted NBD path. This allows tapdisk to be used for booting an HVM.

Use the nbd+unix:/// format specified by the protocol at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md

Please remember to keep tags in chronological order, i.e. R-b after S-o-b.

It's also not clear to me whether this could sensibly go in ahead of patch 1; for now I'll assume it shouldn't.

Jan

#6 lore raw

PATCH v2 3/3 CHANGELOG: Mention libxl blktap/tapback support

In reply to Jason Andryuk

From: Jason Andryuk <[email protected]>

Add entry for backendtype=tap support in libxl. blktap needs some changes to work with libxl, which haven't been merged. They are available from this PR: https://github.com/xapi-project/blktap/pull/394

---
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
CHANGELOG.md +1 -0
index 8041cfb7d2..036328433d 100644
@@ -21,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
      for IPIs and Physical addressing mode for external interrupts.
  - Add a new 9pfs backend running as a daemon in dom0. First user is
    Xenstore-stubdom now being able to support full Xenstore trace capability.
+ - libxl support for backendtype=tap with tapback.
 
 ### Removed
 - caml-stubdom.  It hasn't built since 2014, was pinned to Ocaml 4.02, and has
Signature · 1 line
2.39.2
Acked-by on v2 3/3
#7 lore raw

Re: [PATCH v2 3/3] CHANGELOG: Mention libxl blktap/tapback support

Quoted 19 lines · Jason Andryuk

From: Jason Andryuk <[email protected]>

Add entry for backendtype=tap support in libxl.  blktap needs some changes to work with libxl, which haven't been merged.  They are available from this PR:
https://github.com/xapi-project/blktap/pull/394

---
 CHANGELOG.md | 1 +

 1 file changed, 1 insertion(+)
CHANGELOG.md +0 -0
index 8041cfb7d2..036328433d 100644
@@ -21,6 +21,7 @@ The format is based on [Keep a
 Changelog](https://keepachangelog.com/en/1.0.0/)

      for IPIs and Physical addressing mode for external interrupts.

Quoted 8 lines

  - Add a new 9pfs backend running as a daemon in dom0. First user is Xenstore-stubdom now being able to support full Xenstore trace capability.
+ - libxl support for backendtype=tap with tapback.

 ### Removed
 - caml-stubdom.  It hasn't built since 2014, was pinned to Ocaml 4.02, and has

~ Oleksii