Skip to contentSkip to conversation

libxl: blktap/tapback support

PATCH Superseded Jason Andryuk 4 participants 15 messages Feb 1, 2024 – Feb 12, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover libxl: blktap/tapback support

This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.

blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.

It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.

The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.

This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path. (It works with the kernel blktap driver as well - upstream blktap hasn't removed the blktap driver requirement yet -
https://github.com/xapi-project/blktap/pull/364)

An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'

VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypted VHDs which is not available in QEMU.

blktap's tapback needs minimal changes to work with libxl:
https://github.com/xapi-project/blktap/pull/394

Jason Andryuk (4):
block-common: Fix same_vm for no targets
libxl: Add support for blktap vbd3
hotplug: Update block-tap
libxl: Support blktap with HVM device model

 docs/man/xl-disk-configuration.5.pod.in   |   4 +-
 tools/hotplug/Linux/block-common.sh       |   8 +-
 tools/hotplug/Linux/block-tap             | 162 ++++++++++++++++++++--
 tools/libs/light/libxl_device.c           |  14 +-
 tools/libs/light/libxl_disk.c             |  37 +++--
 tools/libs/light/libxl_dm.c               |   1 -
 tools/libs/light/libxl_linux.c            |   1 +
 tools/libs/light/libxl_types_internal.idl |   1 +
 tools/libs/light/libxl_utils.c            |   2 +
 9 files changed, 201 insertions(+), 29 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/4 block-common: Fix same_vm for no targets

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

---
 tools/hotplug/Linux/block-common.sh | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
tools/hotplug/Linux/block-common.sh +3 -5
index f86a88c4eb..5c80237d99 100644
@@ -112,14 +112,12 @@ same_vm()
                   "$FRONTEND_UUID")
   local target=$(xenstore_read_default  "/local/domain/$FRONTEND_ID/target"   \
                  "-1")
-  local targetvm=$(xenstore_read_default "/local/domain/$target/vm" "-1")
+  local targetvm=$(xenstore_read_default "/local/domain/$target/vm" "No Target")
   local otarget=$(xenstore_read_default  "/local/domain/$otherdom/target"   \
                  "-1")
   local otvm=$(xenstore_read_default  "/local/domain/$otarget/vm"   \
-                 "-1")
-  otvm=${otvm%-1}
-  othervm=${othervm%-1}
-  targetvm=${targetvm%-1}
+                 "No Other Target")
+
   local frontend_uuid=${FRONTEND_UUID%-1}
   
   [ "$frontend_uuid" = "$othervm" -o "$targetvm" = "$othervm" -o \
Signature · 1 line
2.43.0
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] block-common: Fix same_vm for no targets

Quoted 15 lines · Jason Andryuk

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Quoted 7 lines

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

Thanks,

Signature · 1 line
Anthony PERARD
#3 lore raw

Re: [PATCH 1/4] block-common: Fix same_vm for no targets

Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

Quoted 5 lines
Quoted 3 lines

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.

Jan

#4 lore raw

Re: [PATCH 1/4] block-common: Fix same_vm for no targets

Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.

Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.

Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192

f3a7ca02400d1 ("hotplug/Linux: fix same_vm check in block script")

I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.

I'll have to double check there isn't something in the regular block that might prevent that.

Regards,
Jason

#5 lore raw

Re: [PATCH 1/4] block-common: Fix same_vm for no targets

Quoted 38 lines · Jason Andryuk
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.

Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.

Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192

Quoted 8 lines

f3a7ca02400d1 ("hotplug/Linux: fix same_vm check in block script")

I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.

I'll have to double check there isn't something in the regular block that might prevent that.

Okay, I'll wait for a result here before deciding whether to queue.

Jan

#6 lore raw

Re: [PATCH 1/4] block-common: Fix same_vm for no targets

Quoted 42 lines · Jan Beulich
Quoted 40 lines · Jason Andryuk
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk

same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:

++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5

The final comparison returns true since the two empty strings match:

++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'

Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'

I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.

Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.

Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192

f3a7ca02400d1 ("hotplug/Linux: fix same_vm check in block script")

Quoted 8 lines
Quoted 6 lines

I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.

I'll have to double check there isn't something in the regular block that might prevent that.

Okay, I'll wait for a result here before deciding whether to queue.

Yes, it should be backported. This patch prevents sharing writable block devs. Files are still broken because of issues identified previously here:
https://lore.kernel.org/xen-devel/CAKf6xpv-U91nF2Fik7GRN3SFeOWWcdR5R+ZcK5fgojE+-D43sg@mail.gmail.com/

Regards,
Jason

#7 lore raw

PATCH 2/4 libxl: Add support for blktap vbd3

In reply to Jason Andryuk

This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.

blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.

It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.

The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.

This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path.

backendtype=tap was not fully removed previously, but it would never succeed since it would hit the hardcoded error in disk_try_backend(). It is reused now.

An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'

Format raw also works to run an "aio:" tapdisk.

---
VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypting VHDs which is not available in QEMU.

---
 docs/man/xl-disk-configuration.5.pod.in   |  4 +++-
 tools/libs/light/libxl_device.c           | 14 ++++++++++----
 tools/libs/light/libxl_disk.c             | 20 +++++++++++++++-----
 tools/libs/light/libxl_linux.c            |  1 +
 tools/libs/light/libxl_types_internal.idl |  1 +
 tools/libs/light/libxl_utils.c            |  2 ++
 6 files changed, 32 insertions(+), 10 deletions(-)
docs/man/xl-disk-configuration.5.pod.in +3 -1
index bc945cc517..cb442bd5b4 100644
@@ -232,7 +232,7 @@ Specifies the backend implementation to use
 
 =item Supported values
 
-phy, qdisk, standalone
+phy, qdisk, standalone, tap
 
 =item Mandatory
 
@@ -254,6 +254,8 @@ and "standalone" does not support specifications other than "virtio".
 Normally this option should not be specified, in which case libxl will
 automatically determine the most suitable backend.
 
+"tap" needs blktap's tapback to be running.
+
 
 =item B<script>=I<SCRIPT>
 
tools/libs/light/libxl_device.c +10 -4
index 13da6e0573..ae2b71b0bf 100644
@@ -328,9 +328,15 @@ static int disk_try_backend(disk_try_backend_args *a,
         return 0;
 
     case LIBXL_DISK_BACKEND_TAP:
-        LOG(DEBUG, "Disk vdev=%s, backend tap unsuitable because blktap "
-                   "not available", a->disk->vdev);
-        return 0;
+        if (a->disk->format != LIBXL_DISK_FORMAT_RAW &&
+            a->disk->format != LIBXL_DISK_FORMAT_VHD)
+            goto bad_format;
+
+        if (libxl_defbool_val(a->disk->colo_enable))
+            goto bad_colo;
+
+        LOG(DEBUG, "Disk vdev=%s, returning blktap", a->disk->vdev);
+        return backend;
 
     case LIBXL_DISK_BACKEND_QDISK:
         if (a->disk->script) goto bad_script;
@@ -478,7 +484,7 @@ char *libxl__device_disk_string_of_backend(libxl_disk_backend backend)
 {
     switch (backend) {
         case LIBXL_DISK_BACKEND_QDISK: return "qdisk";
-        case LIBXL_DISK_BACKEND_TAP: return "phy";
+        case LIBXL_DISK_BACKEND_TAP: return "vbd3";
         case LIBXL_DISK_BACKEND_PHY: return "phy";
         case LIBXL_DISK_BACKEND_STANDALONE: return "standalone";
         default: return NULL;
tools/libs/light/libxl_disk.c +15 -5
index ea3623dd6f..59ff996837 100644
@@ -56,7 +56,9 @@ static void disk_eject_xswatch_callback(libxl__egc *egc, libxl__ev_xswatch *w,
             "/local/domain/%d/backend/%" TOSTRING(BACKEND_STRING_SIZE)
            "[a-z]/%*d/%*d",
            &disk->backend_domid, backend_type);
-    if (!strcmp(backend_type, "tap") || !strcmp(backend_type, "vbd")) {
+    if (!strcmp(backend_type, "tap") ||
+        !strcmp(backend_type, "vbd") ||
+        !strcmp(backend_type, "vbd3")) {
         disk->backend = LIBXL_DISK_BACKEND_TAP;
     } else if (!strcmp(backend_type, "qdisk")) {
         disk->backend = LIBXL_DISK_BACKEND_QDISK;
@@ -224,7 +226,7 @@ static int libxl__device_from_disk(libxl__gc *gc, uint32_t domid,
             device->backend_kind = LIBXL__DEVICE_KIND_VBD;
             break;
         case LIBXL_DISK_BACKEND_TAP:
-            device->backend_kind = LIBXL__DEVICE_KIND_VBD;
+            device->backend_kind = LIBXL__DEVICE_KIND_VBD3;
             break;
         case LIBXL_DISK_BACKEND_QDISK:
             device->backend_kind = LIBXL__DEVICE_KIND_QDISK;
@@ -368,9 +370,17 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,
                 assert(device->backend_kind == LIBXL__DEVICE_KIND_VIRTIO_DISK);
                 break;
             case LIBXL_DISK_BACKEND_TAP:
-                LOG(ERROR, "blktap is not supported");
-                rc = ERROR_FAIL;
-                goto out;
+                flexarray_append(back, "params");
+                flexarray_append(back, GCSPRINTF("%s:%s",
+                              libxl__device_disk_string_of_format(disk->format),
+                              disk->pdev_path ? : ""));
+
+                script = libxl__abs_path(gc, disk->script?: "block-tap",
+                                         libxl__xen_script_dir_path());
+                flexarray_append_pair(back, "script", script);
+
+                assert(device->backend_kind == LIBXL__DEVICE_KIND_VBD3);
+                break;
             case LIBXL_DISK_BACKEND_QDISK:
                 flexarray_append(back, "params");
                 flexarray_append(back, GCSPRINTF("%s:%s",
tools/libs/light/libxl_linux.c +1 -0
index f7c92ba562..0b4c8bd045 100644
@@ -207,6 +207,7 @@ int libxl__get_hotplug_script_info(libxl__gc *gc, libxl__device *dev,
 
     switch (dev->backend_kind) {
     case LIBXL__DEVICE_KIND_VBD:
+    case LIBXL__DEVICE_KIND_VBD3:
         if (num_exec != 0) {
             LOGD(DEBUG, dev->domid,
                  "num_exec %d, not running hotplug scripts", num_exec);
tools/libs/light/libxl_types_internal.idl +1 -0
index e24288f1a5..56dccac153 100644
@@ -34,6 +34,7 @@ libxl__device_kind = Enumeration("device_kind", [
     (16, "VINPUT"),
     (17, "VIRTIO_DISK"),
     (18, "VIRTIO"),
+    (19, "VBD3"),
     ])
 
 libxl__console_backend = Enumeration("console_backend", [
tools/libs/light/libxl_utils.c +2 -0
index e403bd9bcf..10398a6c86 100644
@@ -295,6 +295,8 @@ int libxl_string_to_backend(libxl_ctx *ctx, char *s, libxl_disk_backend *backend
         *backend = LIBXL_DISK_BACKEND_PHY;
     } else if (!strcmp(s, "file")) {
         *backend = LIBXL_DISK_BACKEND_TAP;
+    } else if (!strcmp(s, "vbd3")) {
+        *backend = LIBXL_DISK_BACKEND_TAP;
     } else if (!strcmp(s, "qdisk")) {
         *backend = LIBXL_DISK_BACKEND_QDISK;
     } else if (!strcmp(s, "standalone")) {
Signature · 1 line
2.43.0
Reviewed-by on v1 2/4
#8 lore raw

Re: [PATCH 2/4] libxl: Add support for blktap vbd3

Quoted 26 lines · Jason Andryuk

This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.

blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.

It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.

The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.

This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path.

backendtype=tap was not fully removed previously, but it would never succeed since it would hit the hardcoded error in disk_try_backend(). It is reused now.

Quoted 6 lines

An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'

Format raw also works to run an "aio:" tapdisk.

Thanks,

Signature · 1 line
Anthony PERARD
#9 lore raw

PATCH 3/4 hotplug: Update block-tap

In reply to Jason Andryuk

Implement a sharing check like the regular block script.

Checking tapback inside block-tap is too late since it needs to be running to transition the backend to InitWait before block-tap is run.

tap-ctl check will be removed when the requirement for the blktap kernel driver is removed. Remove it now as it is of limited use.

find_device() needs to be non-fatal allow a sharing check.

Only write physical-device-path because that is all that tapback needs. Also write_dev doesn't handled files and would incorrectly store physical-device as 0:0 which would confuse the minor inside tapback

---
 tools/hotplug/Linux/block-tap | 162 ++++++++++++++++++++++++++++++++--
 1 file changed, 153 insertions(+), 9 deletions(-)
tools/hotplug/Linux/block-tap +153 -9
index 89247921b9..5eca09f0f6 100755
@@ -37,10 +37,6 @@ check_tools()
     if ! command -v tap-ctl > /dev/null 2>&1; then
         fatal "Unable to find tap-ctl tool"
     fi
-    modprobe blktap
-    if ! tap-ctl check >& /dev/null ; then
-	fatal "Blocktap kernel module not available"
-    fi
 }
 
 # Sets the following global variables based on the params field passed in as
@@ -81,7 +77,105 @@ find_device()
     done
 
     if [ -z "$pid" ] || [ -z "$minor" ]; then
-        fatal "cannot find required parameters"
+        return 1
+    fi
+
+    return 0
+}
+
+count_using()
+{
+  local file="$1"
+  local f
+
+  local i=0
+  local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+  for dom in $(xenstore-list "$base_path")
+  do
+    for dev in $(xenstore-list "$base_path/$dom")
+    do
+      f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
+      f=$(echo "$f" | cut -d ":" -f 2)
+
+      if [ -n "$f" ] && [ "$file" = $f ] ; then
+          i=$(( i + 1 ))
+      fi
+    done
+  done
+
+  echo "$i"
+}
+
+# tap_shared is used to determine if a shared tap can be closed
+# Since a stubdom and a guest both use the same tap, it can only
+# be freed when there is a single one left.
+tap_shared() {
+    [ $( count_using "$file" ) -gt 1 ]
+}
+
+check_tap_sharing()
+{
+  local file="$1"
+  local mode="$2"
+  local dev
+
+  local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+  for dom in $(xenstore-list "$base_path") ; do
+    for dev in $(xenstore-list "$base_path/$dom") ; do
+      f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
+      f=$(echo "$f" | cut -d ":" -f 2)
+
+      if [ -n "$f" ] && [ "$file" = "$f" ] ; then
+        if [ "$mode" = 'w' ] ; then
+          if ! same_vm $dom ; then
+            echo "guest $f"
+            return
+          fi
+        else
+          local m=$(xenstore_read_default "$base_path/$dom/$dev/mode" "")
+          m=$(canonicalise_mode "$m")
+
+          if [ "$m" = 'w' ] ; then
+            if ! same_vm $dom ; then
+              echo "guest $f"
+              return
+            fi
+          fi
+        fi
+      fi
+    done
+  done
+
+  echo 'ok'
+}
+
+tap_create()
+{
+    if ! minor=$( tap-ctl allocate ) ; then
+        fatal "Could not allocate minor"
+    fi
+
+    # Handle with or without kernel blktap
+    minor=${minor#/run/blktap-control/tapdisk/tapdisk-}
+    minor=${minor#/dev/xen/blktap-2/tapdev}
+
+    # tap-ctl is spawning tapdisk which would hold the _lockfd open.
+    # Ensure it is closed before running tap-ctl spawn, which needs to be
+    # done in a subshell to continue holding the lock in the parent.
+    if ! pid=$( ( eval "exec $_lockfd>&-" ; tap-ctl spawn ) ) ; then
+        tap-ctl free -m "$minor"
+        fatal "Could not spawn tapdisk for $minor"
+    fi
+
+    if ! tap-ctl attach -p "$pid" -m "$minor" ; then
+        tap-ctl free -m "$minor"
+        fatal "Could not attach $pid and $minor"
+    fi
+
+    if ! tap-ctl open -p "$pid" -m "$minor" -a "$target" ; then
+        tap-ctl detach -p "$pid" -m "$minor"
+        tap-ctl free -m "$minor"
+        fatal "Could not open \"$target\""
     fi
 }
 
@@ -89,15 +183,57 @@ find_device()
 # the device
 add()
 {
-    dev=$(tap-ctl create -a $target)
-    write_dev $dev
+    local minor
+    local pid
+    local res
+
+    claim_lock "block"
+
+    if find_device; then
+        result=$( check_tap_sharing "$file" "$mode" )
+        if [ "$result" != "ok" ] ; then
+            do_ebusy "tap $type file $file in use " "$mode" "${result%% *}"
+        fi
+    else
+        tap_create
+    fi
+
+    # Create nbd unix path
+    dev=$( printf "/run/blktap-control/nbd%ld.%d" "$pid" "$minor" )
+
+    xenstore_write "$XENBUS_PATH/pid" "$pid"
+    xenstore_write "$XENBUS_PATH/minor" "$minor"
+    # dev, as a unix socket, would end up with major:minor 0:0 in
+    # physical-device if write_dev were used.  tapback would be thrown off by
+    # that incorrect minor, so just skip writing physical-device.
+    xenstore_write "$XENBUS_PATH/physical-device-path" "$dev"
+
+    success
+
+    release_lock "block"
 }
 
 # Disconnects the device
 remove()
 {
-    find_device
-    do_or_die tap-ctl destroy -p ${pid} -m ${minor} > /dev/null
+    local minor
+    local pid
+
+    claim_lock "block"
+
+    if tap_shared ; then
+        return
+    fi
+
+    minor=$( xenstore_read "$XENBUS_PATH/minor" )
+    pid=$( xenstore_read "$XENBUS_PATH/pid" )
+
+    [ -n "$minor" ] || fatal "minor missing"
+    [ -n "$pid" ] || fatal "pid missing"
+    do_or_die tap-ctl close -p "$pid" -m "$minor" > /dev/null
+    do_or_die tap-ctl detach -p "$pid" -m "$minor" > /dev/null
+
+    release_lock "block"
 }
 
 command=$1
@@ -110,6 +246,14 @@ parse_target "$target"
 
 check_tools || exit 1
 
+mode=$( xenstore_read $XENBUS_PATH/mode )
+mode=$( canonicalise_mode $mode )
+
+# needed for same_vm
+FRONTEND_ID=$(xenstore_read "$XENBUS_PATH/frontend-id")
+FRONTEND_UUID=$(xenstore_read_default \
+                    "/local/domain/$FRONTEND_ID/vm" 'unknown')
+
 case $command in
 add)
     add
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH 3/4] hotplug: Update block-tap

Quoted 5 lines · Jason Andryuk

Implement a sharing check like the regular block script.

Checking tapback inside block-tap is too late since it needs to be running to transition the backend to InitWait before block-tap is run.

Quoted 8 lines

tap-ctl check will be removed when the requirement for the blktap kernel driver is removed. Remove it now as it is of limited use.

find_device() needs to be non-fatal allow a sharing check.

Only write physical-device-path because that is all that tapback needs. Also write_dev doesn't handled files and would incorrectly store physical-device as 0:0 which would confuse the minor inside tapback

Missing SOB.

Is block-tap still going to work with the example given in the script header? That is:

    "script=block-tap,vdev=xvda,target=<type>:<file>"

Or maybe, that example is already broken?

Quoted 5 lines

---

tools/hotplug/Linux/block-tap +0 -0
index 89247921b9..5eca09f0f6 100755

tools/hotplug/Linux/block-tap find_device() · line 95

Quoted 10 lines
tools/hotplug/Linux/block-tap +10 -0
+count_using()
+{
+  local file="$1"
+  local f
+
+  local i=0
+  local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+  for dom in $(xenstore-list "$base_path")
+  do
+    for dev in $(xenstore-list "$base_path/$dom")

This function is probably missing "local dom dev".

Quoted 12 lines
tools/hotplug/Linux/block-tap +12 -0
+    do
+      f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
+      f=$(echo "$f" | cut -d ":" -f 2)
+
+      if [ -n "$f" ] && [ "$file" = $f ] ; then
+          i=$(( i + 1 ))
+      fi
+    done
+  done
+
+  echo "$i"
+}

tools/hotplug/Linux/block-tap find_device() · line 123

Quoted 10 lines
tools/hotplug/Linux/block-tap +9 -0
+
 
+check_tap_sharing()
+{
+  local file="$1"
+  local mode="$2"
+  local dev
+
+  local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE"
+  for dom in $(xenstore-list "$base_path") ; do

Should we add "local dom" to the function?

tools/hotplug/Linux/block-tap find_device() · line 125

Quoted 2 lines
tools/hotplug/Linux/block-tap +2 -0
+    for dev in $(xenstore-list "$base_path/$dom") ; do
+      f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")

Same here, maybe "local f" would be good to have too.

Quoted 9 lines
tools/hotplug/Linux/block-tap +3 -2
@@ -89,15 +183,57 @@ find_device()
 # the device
 add()
 {
-    dev=$(tap-ctl create -a $target)
-    write_dev $dev
+    local minor
+    local pid
+    local res

tools/hotplug/Linux/block-tap find_device() · line 198

Quoted 10 lines
tools/hotplug/Linux/block-tap +10 -0
+
+    claim_lock "block"
+
+    if find_device; then
+        result=$( check_tap_sharing "$file" "$mode" )
+        if [ "$result" != "ok" ] ; then
+            do_ebusy "tap $type file $file in use " "$mode" "${result%% *}"
+        fi
+    else
+        tap_create

The new function tap_create() is doing something similar to the replace tap-ctl create call, right?

Quoted 11 lines
tools/hotplug/Linux/block-tap +6 -2
 # Disconnects the device
 remove()
 {
-    find_device
-    do_or_die tap-ctl destroy -p ${pid} -m ${minor} > /dev/null
+    local minor
+    local pid
+
+    claim_lock "block"
+
+    if tap_shared ; then

tools/hotplug/Linux/block-tap find_device() · line 234

Quoted 10 lines
tools/hotplug/Linux/block-tap +10 -0
+        return
+    fi
+
+    minor=$( xenstore_read "$XENBUS_PATH/minor" )
+    pid=$( xenstore_read "$XENBUS_PATH/pid" )
+
+    [ -n "$minor" ] || fatal "minor missing"
+    [ -n "$pid" ] || fatal "pid missing"
+    do_or_die tap-ctl close -p "$pid" -m "$minor" > /dev/null
+    do_or_die tap-ctl detach -p "$pid" -m "$minor" > /dev/null

Should we also call tap-ctl free, like tap-ctl destroy seems to do?

tools/hotplug/Linux/block-tap find_device() · line 239

Quoted 5 lines
tools/hotplug/Linux/block-tap +2 -0
+
+    release_lock "block"
 }
 
 command=$1

Thanks,

Signature · 1 line
Anthony PERARD
#11 lore raw

PATCH 4/4 libxl: Support blktap with HVM device model

In reply to Jason Andryuk

blktap exposes disks over UNIX socket Network Block Device (NBD). Modify libxl__device_disk_find_local_path() to provide back the QEMU-formatted NBD path. This allows tapdisk to be used for booting an HVM.

Use the nbd+unix:/// format specified by the protocol at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md

---
 tools/libs/light/libxl_disk.c | 17 +++++++++++++----
 tools/libs/light/libxl_dm.c   |  1 -
 2 files changed, 13 insertions(+), 5 deletions(-)
tools/libs/light/libxl_disk.c +13 -4
index 59ff996837..b65cad33cc 100644
@@ -1317,7 +1317,8 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,
      * If the format isn't raw and / or we're using a script, then see
      * if the script has written a path to the "cooked" node
      */
-    if (disk->script && guest_domid != INVALID_DOMID) {
+    if ((disk->script && guest_domid != INVALID_DOMID) ||
+        disk->backend == LIBXL_DISK_BACKEND_TAP) {
         libxl__device device;
         char *be_path, *pdpath;
         int rc;
@@ -1337,10 +1338,18 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,
         LOGD(DEBUG, guest_domid, "Attempting to read node %s", pdpath);
         path = libxl__xs_read(gc, XBT_NULL, pdpath);
 
-        if (path)
+        if (path) {
             LOGD(DEBUG, guest_domid, "Accessing cooked block device %s", path);
-        else
-            LOGD(DEBUG, guest_domid, "No physical-device-path, can't access locally.");
+
+            /* tapdisk exposes disks locally over UNIX socket NBD. */
+            if (disk->backend == LIBXL_DISK_BACKEND_TAP) {
+                path = libxl__sprintf(gc, "nbd+unix:///?socket=%s", path);
+                LOGD(DEBUG, guest_domid,
+                     "Directly accessing local TAP target %s", path);
+            }
+        } else
+            LOGD(DEBUG, guest_domid,
+                "No physical-device-path, can't access locally.");
 
         goto out;
     }
tools/libs/light/libxl_dm.c +0 -1
index 14b593110f..f7c796011d 100644
@@ -1915,7 +1915,6 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
                     continue;
                 }
 
-                assert(disks[i].backend != LIBXL_DISK_BACKEND_TAP);
                 target_path = libxl__device_disk_find_local_path(gc,
                                     guest_domid, &disks[i], true);
 
Signature · 1 line
2.43.0
Reviewed-by on v1 4/4
#12 lore raw

Re: [PATCH 4/4] libxl: Support blktap with HVM device model

Quoted 10 lines · Jason Andryuk
tools/libs/light/libxl_disk.c +1 -3
@@ -1337,10 +1338,18 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,
         LOGD(DEBUG, guest_domid, "Attempting to read node %s", pdpath);
         path = libxl__xs_read(gc, XBT_NULL, pdpath);
 
-        if (path)
+        if (path) {
             LOGD(DEBUG, guest_domid, "Accessing cooked block device %s", path);
-        else
-            LOGD(DEBUG, guest_domid, "No physical-device-path, can't access locally.");

tools/libs/light/libxl_disk.c char *libxl__device_disk_find_local_path(libxl__gc *gc, · line 1352

Quoted 10 lines
tools/libs/light/libxl_disk.c +10 -0
+
+            /* tapdisk exposes disks locally over UNIX socket NBD. */
+            if (disk->backend == LIBXL_DISK_BACKEND_TAP) {
+                path = libxl__sprintf(gc, "nbd+unix:///?socket=%s", path);
+                LOGD(DEBUG, guest_domid,
+                     "Directly accessing local TAP target %s", path);
+            }
+        } else
+            LOGD(DEBUG, guest_domid,
+                "No physical-device-path, can't access locally.");

Here, the coding style call to have both side of the if..else to have { } or none of them. Could you add {}-block for the else side?

Beside that, patch looks fine to me:

Thanks,

Signature · 1 line
Anthony PERARD
#13 lore raw
In reply to Jason Andryuk
Quoted 5 lines · Jason Andryuk

Jason Andryuk (4):
block-common: Fix same_vm for no targets
libxl: Add support for blktap vbd3
hotplug: Update block-tap
libxl: Support blktap with HVM device model

If libxl is learning how to drive tapdisk, then it's time for:

https://github.com/xenserver/xen.pg/blob/XS-8/patches/xentop-vbd3.patch

to make it's way upstream.  Otherwise, there's an absence of disk info in xentop.

I previously kept it back because vbd3 was specific to XenServer.

~Andrew

#14 lore raw
In reply to Jason Andryuk
Quoted 28 lines · Jason Andryuk

This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.

blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.

It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.

The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.

This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path. (It works with the kernel blktap driver as well - upstream blktap hasn't removed the blktap driver requirement yet -
https://github.com/xapi-project/blktap/pull/364)

An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'

Quoted 6 lines

VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypted VHDs which is not available in QEMU.

blktap's tapback needs minimal changes to work with libxl:
https://github.com/xapi-project/blktap/pull/394

It might be nice to have an entry in the ChangeLog. But it's looks like the blktap project isn't ready yet, so we might need to point that out.

Cheers,

Signature · 1 line
Anthony PERARD