libxl: blktap/tapback support
cover libxl: blktap/tapback support
This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.
blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.
It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.
The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.
This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path. (It works with the kernel blktap driver as well - upstream blktap hasn't removed the blktap driver requirement yet -
https://github.com/xapi-project/blktap/pull/364)
An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'
VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypted VHDs which is not available in QEMU.
blktap's tapback needs minimal changes to work with libxl:
https://github.com/xapi-project/blktap/pull/394
Jason Andryuk (4):
block-common: Fix same_vm for no targets
libxl: Add support for blktap vbd3
hotplug: Update block-tap
libxl: Support blktap with HVM device model
docs/man/xl-disk-configuration.5.pod.in | 4 +- tools/hotplug/Linux/block-common.sh | 8 +- tools/hotplug/Linux/block-tap | 162 ++++++++++++++++++++-- tools/libs/light/libxl_device.c | 14 +- tools/libs/light/libxl_disk.c | 37 +++-- tools/libs/light/libxl_dm.c | 1 - tools/libs/light/libxl_linux.c | 1 + tools/libs/light/libxl_types_internal.idl | 1 + tools/libs/light/libxl_utils.c | 2 + 9 files changed, 201 insertions(+), 29 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/4 block-common: Fix same_vm for no targets
same_vm is broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:
++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5
The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
--- tools/hotplug/Linux/block-common.sh | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
tools/hotplug/Linux/block-common.sh +3 -5
@@ -112,14 +112,12 @@ same_vm() "$FRONTEND_UUID") local target=$(xenstore_read_default "/local/domain/$FRONTEND_ID/target" \ "-1") - local targetvm=$(xenstore_read_default "/local/domain/$target/vm" "-1") + local targetvm=$(xenstore_read_default "/local/domain/$target/vm" "No Target") local otarget=$(xenstore_read_default "/local/domain/$otherdom/target" \ "-1") local otvm=$(xenstore_read_default "/local/domain/$otarget/vm" \ - "-1") - otvm=${otvm%-1} - othervm=${othervm%-1} - targetvm=${targetvm%-1} + "No Other Target") + local frontend_uuid=${FRONTEND_UUID%-1} [ "$frontend_uuid" = "$othervm" -o "$targetvm" = "$othervm" -o \
Signature · 1 line
2.43.0
Re: [PATCH 1/4] block-common: Fix same_vm for no targets
Quoted 15 lines · Jason Andryuk
same_vmis broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Quoted 7 lines
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
Signed-off-by Jason Andryuk <[email protected]>
Thanks,
Signature · 1 line
Anthony PERARD
Re: [PATCH 1/4] block-common: Fix same_vm for no targets
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk
same_vmis broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
Quoted 5 lines
Quoted 3 lines
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
Signed-off-by Jason Andryuk <[email protected]>Reviewed-by Anthony PERARD <[email protected]>
I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.
Jan
Re: [PATCH 1/4] block-common: Fix same_vm for no targets
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk
same_vmis broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
Signed-off-by Jason Andryuk <[email protected]>Reviewed-by Anthony PERARD <[email protected]>I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.
Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.
Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192
f3a7ca02400d1 ("hotplug/Linux: fix same_vm check in block script")
I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.
I'll have to double check there isn't something in the regular block that might prevent that.
Regards,
Jason
Re: [PATCH 1/4] block-common: Fix same_vm for no targets
Quoted 38 lines · Jason Andryuk
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk
same_vmis broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
Signed-off-by Jason Andryuk <[email protected]>Reviewed-by Anthony PERARD <[email protected]>I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.
Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.
Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192
Quoted 8 lines
f3a7ca02400d1("hotplug/Linux: fixsame_vmcheck in block script")I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.
I'll have to double check there isn't something in the regular block that might prevent that.
Okay, I'll wait for a result here before deciding whether to queue.
Jan
Re: [PATCH 1/4] block-common: Fix same_vm for no targets
Quoted 42 lines · Jan Beulich
Quoted 40 lines · Jason Andryuk
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Anthony PERARD
Quoted 19 lines · Jason Andryuk
same_vmis broken when the two main domains do not have targets. otvm and targetvm are both missing, which means they get set to -1 and then converted to empty strings:++10697+ local targetvm=-1
++10697+ local otvm=-1
++10697+ otvm=
++10697+ othervm=/vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 ++10697+ targetvm=
++10697+ local frontend_uuid=/vm/844dea4e-44f8-4e3e-8145-325132a31ca5The final comparison returns true since the two empty strings match:
++10697+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o '' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = '' -o '' = '' ']'
Replace -1 with distinct strings indicating the lack of a value and remove the collescing to empty stings. The strings themselves will no longer match, and that is correct.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
++12364+ '[' /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o 'No target' = /vm/cc97bc2f-3a91-43f7-8fbc-4cb92f90b4e4 -o /vm/844dea4e-44f8-4e3e-8145-325132a31ca5 = 'No other target' -o 'No target' = 'No other target' ']'
Signed-off-by Jason Andryuk <[email protected]>Reviewed-by Anthony PERARD <[email protected]>I've committed this, but I take the absence of a Fixes: tag as indication that this doesn't want/need backporting.
Hmmm, maybe this should have a Fixes. Sorry I didn't investigate that better before submission.
Looks like this would be the commit:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=f3a7ca02400d1c416e97451b4aebfaf608fc8192
f3a7ca02400d1("hotplug/Linux: fixsame_vmcheck in block script")
Quoted 8 lines
Quoted 6 lines
I need to circle back on this. IIRC, when I set up a conflicting assignment of a writable disk to two VMs with block-tap, it was allowed and not denied. That is what prompted this change.
I'll have to double check there isn't something in the regular block that might prevent that.
Okay, I'll wait for a result here before deciding whether to queue.
Yes, it should be backported. This patch prevents sharing writable block devs. Files are still broken because of issues identified previously here:
https://lore.kernel.org/xen-devel/CAKf6xpv-U91nF2Fik7GRN3SFeOWWcdR5R+ZcK5fgojE+-D43sg@mail.gmail.com/
Regards,
Jason
PATCH 2/4 libxl: Add support for blktap vbd3
In reply to Jason AndryukThis patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.
blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.
It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.
The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.
This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path.
backendtype=tap was not fully removed previously, but it would never succeed since it would hit the hardcoded error in disk_try_backend(). It is reused now.
An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'
Format raw also works to run an "aio:" tapdisk.
---
VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypting VHDs which is not available in QEMU.
--- docs/man/xl-disk-configuration.5.pod.in | 4 +++- tools/libs/light/libxl_device.c | 14 ++++++++++---- tools/libs/light/libxl_disk.c | 20 +++++++++++++++----- tools/libs/light/libxl_linux.c | 1 + tools/libs/light/libxl_types_internal.idl | 1 + tools/libs/light/libxl_utils.c | 2 ++ 6 files changed, 32 insertions(+), 10 deletions(-)
docs/man/xl-disk-configuration.5.pod.in +3 -1
@@ -232,7 +232,7 @@ Specifies the backend implementation to use =item Supported values -phy, qdisk, standalone +phy, qdisk, standalone, tap =item Mandatory
@@ -254,6 +254,8 @@ and "standalone" does not support specifications other than "virtio". Normally this option should not be specified, in which case libxl will automatically determine the most suitable backend. +"tap" needs blktap's tapback to be running. + =item B<script>=I<SCRIPT>
tools/libs/light/libxl_device.c +10 -4
@@ -328,9 +328,15 @@ static int disk_try_backend(disk_try_backend_args *a,return 0 ;case LIBXL_DISK_BACKEND_TAP : -LOG (DEBUG ,"Disk vdev=%s, backend tap unsuitable because blktap " -"not available" , a->disk->vdev); -return 0 ; +if (a->disk->format !=LIBXL_DISK_FORMAT_RAW && + a->disk->format !=LIBXL_DISK_FORMAT_VHD ) + goto bad_format; + +if (libxl_defbool_val (a->disk->colo_enable)) + goto bad_colo; + +LOG (DEBUG ,"Disk vdev=%s, returning blktap" , a->disk->vdev); +return backend;case LIBXL_DISK_BACKEND_QDISK :if (a->disk->script) goto bad_script;
@@ -478,7 +484,7 @@ char *libxl__device_disk_string_of_backend(libxl_disk_backend backend) {switch (backend) {case LIBXL_DISK_BACKEND_QDISK :return "qdisk" ; -case LIBXL_DISK_BACKEND_TAP :return "phy" ; +case LIBXL_DISK_BACKEND_TAP :return "vbd3" ;case LIBXL_DISK_BACKEND_PHY :return "phy" ;case LIBXL_DISK_BACKEND_STANDALONE :return "standalone" ;default :return NULL ;
tools/libs/light/libxl_disk.c +15 -5
@@ -56,7 +56,9 @@ static void disk_eject_xswatch_callback(libxl__egc *egc, libxl__ev_xswatch *w, "/local/domain/%d/backend/%" TOSTRING(BACKEND_STRING_SIZE)"[a-z]/%*d/%*d" , &disk->backend_domid, backend_type); -if (!strcmp (backend_type,"tap" ) || !strcmp (backend_type,"vbd" )) { +if (!strcmp (backend_type,"tap" ) || + !strcmp (backend_type,"vbd" ) || + !strcmp (backend_type,"vbd3" )) { disk->backend =LIBXL_DISK_BACKEND_TAP ; }else if (!strcmp (backend_type,"qdisk" )) { disk->backend =LIBXL_DISK_BACKEND_QDISK ;
@@ -224,7 +226,7 @@ static int libxl__device_from_disk(libxl__gc *gc, uint32_t domid, device->backend_kind =LIBXL__DEVICE_KIND_VBD ;break ;case LIBXL_DISK_BACKEND_TAP : - device->backend_kind =LIBXL__DEVICE_KIND_VBD ; + device->backend_kind =LIBXL__DEVICE_KIND_VBD3 ;break ;case LIBXL_DISK_BACKEND_QDISK : device->backend_kind =LIBXL__DEVICE_KIND_QDISK ;
@@ -368,9 +370,17 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,assert (device->backend_kind ==LIBXL__DEVICE_KIND_VIRTIO_DISK );break ;case LIBXL_DISK_BACKEND_TAP : -LOG (ERROR ,"blktap is not supported" ); - rc =ERROR_FAIL ; - goto out; +flexarray_append (back,"params" ); +flexarray_append (back,GCSPRINTF ("%s:%s" , +libxl__device_disk_string_of_format (disk->format), + disk->pdev_path ? :"" )); + + script =libxl__abs_path (gc, disk->script?:"block-tap" , +libxl__xen_script_dir_path ()); +flexarray_append_pair (back,"script" , script); + +assert (device->backend_kind ==LIBXL__DEVICE_KIND_VBD3 ); +break ;case LIBXL_DISK_BACKEND_QDISK :flexarray_append (back,"params" ); flexarray_append(back, GCSPRINTF("%s:%s",
tools/libs/light/libxl_linux.c +1 -0
@@ -207,6 +207,7 @@ int libxl__get_hotplug_script_info(libxl__gc *gc, libxl__device *dev,switch (dev->backend_kind) {case LIBXL__DEVICE_KIND_VBD : +case LIBXL__DEVICE_KIND_VBD3 :if (num_exec !=0 ) {LOGD (DEBUG , dev->domid,"num_exec %d, not running hotplug scripts" , num_exec);
tools/libs/light/libxl_types_internal.idl +1 -0
@@ -34,6 +34,7 @@ libxl__device_kind = Enumeration("device_kind", [ (16, "VINPUT"), (17, "VIRTIO_DISK"), (18, "VIRTIO"), + (19, "VBD3"), ]) libxl__console_backend = Enumeration("console_backend", [
tools/libs/light/libxl_utils.c +2 -0
@@ -295,6 +295,8 @@ int libxl_string_to_backend(libxl_ctx *ctx, char *s, libxl_disk_backend *backend *backend =LIBXL_DISK_BACKEND_PHY ; }else if (!strcmp (s,"file" )) { *backend =LIBXL_DISK_BACKEND_TAP ; + }else if (!strcmp (s,"vbd3" )) { + *backend =LIBXL_DISK_BACKEND_TAP ; }else if (!strcmp (s,"qdisk" )) { *backend =LIBXL_DISK_BACKEND_QDISK ; }else if (!strcmp (s,"standalone" )) {
Signature · 1 line
2.43.0
Re: [PATCH 2/4] libxl: Add support for blktap vbd3
Quoted 26 lines · Jason Andryuk
This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.
blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.
It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.
The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.
This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path.
backendtype=tap was not fully removed previously, but it would never succeed since it would hit the hardcoded error in
disk_try_backend(). It is reused now.
Quoted 6 lines
An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'Format raw also works to run an "aio:" tapdisk.
Signed-off-by Jason Andryuk <[email protected]>
Thanks,
Signature · 1 line
Anthony PERARD
PATCH 3/4 hotplug: Update block-tap
In reply to Jason AndryukImplement a sharing check like the regular block script.
Checking tapback inside block-tap is too late since it needs to be running to transition the backend to InitWait before block-tap is run.
tap-ctl check will be removed when the requirement for the blktap kernel driver is removed. Remove it now as it is of limited use.
find_device() needs to be non-fatal allow a sharing check.
Only write physical-device-path because that is all that tapback needs. Also write_dev doesn't handled files and would incorrectly store physical-device as 0:0 which would confuse the minor inside tapback
--- tools/hotplug/Linux/block-tap | 162 ++++++++++++++++++++++++++++++++-- 1 file changed, 153 insertions(+), 9 deletions(-)
tools/hotplug/Linux/block-tap +153 -9
@@ -37,10 +37,6 @@ check_tools() if ! command -v tap-ctl > /dev/null 2>&1; then fatal "Unable to find tap-ctl tool" fi - modprobe blktap - if ! tap-ctl check >& /dev/null ; then - fatal "Blocktap kernel module not available" - fi } # Sets the following global variables based on the params field passed in as
@@ -81,7 +77,105 @@ find_device() done if [ -z "$pid" ] || [ -z "$minor" ]; then - fatal "cannot find required parameters" + return 1 + fi + + return 0 +} + +count_using() +{ + local file="$1" + local f + + local i=0 + local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE" + for dom in $(xenstore-list "$base_path") + do + for dev in $(xenstore-list "$base_path/$dom") + do + f=$(xenstore_read_default "$base_path/$dom/$dev/params" "") + f=$(echo "$f" | cut -d ":" -f 2) + + if [ -n "$f" ] && [ "$file" = $f ] ; then + i=$(( i + 1 )) + fi + done + done + + echo "$i" +} + +# tap_shared is used to determine if a shared tap can be closed +# Since a stubdom and a guest both use the same tap, it can only +# be freed when there is a single one left. +tap_shared() { + [ $( count_using "$file" ) -gt 1 ] +} + +check_tap_sharing() +{ + local file="$1" + local mode="$2" + local dev + + local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE" + for dom in $(xenstore-list "$base_path") ; do + for dev in $(xenstore-list "$base_path/$dom") ; do + f=$(xenstore_read_default "$base_path/$dom/$dev/params" "") + f=$(echo "$f" | cut -d ":" -f 2) + + if [ -n "$f" ] && [ "$file" = "$f" ] ; then + if [ "$mode" = 'w' ] ; then + if ! same_vm $dom ; then + echo "guest $f" + return + fi + else + local m=$(xenstore_read_default "$base_path/$dom/$dev/mode" "") + m=$(canonicalise_mode "$m") + + if [ "$m" = 'w' ] ; then + if ! same_vm $dom ; then + echo "guest $f" + return + fi + fi + fi + fi + done + done + + echo 'ok' +} + +tap_create() +{ + if ! minor=$( tap-ctl allocate ) ; then + fatal "Could not allocate minor" + fi + + # Handle with or without kernel blktap + minor=${minor#/run/blktap-control/tapdisk/tapdisk-} + minor=${minor#/dev/xen/blktap-2/tapdev} + + # tap-ctl is spawning tapdisk which would hold the _lockfd open. + # Ensure it is closed before running tap-ctl spawn, which needs to be + # done in a subshell to continue holding the lock in the parent. + if ! pid=$( ( eval "exec $_lockfd>&-" ; tap-ctl spawn ) ) ; then + tap-ctl free -m "$minor" + fatal "Could not spawn tapdisk for $minor" + fi + + if ! tap-ctl attach -p "$pid" -m "$minor" ; then + tap-ctl free -m "$minor" + fatal "Could not attach $pid and $minor" + fi + + if ! tap-ctl open -p "$pid" -m "$minor" -a "$target" ; then + tap-ctl detach -p "$pid" -m "$minor" + tap-ctl free -m "$minor" + fatal "Could not open \"$target\"" fi }
@@ -89,15 +183,57 @@ find_device() # the device add() { - dev=$(tap-ctl create -a $target) - write_dev $dev + local minor + local pid + local res + + claim_lock "block" + + if find_device; then + result=$( check_tap_sharing "$file" "$mode" ) + if [ "$result" != "ok" ] ; then + do_ebusy "tap $type file $file in use " "$mode" "${result%% *}" + fi + else + tap_create + fi + + # Create nbd unix path + dev=$( printf "/run/blktap-control/nbd%ld.%d" "$pid" "$minor" ) + + xenstore_write "$XENBUS_PATH/pid" "$pid" + xenstore_write "$XENBUS_PATH/minor" "$minor" + # dev, as a unix socket, would end up with major:minor 0:0 in + # physical-device if write_dev were used. tapback would be thrown off by + # that incorrect minor, so just skip writing physical-device. + xenstore_write "$XENBUS_PATH/physical-device-path" "$dev" + + success + + release_lock "block" } # Disconnects the device remove() { - find_device - do_or_die tap-ctl destroy -p ${pid} -m ${minor} > /dev/null + local minor + local pid + + claim_lock "block" + + if tap_shared ; then + return + fi + + minor=$( xenstore_read "$XENBUS_PATH/minor" ) + pid=$( xenstore_read "$XENBUS_PATH/pid" ) + + [ -n "$minor" ] || fatal "minor missing" + [ -n "$pid" ] || fatal "pid missing" + do_or_die tap-ctl close -p "$pid" -m "$minor" > /dev/null + do_or_die tap-ctl detach -p "$pid" -m "$minor" > /dev/null + + release_lock "block" } command=$1
@@ -110,6 +246,14 @@ parse_target "$target" check_tools || exit 1 +mode=$( xenstore_read $XENBUS_PATH/mode ) +mode=$( canonicalise_mode $mode ) + +# needed for same_vm +FRONTEND_ID=$(xenstore_read "$XENBUS_PATH/frontend-id") +FRONTEND_UUID=$(xenstore_read_default \ + "/local/domain/$FRONTEND_ID/vm" 'unknown') + case $command in add) add
Signature · 1 line
2.43.0
Re: [PATCH 3/4] hotplug: Update block-tap
Quoted 5 lines · Jason Andryuk
Implement a sharing check like the regular block script.
Checking tapback inside block-tap is too late since it needs to be running to transition the backend to InitWait before block-tap is run.
Quoted 8 lines
tap-ctl check will be removed when the requirement for the blktap kernel driver is removed. Remove it now as it is of limited use.
find_device()needs to be non-fatal allow a sharing check.Only write physical-device-path because that is all that tapback needs. Also
write_devdoesn't handled files and would incorrectly store physical-device as 0:0 which would confuse the minor inside tapback
Missing SOB.
Is block-tap still going to work with the example given in the script header? That is:
"script=block-tap,vdev=xvda,target=<type>:<file>"
Or maybe, that example is already broken?
Quoted 5 lines
---
tools/hotplug/Linux/block-tap +0
-0
tools/hotplug/Linux/block-tap find_device() · line 95
Quoted 10 lines
tools/hotplug/Linux/block-tap +10
-0+count_using() +{ + local file="$1" + local f + + local i=0 + local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE" + for dom in $(xenstore-list "$base_path") + do + for dev in $(xenstore-list "$base_path/$dom")
This function is probably missing "local dom dev".
Quoted 12 lines
tools/hotplug/Linux/block-tap +12
-0+ do + f=$(xenstore_read_default "$base_path/$dom/$dev/params" "") + f=$(echo "$f" | cut -d ":" -f 2) + + if [ -n "$f" ] && [ "$file" = $f ] ; then + i=$(( i + 1 )) + fi + done + done + + echo "$i" +}
tools/hotplug/Linux/block-tap find_device() · line 123
Quoted 10 lines
tools/hotplug/Linux/block-tap +9
-0+ +check_tap_sharing() +{ + local file="$1" + local mode="$2" + local dev + + local base_path="$XENBUS_BASE_PATH/$XENBUS_TYPE" + for dom in $(xenstore-list "$base_path") ; do
Should we add "local dom" to the function?
tools/hotplug/Linux/block-tap find_device() · line 125
Quoted 2 lines
tools/hotplug/Linux/block-tap +2
-0+ for dev in $(xenstore-list "$base_path/$dom") ; do + f=$(xenstore_read_default "$base_path/$dom/$dev/params" "")
Same here, maybe "local f" would be good to have too.
Quoted 9 lines
tools/hotplug/Linux/block-tap +3
-2@@ -89,15 +183,57 @@ find_device() # the device add() { - dev=$(tap-ctl create -a $target) - write_dev $dev + local minor + local pid + local res
tools/hotplug/Linux/block-tap find_device() · line 198
Quoted 10 lines
tools/hotplug/Linux/block-tap +10
-0+ + claim_lock "block" + + if find_device; then + result=$( check_tap_sharing "$file" "$mode" ) + if [ "$result" != "ok" ] ; then + do_ebusy "tap $type file $file in use " "$mode" "${result%% *}" + fi + else + tap_create
The new function tap_create() is doing something similar to the replace tap-ctl create call, right?
Quoted 11 lines
tools/hotplug/Linux/block-tap +6
-2# Disconnects the device remove() { - find_device - do_or_die tap-ctl destroy -p ${pid} -m ${minor} > /dev/null + local minor + local pid + + claim_lock "block" + + if tap_shared ; then
tools/hotplug/Linux/block-tap find_device() · line 234
Quoted 10 lines
tools/hotplug/Linux/block-tap +10
-0+ return + fi + + minor=$( xenstore_read "$XENBUS_PATH/minor" ) + pid=$( xenstore_read "$XENBUS_PATH/pid" ) + + [ -n "$minor" ] || fatal "minor missing" + [ -n "$pid" ] || fatal "pid missing" + do_or_die tap-ctl close -p "$pid" -m "$minor" > /dev/null + do_or_die tap-ctl detach -p "$pid" -m "$minor" > /dev/null
Should we also call tap-ctl free, like tap-ctl destroy seems to do?
tools/hotplug/Linux/block-tap find_device() · line 239
Quoted 5 lines
tools/hotplug/Linux/block-tap +2
-0+ + release_lock "block" } command=$1
Thanks,
Signature · 1 line
Anthony PERARD
PATCH 4/4 libxl: Support blktap with HVM device model
In reply to Jason Andryukblktap exposes disks over UNIX socket Network Block Device (NBD). Modify libxl__device_disk_find_local_path() to provide back the QEMU-formatted NBD path. This allows tapdisk to be used for booting an HVM.
Use the nbd+unix:/// format specified by the protocol at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
--- tools/libs/light/libxl_disk.c | 17 +++++++++++++---- tools/libs/light/libxl_dm.c | 1 - 2 files changed, 13 insertions(+), 5 deletions(-)
tools/libs/light/libxl_disk.c +13 -4
@@ -1317,7 +1317,8 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,* If the format isn't raw and / or we're using a script, then see * if the script has written a path to the "cooked" node */ -if (disk->script && guest_domid !=INVALID_DOMID ) { +if ((disk->script && guest_domid !=INVALID_DOMID ) || + disk->backend ==LIBXL_DISK_BACKEND_TAP ) {libxl__device device;char *be_path, *pdpath;int rc;
@@ -1337,10 +1338,18 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,LOGD (DEBUG , guest_domid,"Attempting to read node %s" , pdpath); path =libxl__xs_read (gc,XBT_NULL , pdpath); -if (path) +if (path) {LOGD (DEBUG , guest_domid,"Accessing cooked block device %s" , path); -else -LOGD (DEBUG , guest_domid,"No physical-device-path, can't access locally." ); + +/* tapdisk exposes disks locally over UNIX socket NBD. */ +if (disk->backend ==LIBXL_DISK_BACKEND_TAP ) { + path =libxl__sprintf (gc,"nbd+unix:///?socket=%s" , path); +LOGD (DEBUG , guest_domid, +"Directly accessing local TAP target %s" , path); + } + }else +LOGD (DEBUG , guest_domid, +"No physical-device-path, can't access locally." ); goto out; }
tools/libs/light/libxl_dm.c +0 -1
@@ -1915,7 +1915,6 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,continue ; } -assert (disks[i].backend !=LIBXL_DISK_BACKEND_TAP ); target_path =libxl__device_disk_find_local_path (gc, guest_domid, &disks[i], true);
Signature · 1 line
2.43.0
Re: [PATCH 4/4] libxl: Support blktap with HVM device model
Quoted 10 lines · Jason Andryuk
tools/libs/light/libxl_disk.c +1
-3@@ -1337,10 +1338,18 @@ char *libxl__device_disk_find_local_path(libxl__gc *gc,LOGD (DEBUG , guest_domid,"Attempting to read node %s" , pdpath); path =libxl__xs_read (gc,XBT_NULL , pdpath); -if (path) +if (path) {LOGD (DEBUG , guest_domid,"Accessing cooked block device %s" , path); -else -LOGD (DEBUG , guest_domid,"No physical-device-path, can't access locally." );
tools/libs/light/libxl_disk.c char *libxl__device_disk_find_local_path(libxl__gc *gc, · line 1352
Quoted 10 lines
tools/libs/light/libxl_disk.c +10
-0+ +/* tapdisk exposes disks locally over UNIX socket NBD. */ +if (disk->backend ==LIBXL_DISK_BACKEND_TAP ) { + path =libxl__sprintf (gc,"nbd+unix:///?socket=%s" , path); +LOGD (DEBUG , guest_domid, +"Directly accessing local TAP target %s" , path); + } + }else +LOGD (DEBUG , guest_domid, +"No physical-device-path, can't access locally." );
Here, the coding style call to have both side of the if..else to have { } or none of them. Could you add {}-block for the else side?
Beside that, patch looks fine to me:
Thanks,
Signature · 1 line
Anthony PERARD
Quoted 5 lines · Jason Andryuk
Jason Andryuk (4):
block-common: Fixsame_vmfor no targets
libxl: Add support for blktap vbd3
hotplug: Update block-tap
libxl: Support blktap with HVM device model
If libxl is learning how to drive tapdisk, then it's time for:
https://github.com/xenserver/xen.pg/blob/XS-8/patches/xentop-vbd3.patch
to make it's way upstream. Otherwise, there's an absence of disk info in xentop.
I previously kept it back because vbd3 was specific to XenServer.
~Andrew
Quoted 28 lines · Jason Andryuk
This patch re-introduces blktap support to libxl. Unlike earlier versions, it does not link against any blktap library. libxl changes are needed to write to the vbd3 backend XenStore nodes.
blktap has three components. tapdisk is a daemon implementing the disk IO, NBD (Network Block Device), and Xen PV interfaces. tap-ctl is a tool to control tapdisks - creating, starting, stopping and freeing. tapback manages the XenStore operations and instructs tapdisk to connect.
It is notable that tapdisk performs the grant and event channel ops, but doesn't interact with XenStore. tapback performs XenStore operations and notifies tapdisks of values and changes.
The flow is: libxl writes to the "vbd3" XenStore nodes and runs the block-tap script. The block-tap script runs tap-ctl to create a tapdisk instance as the physical device. tapback then sees the tapdisk and instructs the tapdisk to connect up the PV blkif interface.
This is expected to work without the kernel blktap driver, so the block-tap script is modified accordingly to write the UNIX NBD path. (It works with the kernel blktap driver as well - upstream blktap hasn't removed the blktap driver requirement yet -
https://github.com/xapi-project/blktap/pull/364)An example command to attach a vhd:
xl block-attach vm 'vdev=xvdf,backendtype=tap,format=vhd,target=/srv/target.vhd'
Quoted 6 lines
VHD support is important for OpenXT since there are lots of existing VHDs which still need supporting. tapdisk also supports encrypted VHDs which is not available in QEMU.
blktap's tapback needs minimal changes to work with libxl:
https://github.com/xapi-project/blktap/pull/394
It might be nice to have an entry in the ChangeLog. But it's looks like the blktap project isn't ready yet, so we might need to point that out.
Cheers,
Signature · 1 line
Anthony PERARD