Introduce a driver to support host accelerated access to Microsoft Azure Blob
v4 cover Introduce a driver to support host accelerated access to Microsoft Azure Blob
From: Long Li <[email protected]>
Microsoft Azure Blob storage service exposes a REST API to applications for data access.
(https://docs.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api)
This patchset implements a VSC (Virtualization Service Consumer) that communicates with a VSP (Virtualization Service Provider) on the Hyper-V host to execute Blob storage access via native network stack on the host. This VSC doesn't implement the semantics of REST API. Those are implemented in user-space. The VSC provides a fast data path to VSP.
Answers to some previous questions discussing the driver:
Q: Why this driver doesn't use the block layer
A: The Azure Blob is based on a model of object oriented storage. The storage object is not modeled in block sectors. While it's possible to present the storage object as a block device (assuming it makes sense to fake all the block device attributes), we lose the ability to express functionality that are defined in the REST API.
Q: You just lost all use of caching and io_uring and loads of other kernel infrastructure that has been developed and relied on for decades?
A: The REST API is not designed to have caching at system level. This driver doesn't attempt to improve on this. There are discussions on supporting ioctl() on io_uring (https://lwn.net/Articles/844875/), that will benefit this driver. The block I/O scheduling is not helpful in this case, as the Blob application and Blob storage server have complete knowledge on the I/O pattern based on storage object type. This knowledge doesn't get easily consumed by the block layer.
Q: You also just abandoned the POSIX model and forced people to use a random-custom-library just to access their storage devices, breaking all existing programs in the world?
A: The existing Blob applications access storage via HTTP (REST API). They don't use POSIX interface. The interface for Azure Blob is not designed on POSIX.
Q: What programs today use this new API?
A: Currently none is released. But per above, there are also none using POSIX.
Q: Where is the API published and what ensures that it will remain stable?
A: Cloud based REST protocols have similar considerations to the kernel in terms of interface stability. Applications depend on cloud services via REST in much the same way as they depend on kernel services. Because applications can consume cloud APIs over the Internet, there is no opportunity to recompile applications to ensure compatibility. This causes the underlying APIs to be exceptionally stable, and Azure Blob has not removed support for an exposed API to date. This driver is supporting a pass-through model where requests in a guest process can be reflected to a VM host environment. Like the current REST interface, the goal is to ensure that each host provide a high degree of compatibility with each guest, but that task is largely outside the scope of this driver, which exists to communicate requests in the same way an HTTP stack would. Just like an HTTP stack does not require updates to add a new custom header or receive one from a server, this driver does not require updates for new functionality so long as the high level request/response model is retained.
Q: What happens when it changes over time, do we have to rebuild all userspace applications?
A: No. We don’t rebuild them all to talk HTTP either. In the current HTTP scheme, applications specify the version of the protocol they talk, and the storage backend responds with that version.
Q: What happens to the kernel code over time, how do you handle changes to the API there?
A: The goal of this driver is to get requests to the Hyper-V host, so the kernel isn’t involved in API changes, in the same way that HTTP implementations are robust to extra functionality being added to HTTP.
Long Li (3):
Drivers: hv: vmbus: add support to ignore certain PCIE devices
Drivers: hv: add Azure Blob driver
Drivers: hv: Add to maintainer for Azure Blob driver
Changes:
v2:
Refactored the code in vmbus to scan devices
Reworked Azure Blob driver and moved user-mode interfaces to uapi
v3:
Changed licensing language
Patch format passed "checkpatch --strict"
debugfs and logging, module parameter cleanup
General code clean up
Fix device removal race conditions
v4:
addressed licencing issues
changed to dynamic device model
Long Li (3):
Drivers: hv: vmbus: add support to ignore certain PCIE devices
Drivers: hv: add Azure Blob driver
Drivers: hv: Add to maintainer for Hyper-V/Azure drivers
.../userspace-api/ioctl/ioctl-number.rst | 2 + MAINTAINERS | 2 + drivers/hv/Kconfig | 11 + drivers/hv/Makefile | 1 + drivers/hv/channel_mgmt.c | 55 +- drivers/hv/hv_azure_blob.c | 628 ++++++++++++++++++ include/linux/hyperv.h | 9 + include/uapi/misc/hv_azure_blob.h | 34 + 8 files changed, 736 insertions(+), 6 deletions(-) create mode 100644 drivers/hv/hv_azure_blob.c create mode 100644 include/uapi/misc/hv_azure_blob.h
Signature · 1 line
2.25.1
PATCH v4 1/3 Drivers: hv: vmbus: add support to ignore certain PCIE devices
From: Long Li <[email protected]>
Under certain configurations when Hyper-v presents a device to VMBUS, it may have a VMBUS channel and a PCIe channel. The PCIe channel is not used in Linux and does not have a backing PCIE device on Hyper-v. For such devices, ignore those PCIe channels so they are not getting probed by the PCI subsystem.
--- drivers/hv/channel_mgmt.c | 48 ++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 6 deletions(-)
drivers/hv/channel_mgmt.c +42 -6
@@ -26,6 +26,21 @@static void init_vp_index (struct vmbus_channel *channel); +/* +* For some VMBUS devices, Hyper-v also presents certain PCIE devices as +* part of the host device implementation. Those devices have no real +* PCI implementation in Hyper-V, and should be ignored and not presented +* to the PCI layer. +*/ +static const guid_t vpci_ignore_instances[] = { +/* +* Azure Blob instance ID in VPCI +* {d4573da2-2caa-4711-a8f9-bbabf4ee9685} +*/ +GUID_INIT (0xd4573da2 ,0x2caa ,0x4711 ,0xa8 ,0xf9 , +0xbb ,0xab ,0xf4 ,0xee ,0x96 ,0x85 ), +}; +const struct vmbus_device vmbus_devs[] = {/* IDE */ { .dev_type =HV_IDE ,
@@ -187,20 +202,19 @@ static bool is_unsupported_vmbus_devs(const guid_t *guid)return false; } -static u16 hv_get_dev_type (const struct vmbus_channel *channel) +static u16 hv_get_dev_type (const guid_t *guid) { -const guid_t *guid = &channel->offermsg.offer.if_type;u16 i; -if (is_hvsock_channel (channel) ||is_unsupported_vmbus_devs (guid)) +if (is_unsupported_vmbus_devs (guid))return HV_UNKNOWN ;for (i =HV_IDE ; i <HV_UNKNOWN ; i++) {if (guid_equal (guid, &vmbus_devs[i].guid)) -return i; +return vmbus_devs[i].dev_type; }pr_info ("Unknown GUID: %pUl\n" , guid); -return i; +return HV_UNKNOWN ; }/**
@@ -487,6 +501,16 @@ void vmbus_free_channels(void) } } +static bool ignore_pcie_device (guid_t *if_instance) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (vpci_ignore_instances); i++) +if (guid_equal (&vpci_ignore_instances[i], if_instance)) +return true; +return false; +} +/* Note: the function can run concurrently for primary/sub channels. */ static void vmbus_add_channel_work (struct work_struct *work) {
@@ -910,7 +934,11 @@ static void vmbus_setup_channel_state(struct vmbus_channel *channel, sizeof(struct vmbus_channel_offer_channel)); channel->monitor_grp = (u8 )offer->monitorid /32 ; channel->monitor_bit = (u8 )offer->monitorid %32 ; - channel->device_id =hv_get_dev_type (channel); +if (is_hvsock_channel (channel)) + channel->device_id =HV_UNKNOWN ; +else + channel->device_id = +hv_get_dev_type (&channel->offermsg.offer.if_type); }/*
@@ -972,6 +1000,14 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr)trace_vmbus_onoffer (offer); +/* Check to see if we should ignore this PCIe channel */ +if (hv_get_dev_type (&offer->offer.if_type) ==HV_PCIE && +ignore_pcie_device (&offer->offer.if_instance)) { +pr_debug ("Ignore instance %pUl over VPCI\n" , + &offer->offer.if_instance); +return ; + } +if (!vmbus_is_valid_device (&offer->offer.if_type)) {pr_err_ratelimited ("Invalid offer %d from the host supporting isolation\n" , offer->child_relid);
Signature · 1 line
2.25.1
PATCH v4 2/3 Drivers: hv: add Azure Blob driver
In reply to [email protected]From: Long Li <[email protected]>
Azure Blob provides scalable, secure and shared storage services for Azure.
This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and direct data link for storage server access.
--- .../userspace-api/ioctl/ioctl-number.rst | 2 + drivers/hv/Kconfig | 11 + drivers/hv/Makefile | 1 + drivers/hv/channel_mgmt.c | 7 + drivers/hv/hv_azure_blob.c | 628 ++++++++++++++++++ include/linux/hyperv.h | 9 + include/uapi/misc/hv_azure_blob.h | 34 + 7 files changed, 692 insertions(+) create mode 100644 drivers/hv/hv_azure_blob.c create mode 100644 include/uapi/misc/hv_azure_blob.h
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
@@ -180,6 +180,8 @@ Code Seq# Include File Comments 'R' 01 linux/rfkill.h conflict! 'R' C0-DF net/bluetooth/rfcomm.h 'R' E0 uapi/linux/fsl_mc.h +'R' F0-FF uapi/misc/hv_azure_blob.h Microsoft Azure Blob driver + <mailto:[email protected]> 'S' all linux/cdrom.h conflict! 'S' 80-81 scsi/scsi_ioctl.h conflict! 'S' 82-FF scsi/scsi.h conflict!
drivers/hv/Kconfig +11 -0
@@ -27,4 +27,15 @@ config HYPERV_BALLOON help Select this option to enable Hyper-V Balloon driver. +config HYPERV_AZURE_BLOB + tristate "Microsoft Azure Blob driver" + depends on HYPERV && X86_64 + help + Select this option to enable Microsoft Azure Blob driver. + + This driver implements a fast datapath over Hyper-V to support + accelerated access to Microsoft Azure Blob services. + To compile this driver as a module, choose M here. The module will be + called azure_blob. + endmenu
drivers/hv/Makefile +1 -0
@@ -2,6 +2,7 @@ obj-$(CONFIG_HYPERV) += hv_vmbus.o obj-$(CONFIG_HYPERV_UTILS) += hv_utils.o obj-$(CONFIG_HYPERV_BALLOON) += hv_balloon.o +obj-$(CONFIG_HYPERV_AZURE_BLOB) += hv_azure_blob.o CFLAGS_hv_trace.o = -I$(src) CFLAGS_hv_balloon.o = -I$(src)
drivers/hv/channel_mgmt.c +7 -0
@@ -154,6 +154,13 @@ const struct vmbus_device vmbus_devs[] = { .allowed_in_isolated = false, }, +/* Azure Blob */ + { .dev_type = HV_AZURE_BLOB, +HV_AZURE_BLOB_GUID , + .perf_device = false, + .allowed_in_isolated = false, + }, + /* Unknown GUID */ { .dev_type = HV_UNKNOWN, .perf_device = false,
drivers/hv/hv_azure_blob.c new file +628 -0
628 lines not on this page · raw message
include/linux/hyperv.h +9 -0
@@ -772,6 +772,7 @@ enum vmbus_device_type {HV_FCOPY ,HV_BACKUP ,HV_DM , +HV_AZURE_BLOB , HV_UNKNOWN, };
@@ -1349,6 +1350,14 @@ void vmbus_free_mmio(resource_size_t start, resource_size_t size); .guid = GUID_INIT(0xba6163d9, 0x04a1, 0x4d29, 0xb6, 0x05, \0x72 ,0xe2 ,0xff ,0xb1 ,0xdc ,0x7f ) +/* +* Azure Blob GUID +* {0590b792-db64-45cc-81db-b8d70c577c9e} + */ +#define HV_AZURE_BLOB_GUID \ + .guid = GUID_INIT(0x0590b792, 0xdb64, 0x45cc, 0x81, 0xdb, \ + 0xb8, 0xd7, 0x0c, 0x57, 0x7c, 0x9e) +/* * Shutdown GUID * {0e0b6031-5213-4934-818b-38d90ced39db}
include/uapi/misc/hv_azure_blob.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* Copyright (c) 2021 Microsoft Corporation. */ + +#ifndef _AZ_BLOB_H +#define _AZ_BLOB_H + +#include <linux/kernel.h> +#include <linux/uuid.h> + +/* user-mode sync request sent through ioctl */ +struct az_blob_request_sync_response { +__u32 status; +__u32 response_len; +}; + +struct az_blob_request_sync { +guid_t guid; +__u32 timeout; +__u32 request_len; +__u32 response_len; +__u32 data_len; +__u32 data_valid; +__aligned_u64 request_buffer; +__aligned_u64 response_buffer; +__aligned_u64 data_buffer; +struct az_blob_request_sync_response response; +}; + +#define AZ_BLOB_MAGIC_NUMBER 'R' +#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \ + _IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \ + struct az_blob_request_sync) + +#endif /* define _AZ_BLOB_H */
Signature · 1 line
2.25.1
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 7 lines · On Mon, Jul 19, 2021 at 08:31:05PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
Azure Blob provides scalable, secure and shared storage services for Azure.
This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and direct data link for storage server access.
Thank you for fixing up the license info, but you did not include any information here about how to use this, where the userspace tools are, nor why you are going around the existing kernel block layer and proof that going around the block layer is a good idea.
thanks,
greg k-h
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
In reply to [email protected]Quoted 27 lines · On Mon, Jul 19, 2021 at 08:31:05PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
Azure Blob provides scalable, secure and shared storage services for Azure.
This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and direct data link for storage server access.
Cc Jonathan Corbet <[email protected]>Cc "K. Y. Srinivasan" <[email protected]>Cc Haiyang Zhang <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Wei Liu <[email protected]>Cc Dexuan Cui <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Bjorn Andersson <[email protected]>Cc Hans de Goede <[email protected]>Cc Dan Williams <[email protected]>Cc Maximilian Luz <[email protected]>Cc Mike Rapoport <[email protected]>Cc Ben Widawsky <[email protected]>Cc Jiri Slaby <[email protected]>Cc Andra Paraschiv <[email protected]>Cc Siddharth Gupta <[email protected]>Cc Hannes Reinecke <[email protected]>Signed-off-by Long Li <[email protected]>
Quoted 12 lines
--- .../userspace-api/ioctl/ioctl-number.rst | 2 + drivers/hv/Kconfig | 11 + drivers/hv/Makefile | 1 + drivers/hv/channel_mgmt.c | 7 + drivers/hv/hv_azure_blob.c | 628 ++++++++++++++++++ include/linux/hyperv.h | 9 + include/uapi/misc/hv_azure_blob.h | 34 + 7 files changed, 692 insertions(+) create mode 100644 drivers/hv/hv_azure_blob.c create mode 100644 include/uapi/misc/hv_azure_blob.h
Hi,
This is the friendly patch-bot of Greg Kroah-Hartman. You have sent him a patch that has triggered this response. He used to manually respond to these common problems, but in order to save his sanity (he kept writing the same thing over and over, yet to different people), I was created. Hopefully you will not take offence and will fix the problem in your patch and resubmit it so that it can be accepted into the Linux kernel tree.
You are receiving this message because of the following common error(s) as indicated below:
- This looks like a new version of a previously submitted patch, but you did not list below the --- line any changes from the previous version. Please read the section entitled "The canonical patch format" in the kernel file, Documentation/SubmittingPatches for what needs to be done here to properly describe this.
If you wish to discuss this problem further, or you have questions about how to resolve this issue, please feel free to respond to this email and Greg will reply once he has dug out from the pending patches received from other developers.
thanks,
greg k-h's patch email bot
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
In reply to [email protected]drivers/hv/hv_azure_blob.c line 24
Quoted 10 lines · On Mon, Jul 19, 2021 at 08:31:05PM -0700, [email protected] wrote:
drivers/hv/hv_azure_blob.c new file +10
-0+struct az_blob_device { +struct hv_device *device; + +/* Opened files maintained by this device */ +struct list_head file_list; +/* Lock for protecting file_list */ +spinlock_t file_lock; + +/* The refcount for this device */ +refcount_t count;
Just use a kref please if you really need this. Are you sure you do? You already have 2 other reference counted objects being used here, why make it 3?
drivers/hv/hv_azure_blob.c line 27
Quoted 2 lines
drivers/hv/hv_azure_blob.c new file +2
-0+/* Pending requests to VSP */ +atomic_t pending;
Why does this need to be atomic?
drivers/hv/hv_azure_blob.c line 30
Quoted 3 lines
drivers/hv/hv_azure_blob.c new file +3
-0+wait_queue_head_t waiting_to_drain; + +bool removing;
Are you sure this actually works properly? Why is it needed vs. any other misc device?
Quoted 4 lines
drivers/hv/hv_azure_blob.c new file +4
-0+/* VSC->VSP request */ +struct az_blob_vsp_request { +u32 version; +u32 timeout_ms;
drivers/hv/hv_azure_blob.c line 54
Quoted 10 lines
drivers/hv/hv_azure_blob.c new file +10
-0+u32 data_buffer_offset; +u32 data_buffer_length; +u32 data_buffer_valid; +u32 operation_type; +u32 request_buffer_offset; +u32 request_buffer_length; +u32 response_buffer_offset; +u32 response_buffer_length; +guid_t transaction_id; +} __packed;
Why packed? If this is going across the wire somewhere, you need to specify the endian-ness of these values, right? If this is not going across the wire, no need for it to be packed.
drivers/hv/hv_azure_blob.c line 61
Quoted 7 lines
drivers/hv/hv_azure_blob.c new file +7
-0+ +/* VSP->VSC response */ +struct az_blob_vsp_response { +u32 length; +u32 error; +u32 response_len; +} __packed;
Same here.
Quoted 7 lines
drivers/hv/hv_azure_blob.c new file +7
-0+ +struct az_blob_vsp_request_ctx { +struct list_head list; +struct completion wait_vsp; +struct az_blob_request_sync *request; +}; +
drivers/hv/hv_azure_blob.c line 78
Quoted 10 lines
drivers/hv/hv_azure_blob.c new file +10
-0+struct az_blob_file_ctx { +struct list_head list; + +/* List of pending requests to VSP */ +struct list_head vsp_pending_requests; +/* Lock for protecting vsp_pending_requests */ +spinlock_t vsp_pending_lock; +wait_queue_head_t wait_vsp_pending; + +pid_t pid;
Why do you need a pid? What namespace is this pid in?
Quoted 21 lines
drivers/hv/hv_azure_blob.c new file +21
-0+static int az_blob_probe (struct hv_device *device, +const struct hv_vmbus_device_id *dev_id) +{ +int ret; +struct az_blob_device *dev; + + dev =kzalloc (sizeof (*dev),GFP_KERNEL ); +if (!dev) +return -ENOMEM ; + +spin_lock_init (&dev->file_lock); +INIT_LIST_HEAD (&dev->file_list); +atomic_set (&dev->pending,0 ); +init_waitqueue_head (&dev->waiting_to_drain); + + ret =az_blob_connect_to_vsp (device, dev,AZ_BLOB_RING_SIZE ); +if (ret) + goto fail; + +refcount_set (&dev->count,1 ); + az_blob_dev = dev;
drivers/hv/hv_azure_blob.c line 574
Quoted 10 lines
drivers/hv/hv_azure_blob.c new file +10
-0+ +// create user-mode client library facing device + ret =az_blob_create_device (dev); +if (ret) { +dev_err (AZ_DEV ,"failed to create device ret=%d\n" , ret); +az_blob_remove_vmbus (device); + goto fail; + } + +dev_info (AZ_DEV ,"successfully probed device\n" );
When drivers are working properly, they should be quiet.
And what is with the AZ_DEV macro mess?
And can you handle more than one device in the system at one time? I think your debugfs logic will get really confused.
thanks,
greg k-h
RE: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 17 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
drivers/hv/hv_azure_blob.c line 24
Quoted 10 lines · On Mon, Jul 19, 2021 at 08:31:05PM -0700, [email protected] wrote:
drivers/hv/hv_azure_blob.c new file +10
-0+struct az_blob_device { +struct hv_device *device; + +/* Opened files maintained by this device */ +struct list_head file_list; +/* Lock for protecting file_list */ +spinlock_t file_lock; + +/* The refcount for this device */ +refcount_t count;Just use a kref please if you really need this. Are you sure you do? You already have 2 other reference counted objects being used here, why make it 3?
The "count" is to keep track how many user-mode instances and vmbus instance are opened on this device. Being a VMBUS device, this device can be removed at any time (host servicing etc). We must remove the device when this happens even if the device is still opened by some user-mode program. The "count" will guarantee the lifecycle of the device object after all user-mode has released the device.
I looked at using "file_list" (it's used for tracking opened files by user-mode) for this purpose, but I found out I still need to manage the device count at the vmbus side.
Quoted 5 lines
drivers/hv/hv_azure_blob.c line 27
Quoted 3 lines
drivers/hv/hv_azure_blob.c new file +2
-0+/* Pending requests to VSP */ +atomic_t pending;Why does this need to be atomic?
"pending' is per-device maintained that could change when multiple-user access the device at the same time.
Quoted 8 lines
drivers/hv/hv_azure_blob.c line 30
Quoted 5 lines
drivers/hv/hv_azure_blob.c new file +3
-0+wait_queue_head_t waiting_to_drain; + +bool removing;Are you sure this actually works properly? Why is it needed vs. any other misc device?
When removing this device from vmbus, we need to guarantee there is no possible packets to vmbus. This is a requirement before calling vmbus_close(). Other drivers of vmbus follows the same procedure.
The reason why this driver needs this is that the device removal can happen in the middle of az_blob_ioctl_user_request(), which can send packet over vmbus.
Quoted 12 lines
Quoted 12 lines
drivers/hv/hv_azure_blob.c new file +10
-0+/* VSC->VSP request */ +struct az_blob_vsp_request { +u32 version; +u32 timeout_ms; +u32 data_buffer_offset; +u32 data_buffer_length; +u32 data_buffer_valid; +u32 operation_type; +u32 request_buffer_offset; +u32 request_buffer_length;
Quoted 8 lines
drivers/hv/hv_azure_blob.c line 54
Quoted 4 lines
drivers/hv/hv_azure_blob.c new file +4
-0+u32 response_buffer_offset; +u32 response_buffer_length; +guid_t transaction_id; +} __packed;Why packed? If this is going across the wire somewhere, you need to specify the endian-ness of these values, right? If this is not going across the wire, no need for it to be packed.
Those data go through the wire.
All data structures specified in the Hyper-V and guest VM use Little Endian byte ordering. All HV core drivers have a dependence on X86, that guarantees this ordering.
Quoted 21 lines
drivers/hv/hv_azure_blob.c line 61
Quoted 8 lines
drivers/hv/hv_azure_blob.c new file +7
-0+ +/* VSP->VSC response */ +struct az_blob_vsp_response { +u32 length; +u32 error; +u32 response_len; +} __packed;Same here.
Quoted 10 lines
drivers/hv/hv_azure_blob.c new file +10
-0+ +struct az_blob_vsp_request_ctx { +struct list_head list; +struct completion wait_vsp; +struct az_blob_request_sync *request; }; + +struct az_blob_file_ctx { +struct list_head list; + +/* List of pending requests to VSP */
Quoted 8 lines
drivers/hv/hv_azure_blob.c line 78
Quoted 6 lines
drivers/hv/hv_azure_blob.c new file +6
-0+struct list_head vsp_pending_requests; +/* Lock for protecting vsp_pending_requests */ +spinlock_t vsp_pending_lock; +wait_queue_head_t wait_vsp_pending; + +pid_t pid;Why do you need a pid? What namespace is this pid in?
It's a request from user library team for production troubleshooting purposes. It's exposed as informal in debugfs.
Quoted 25 lines
Quoted 25 lines
drivers/hv/hv_azure_blob.c new file +24
-0+static int az_blob_probe (struct hv_device *device, +const struct hv_vmbus_device_id *dev_id) { +int ret; +struct az_blob_device *dev; + + dev =kzalloc (sizeof (*dev),GFP_KERNEL ); +if (!dev) +return -ENOMEM ; + +spin_lock_init (&dev->file_lock); +INIT_LIST_HEAD (&dev->file_list); +atomic_set (&dev->pending,0 ); +init_waitqueue_head (&dev->waiting_to_drain); + + ret =az_blob_connect_to_vsp (device, dev,AZ_BLOB_RING_SIZE ); +if (ret) + goto fail; + +refcount_set (&dev->count,1 ); + az_blob_dev = dev; + +// create user-mode client library facing device + ret =az_blob_create_device (dev); +if (ret) {
Quoted 8 lines
drivers/hv/hv_azure_blob.c line 574
Quoted 6 lines
drivers/hv/hv_azure_blob.c new file +6
-0+dev_err (AZ_DEV ,"failed to create device ret=%d\n" , ret); +az_blob_remove_vmbus (device); + goto fail; + } + +dev_info (AZ_DEV ,"successfully probed device\n" );When drivers are working properly, they should be quiet.
The reason is that in production environment when dealing with custom support cases, there is no good way to check if the channel is opened on the device. Having this message will greatly clear confusions on possible mis-configurations.
Quoted 2 lines
And what is with the
AZ_DEVmacro mess?
It's not required, it's just for saving code length. I can put "&az_blob_dev->device->device" in every dev_err(), but it makes the code look a lot longer.
Quoted 3 lines
And can you handle more than one device in the system at one time? I think your debugfs logic will get really confused.
There can be one device object active in the system at any given time. The debugfs grabs the current active device object. If the device is being removed, removed or added, the current active device object is updated accordingly.
Quoted 4 lines
thanks,
greg k-h
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 18 lines · Long Li
Quoted 17 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
drivers/hv/hv_azure_blob.c line 24
Quoted 10 lines · On Mon, Jul 19, 2021 at 08:31:05PM -0700, [email protected] wrote:
drivers/hv/hv_azure_blob.c new file +10
-0+struct az_blob_device { +struct hv_device *device; + +/* Opened files maintained by this device */ +struct list_head file_list; +/* Lock for protecting file_list */ +spinlock_t file_lock; + +/* The refcount for this device */ +refcount_t count;Just use a kref please if you really need this. Are you sure you do? You already have 2 other reference counted objects being used here, why make it 3?
Quoted 2 lines
The "count" is to keep track how many user-mode instances and vmbus instance are opened on this device.
That will not work at all, sorry. Please NEVER try to count "open" calls to a driver, as the driver will never be told how many userspace programs are accessing it at any time, nor should it even ever care.
Use the existing apis, there is no need to attempt to count this as you will always get it wrong (hint, what happens when processes share file descriptors...)
Quoted 4 lines
Being a VMBUS device, this device can be removed
at any time (host servicing etc). We must remove the device when this happens even if the device is still opened by some user-mode program. The "count" will guarantee the lifecycle of the device object after all user-mode has released the device.
No it will not, just use the existing apis and all will be fine.
Quoted 2 lines
I looked at using "
file_list" (it's used for tracking opened files by user-mode) for this purpose, but I found out I still need to manage the device count at the vmbus side.
Again, you should not need this. As proof, no other driver needs this...
Quoted 7 lines
Quoted 4 lines
drivers/hv/hv_azure_blob.c line 27
Quoted 2 lines
drivers/hv/hv_azure_blob.c new file +2
-0+/* Pending requests to VSP */ +atomic_t pending;Why does this need to be atomic?
"pending' is per-device maintained that could change when multiple-user access the device at the same time.
How do you know this, and why does it matter?
Quoted 9 lines
Quoted 6 lines
drivers/hv/hv_azure_blob.c line 30
Quoted 3 lines
drivers/hv/hv_azure_blob.c new file +3
-0+wait_queue_head_t waiting_to_drain; + +bool removing;Are you sure this actually works properly? Why is it needed vs. any other misc device?
When removing this device from vmbus, we need to guarantee there is no possible packets to vmbus.
Why? Shouldn't the vmbus api handle this for you automatically? Why is this driver unique here?
Quoted 2 lines
This is a requirement before calling
vmbus_close(). Other drivers of vmbus follows the same procedure.
Ah. Why not fix this in the vmbus core? That sounds like extra logic that everyone would have to duplicate for no good reason.
Quoted 2 lines
The reason why this driver needs this is that the device removal can happen in the middle of
az_blob_ioctl_user_request(), which can send packet over vmbus.
Sure, but the bus should handle that for you.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
drivers/hv/hv_azure_blob.c new file +10
-0+/* VSC->VSP request */ +struct az_blob_vsp_request { +u32 version; +u32 timeout_ms; +u32 data_buffer_offset; +u32 data_buffer_length; +u32 data_buffer_valid; +u32 operation_type; +u32 request_buffer_offset; +u32 request_buffer_length;Quoted 8 lines
drivers/hv/hv_azure_blob.c line 54
Quoted 4 lines
drivers/hv/hv_azure_blob.c new file +4
-0+u32 response_buffer_offset; +u32 response_buffer_length; +guid_t transaction_id; +} __packed;Why packed? If this is going across the wire somewhere, you need to specify the endian-ness of these values, right? If this is not going across the wire, no need for it to be packed.
Quoted 5 lines
Those data go through the wire.
All data structures specified in the Hyper-V and guest VM use Little Endian byte ordering. All HV core drivers have a dependence on X86, that guarantees this ordering.
Then specify little endian please.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/hv/hv_azure_blob.c new file +7
-0+struct completion wait_vsp; +struct az_blob_request_sync *request; }; + +struct az_blob_file_ctx { +struct list_head list; + +/* List of pending requests to VSP */ Quoted 8 lines
drivers/hv/hv_azure_blob.c line 78
Quoted 6 lines
drivers/hv/hv_azure_blob.c new file +6
-0+struct list_head vsp_pending_requests; +/* Lock for protecting vsp_pending_requests */ +spinlock_t vsp_pending_lock; +wait_queue_head_t wait_vsp_pending; + +pid_t pid;Why do you need a pid? What namespace is this pid in?
Quoted 2 lines
It's a request from user library team for production troubleshooting purposes. It's exposed as informal in debugfs.
Then it will be wrong. Put all of your "debugging" stuff into one place so it is obvious what it is for, and so that people can ignore it.
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
drivers/hv/hv_azure_blob.c new file +24
-0+static int az_blob_probe (struct hv_device *device, +const struct hv_vmbus_device_id *dev_id) { +int ret; +struct az_blob_device *dev; + + dev =kzalloc (sizeof (*dev),GFP_KERNEL ); +if (!dev) +return -ENOMEM ; + +spin_lock_init (&dev->file_lock); +INIT_LIST_HEAD (&dev->file_list); +atomic_set (&dev->pending,0 ); +init_waitqueue_head (&dev->waiting_to_drain); + + ret =az_blob_connect_to_vsp (device, dev,AZ_BLOB_RING_SIZE ); +if (ret) + goto fail; + +refcount_set (&dev->count,1 ); + az_blob_dev = dev; + +// create user-mode client library facing device + ret =az_blob_create_device (dev); +if (ret) {Quoted 8 lines
drivers/hv/hv_azure_blob.c line 574
Quoted 6 lines
drivers/hv/hv_azure_blob.c new file +6
-0+dev_err (AZ_DEV ,"failed to create device ret=%d\n" , ret); +az_blob_remove_vmbus (device); + goto fail; + } + +dev_info (AZ_DEV ,"successfully probed device\n" );When drivers are working properly, they should be quiet.
Quoted 3 lines
The reason is that in production environment when dealing with custom support cases, there is no good way to check if the channel is opened on the device. Having this message will greatly clear confusions on possible mis-configurations.
Again, no, the driver should be quiet. If you REALLY need this type of thing, the bus should be the one doing that type of notification for all devices on the bus. Do not make this a per-driver choice.
Quoted 4 lines
Quoted 1 line
And what is with the
AZ_DEVmacro mess?It's not required, it's just for saving code length. I can put "
&az_blob_dev->device->device" in everydev_err(), but it makes the code look a lot longer.
Then perhaps your structures are not correct? Please spell it out so that we can see that your implementation needs fixing.
Quoted 6 lines
Quoted 2 lines
And can you handle more than one device in the system at one time? I think your debugfs logic will get really confused.
There can be one device object active in the system at any given time. The debugfs grabs the current active device object. If the device is being removed, removed or added, the current active device object is updated accordingly.
If I remember, your debugfs initialization seems to not use the device name, but rather a "driver" name, which implies that this is not going to work well with multiple devices. But I could be wrong.
thanks,
greg k-h
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
In reply to [email protected]include/uapi/misc/hv_azure_blob.h line 8
Quoted 11 lines · On 20. 07. 21, 5:31, [email protected] wrote:
include/uapi/misc/hv_azure_blob.h new file +8
-0@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* Copyright (c) 2021 Microsoft Corporation. */ + +#ifndef _AZ_BLOB_H +#define _AZ_BLOB_H + +#include <linux/kernel.h> +#include <linux/uuid.h>
Quoting from https://lore.kernel.org/linux-doc/MWHPR21MB159375586D810EC5DCB66AF0D7039@MWHPR21MB1593.namprd21.prod.outlook.com/:
=====
Seems like a #include of asm/ioctl.h (or something similar)
is needed so that _IOWR is defined. Also, a #include
is needed for __u32, __aligned_u64, guid_t, etc.
=====
Why was no include added?
Quoted 17 lines
include/uapi/misc/hv_azure_blob.h new file +16
-0+ +/* user-mode sync request sent through ioctl */ +struct az_blob_request_sync_response { +__u32 status; +__u32 response_len; +}; + +struct az_blob_request_sync { +guid_t guid; +__u32 timeout; +__u32 request_len; +__u32 response_len; +__u32 data_len; +__u32 data_valid; +__aligned_u64 request_buffer; +__aligned_u64 response_buffer;
include/uapi/misc/hv_azure_blob.h line 34
Quoted 10 lines
include/uapi/misc/hv_azure_blob.h new file +10
-0+__aligned_u64 data_buffer; + struct az_blob_request_sync_response response; +}; + +#define AZ_BLOB_MAGIC_NUMBER 'R' +#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \ + _IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \ + struct az_blob_request_sync) + +#endif /* define _AZ_BLOB_H */
thanks,
Signature · 2 lines
js suse labs
RE: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 22 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
include/uapi/misc/hv_azure_blob.h line 8
Quoted 11 lines · On 20. 07. 21, 5:31, [email protected] wrote:
include/uapi/misc/hv_azure_blob.h new file +8
-0@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* Copyright (c) 2021 Microsoft Corporation. */ + +#ifndef _AZ_BLOB_H +#define _AZ_BLOB_H + +#include <linux/kernel.h> +#include <linux/uuid.h> Quoting from
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Flinux- doc%2FMWHPR21MB159375586D810EC5DCB66AF0D7039%40MWHPR21MB1 593.namprd21.prod.outlook.com%2F&data=04%7C01%7Clongli%40micr osoft.com%7C7fdf2d6ed15d4d4122a308d94b6eeed0%7C72f988bf86f141af91 ab2d7cd011db47%7C1%7C0%7C637623762292949381%7CUnknown%7CTWFp
Quoted 8 lines
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC I6Mn0%3D%7C3000&sdata=kv0ZkU1QL6TxlJJZEQEsT7aqLFL9lmP2SStz8k U5sIs%3D&reserved=0:
=====
Seems like a #include ofasm/ioctl.h(or something similar) is needed so that_IOWRis defined. Also, a #include is needed for__u32,__aligned_u64,guid_t, etc.
=====
The user-space code includes "sys/ioctl.h" for calling into ioctl(). "sys/ioctl.h" includes <linux/ioctl.h>, so it has no problem finding _IOWR.
guid_t is defined in <uapi/linux/uuid.h>, included from <linux/uuid.h> (in this file)__u32 and __aligned_u64 are defined in <uapi/linux/types.>, which is included from <linux/kernel.h> (in this file)
Quoted 34 lines
Why was no include added?
Quoted 16 lines
include/uapi/misc/hv_azure_blob.h new file +15
-0+ +/* user-mode sync request sent through ioctl */ struct +az_blob_request_sync_response { +__u32 status; +__u32 response_len; +}; + +struct az_blob_request_sync { +guid_t guid; +__u32 timeout; +__u32 request_len; +__u32 response_len; +__u32 data_len; +__u32 data_valid; +__aligned_u64 request_buffer;include/uapi/misc/hv_azure_blob.h line 34
Quoted 10 lines
include/uapi/misc/hv_azure_blob.h new file +10
-0+__aligned_u64 response_buffer; +__aligned_u64 data_buffer; + struct az_blob_request_sync_response response; }; + +#define AZ_BLOB_MAGIC_NUMBER 'R' +#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \ + _IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \ + struct az_blob_request_sync) + +#endif /* define _AZ_BLOB_H */ thanks,
--
js
suse labs
Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 31 lines · Long Li
Quoted 22 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
include/uapi/misc/hv_azure_blob.h line 8
Quoted 11 lines · On 20. 07. 21, 5:31, [email protected] wrote:
include/uapi/misc/hv_azure_blob.h new file +8
-0@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* Copyright (c) 2021 Microsoft Corporation. */ + +#ifndef _AZ_BLOB_H +#define _AZ_BLOB_H + +#include <linux/kernel.h> +#include <linux/uuid.h> Quoting from
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Flinux- doc%2FMWHPR21MB159375586D810EC5DCB66AF0D7039%40MWHPR21MB1 593.namprd21.prod.outlook.com%2F&data=04%7C01%7Clongli%40micr osoft.com%7C7fdf2d6ed15d4d4122a308d94b6eeed0%7C72f988bf86f141af91 ab2d7cd011db47%7C1%7C0%7C637623762292949381%7CUnknown%7CTWFpQuoted 8 lines
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC I6Mn0%3D%7C3000&sdata=kv0ZkU1QL6TxlJJZEQEsT7aqLFL9lmP2SStz8k U5sIs%3D&reserved=0:
=====
Seems like a #include ofasm/ioctl.h(or something similar) is needed so that_IOWRis defined. Also, a #include is needed for__u32,__aligned_u64,guid_t, etc.
=====
Quoted 5 lines
The user-space code includes "
sys/ioctl.h" for calling intoioctl(). "sys/ioctl.h" includes <linux/ioctl.h>, so it has no problem finding_IOWR.
guid_tis defined in <uapi/linux/uuid.h>, included from <linux/uuid.h> (in this file)__u32and__aligned_u64are defined in <uapi/linux/types.>, which is included from <linux/kernel.h> (in this file)
No, please don't rely on implicit include chains. Nor that userspace solves the includes for you.
thanks,
Signature · 2 lines
js suse labs
RE: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 47 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
Quoted 17 lines · Long Li
Quoted 17 lines
Subject: Re: [Patch v4 2/3] Drivers: hv: add Azure Blob driver
include/uapi/misc/hv_azure_blob.h line 8
Quoted 11 lines · On 20. 07. 21, 5:31, [email protected] wrote:
include/uapi/misc/hv_azure_blob.h new file +8
-0@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ +/* Copyright (c) 2021 Microsoft Corporation. */ + +#ifndef _AZ_BLOB_H +#define _AZ_BLOB_H + +#include <linux/kernel.h> +#include <linux/uuid.h> Quoting from
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
Quoted 2 lines
Quoted 2 lines
kernel.org%2Flinux-
doc%2FMWHPR21MB159375586D810EC5DCB66AF0D7039%40MWHPR21M B1
Quoted 1 line
Quoted 1 line
593.namprd21.prod.outlook.com%2F&data=04%7C01%7Clongli%40micr
Quoted 1 line
Quoted 1 line
osoft.com%7C7fdf2d6ed15d4d4122a308d94b6eeed0%7C72f988bf86f141af 91
Quoted 1 line
Quoted 1 line
ab2d7cd011db47%7C1%7C0%7C637623762292949381%7CUnknown%7CTW Fp
Quoted 1 line
Quoted 1 line
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
Quoted 1 line
Quoted 1 line
I6Mn0%3D%7C3000&sdata=kv0ZkU1QL6TxlJJZEQEsT7aqLFL9lmP2SStz8k
Quoted 10 lines
Quoted 6 lines
U5sIs%3D&reserved=0:
=====
Seems like a #include ofasm/ioctl.h(or something similar) is needed so that_IOWRis defined. Also, a #include is needed for__u32,__aligned_u64,guid_t, etc.
=====The user-space code includes "
sys/ioctl.h" for calling intoioctl(). "sys/ioctl.h" includes <linux/ioctl.h>, so it has no problem finding_IOWR.
Quoted 7 lines
Quoted 4 lines
guid_tis defined in <uapi/linux/uuid.h>, included from <linux/uuid.h> (in this file)__u32and__aligned_u64are defined in <uapi/linux/types.>, which is included from <linux/kernel.h> (in this file)No, please don't rely on implicit include chains. Nor that userspace solves the includes for you.
Will fix this.
Quoted 5 lines
thanks,
--
js
suse labs
PATCH v4 3/3 Drivers: hv: Add to maintainer for Hyper-V/Azure drivers
In reply to [email protected]From: Long Li <[email protected]>
Add [email protected] to maintainer list.
--- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+)
MAINTAINERS +2 -0
@@ -8440,6 +8440,7 @@ M: Haiyang Zhang <[email protected]> M: Stephen Hemminger <[email protected]> M: Wei Liu <[email protected]> M: Dexuan Cui <[email protected]> +M: Long Li <[email protected]> L: [email protected] S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
@@ -8468,6 +8469,7 @@ F: include/asm-generic/mshyperv.h F: include/clocksource/hyperv_timer.h F: include/linux/hyperv.h F: include/uapi/linux/hyperv.h +F: include/uapi/misc/hv_azure_blob.h F: net/vmw_vsock/hyperv_transport.c F: tools/hv/
Signature · 1 line
2.25.1
Quoted 71 lines · On 7/19/21 8:31 PM, [email protected] wrote:
From: Long Li <[email protected]>
Microsoft Azure Blob storage service exposes a REST API to applications for data access.
(https://docs.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api)This patchset implements a VSC (Virtualization Service Consumer) that communicates with a VSP (Virtualization Service Provider) on the Hyper-V host to execute Blob storage access via native network stack on the host. This VSC doesn't implement the semantics of REST API. Those are implemented in user-space. The VSC provides a fast data path to VSP.
Answers to some previous questions discussing the driver:
Q: Why this driver doesn't use the block layer
A: The Azure Blob is based on a model of object oriented storage. The storage object is not modeled in block sectors. While it's possible to present the storage object as a block device (assuming it makes sense to fake all the block device attributes), we lose the ability to express functionality that are defined in the REST API.
Q: You just lost all use of caching and
io_uringand loads of other kernel infrastructure that has been developed and relied on for decades?A: The REST API is not designed to have caching at system level. This driver doesn't attempt to improve on this. There are discussions on supporting
ioctl()onio_uring(https://lwn.net/Articles/844875/), that will benefit this driver. The block I/O scheduling is not helpful in this case, as the Blob application and Blob storage server have complete knowledge on the I/O pattern based on storage object type. This knowledge doesn't get easily consumed by the block layer.Q: You also just abandoned the POSIX model and forced people to use a random-custom-library just to access their storage devices, breaking all existing programs in the world?
A: The existing Blob applications access storage via HTTP (REST API). They don't use POSIX interface. The interface for Azure Blob is not designed on POSIX.
Q: What programs today use this new API?
A: Currently none is released. But per above, there are also none using POSIX.
Q: Where is the API published and what ensures that it will remain stable?
A: Cloud based REST protocols have similar considerations to the kernel in terms of interface stability. Applications depend on cloud services via REST in much the same way as they depend on kernel services. Because applications can consume cloud APIs over the Internet, there is no opportunity to recompile applications to ensure compatibility. This causes the underlying APIs to be exceptionally stable, and Azure Blob has not removed support for an exposed API to date. This driver is supporting a pass-through model where requests in a guest process can be reflected to a VM host environment. Like the current REST interface, the goal is to ensure that each host provide a high degree of compatibility with each guest, but that task is largely outside the scope of this driver, which exists to communicate requests in the same way an HTTP stack would. Just like an HTTP stack does not require updates to add a new custom header or receive one from a server, this driver does not require updates for new functionality so long as the high level request/response model is retained.
Q: What happens when it changes over time, do we have to rebuild all userspace applications?
A: No. We don’t rebuild them all to talk HTTP either. In the current HTTP scheme, applications specify the version of the protocol they talk, and the storage backend responds with that version.
Quoted 6 lines
Q: What happens to the kernel code over time, how do you handle changes to the API there?
A: The goal of this driver is to get requests to the Hyper-V host, so the kernel isn’t involved in API changes, in the same way that HTTP implementations are robust to extra functionality being added to HTTP.
Another question is why do we need this in the kernel? Has it been considered to provide a driver similar to vfio on top of the Hyper-V bus such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like io_uring can be built easily on top of vfio.
Thanks,
Bart.
Quoted 4 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like
io_uringcan be built easily on top of vfio.
Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
Quoted 11 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 4 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like
io_uringcan be built easily on top of vfio.Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
The driver is modeled to support multiple processes/users over a VMBUS channel. I don't see a way that this can be implemented through VFIO?
Even if it can be done, this exposes a security risk as the same VMBUS channel is shared by multiple processes in user-mode.
Quoted 9 lines · Long Li
Quoted 9 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 4 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like
io_uringcan be built easily on top of vfio.
Quoted 8 lines
Quoted 2 lines
Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
The driver is modeled to support multiple processes/users over a VMBUS channel. I don't see a way that this can be implemented through VFIO?
Even if it can be done, this exposes a security risk as the same VMBUS channel is shared by multiple processes in user-mode.
Sharing a VMBUS channel among processes is not necessary. I propose to assign one VMBUS channel to each process and to multiplex I/O submitted to channels associated with the same blob storage object inside e.g. the hypervisor. This is not a new idea. In the NVMe specification there is a diagram that shows that multiple NVMe controllers can provide access to the same NVMe namespace. See also diagram "Figure 416: NVM Subsystem with Three I/O Controllers" in version 1.4 of the NVMe specification.
Bart.
Quoted 30 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 9 lines · Long Li
Quoted 9 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 4 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like
io_uringcan be built easily on top of vfio.Quoted 8 lines
Quoted 2 lines
Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
The driver is modeled to support multiple processes/users over a VMBUS channel. I don't see a way that this can be implemented through VFIO?
Even if it can be done, this exposes a security risk as the same VMBUS channel is shared by multiple processes in user-mode.
Sharing a VMBUS channel among processes is not necessary. I propose to assign one VMBUS channel to each process and to multiplex I/O submitted to channels associated with the same blob storage object inside e.g. the hypervisor. This is not a new idea. In the NVMe specification there is a diagram that shows that multiple NVMe controllers can provide access to the same NVMe namespace. See also diagram "Figure 416: NVM Subsystem with Three I/O Controllers" in version 1.4 of the NVMe specification.
Quoted 1 line
Bart.
Currently, the Hyper-V is not designed to have one VMBUS channel for each process. In Hyper-V, a channel is offered from the host to the guest VM. The host doesn't know in advance how many processes are going to use this service so it can't offer those channels in advance. There is no mechanism to offer dynamic per-process allocated channels based on guest needs. Some devices (e.g. network and storage) use multiple channels for scalability but they are not for serving individual processes.
Assigning one VMBUS channel per process needs significant change on the Hyper-V side.
Long
Quoted 30 lines · Long Li
Quoted 30 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 9 lines · Long Li
Quoted 9 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 4 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications or polling. An interface like
io_uringcan be built easily on top of vfio.Quoted 8 lines
Quoted 2 lines
Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
The driver is modeled to support multiple processes/users over a VMBUS channel. I don't see a way that this can be implemented through VFIO?
Even if it can be done, this exposes a security risk as the same VMBUS channel is shared by multiple processes in user-mode.
Sharing a VMBUS channel among processes is not necessary. I propose to assign one VMBUS channel to each process and to multiplex I/O submitted to channels associated with the same blob storage object inside e.g. the hypervisor. This is not a new idea. In the NVMe specification there is a diagram that shows that multiple NVMe controllers can provide access to the same NVMe namespace. See also diagram "Figure 416: NVM Subsystem with Three I/O Controllers" in version 1.4 of the NVMe specification.
Quoted 3 lines
Quoted 1 line
Bart.
Currently, the Hyper-V is not designed to have one VMBUS channel for each process.
So it's a slow interface :(
Quoted 8 lines
In Hyper-V, a channel is offered from the host to the guest VM. The host doesn't know in advance how many processes are going to use this service so it can't offer those channels in advance. There is no mechanism to offer dynamic per-process allocated channels based on guest needs. Some devices (e.g. network and storage) use multiple channels for scalability but they are not for serving individual processes.
Assigning one VMBUS channel per process needs significant change on the Hyper-V side.
What is the throughput of a single channel as-is? You provided no benchmarks or numbers at all in this patchset which would justify this new kernel driver :(
thanks,
greg k-h
Quoted 52 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 11 lines · Long Li
Quoted 11 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 7 lines · Long Li
Quoted 7 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 3 lines · Bart Van Assche
such that this object storage driver can be implemented as a user-space library instead of as a kernel driver? As you may know vfio users can either use eventfds for completion notifications
or polling.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
An interface like
io_uringcan be built easily on top of vfio.Yes. Similar to say the NVMe K/V command set this does not look like a candidate for a kernel driver.
The driver is modeled to support multiple processes/users over a VMBUS channel. I don't see a way that this can be implemented
through VFIO?
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Even if it can be done, this exposes a security risk as the same VMBUS channel is shared by multiple processes in user-mode.
Sharing a VMBUS channel among processes is not necessary. I propose to assign one VMBUS channel to each process and to multiplex I/O submitted to channels associated with the same blob storage object inside e.g. the hypervisor. This is not a new idea. In the NVMe specification there is a diagram that shows that multiple NVMe controllers can provide access to the same NVMe namespace. See also diagram "Figure 416: NVM Subsystem with Three I/O Controllers" in
version 1.4 of the NVMe specification.
Quoted 4 lines
Quoted 2 lines
Bart.
Currently, the Hyper-V is not designed to have one VMBUS channel for
each process.
So it's a slow interface :(
Quoted 4 lines
In Hyper-V, a channel is offered from the host to the guest VM. The host doesn't know in advance how many processes are going to use this service so it can't offer those channels in advance. There is no mechanism to offer dynamic per-process allocated channels based on
guest needs. Some devices (e.g.
Quoted 3 lines
network and storage) use multiple channels for scalability but they are not for serving individual processes.
Quoted 6 lines
Quoted 1 line
Assigning one VMBUS channel per process needs significant change on the
Hyper-V side.
What is the throughput of a single channel as-is? You provided no benchmarks or numbers at all in this patchset which would justify this new kernel driver :(
Test data shows a single channel is not a limitation of the target workload. The VSP/VSC protocol is designed to avoid data copy as much as possible. Being a VMBUS device, the Hyper-V is capable of allocating multiple channels on different CPUs if a single channel proves to be a bottleneck.
Preliminary test results show the performance increase of up to 30% in data center environment, while at the same time reducing the number of servers and CPUs serving Blob requests, as compared to going through the complete HTTP stack. This also enables the use of transport technology directly to backend server that are not available to VMs (for example RDMA transport) due to security reasons.
Long
Quoted 4 lines
thanks,
greg k-h
Quoted 14 lines · On Mon, Jul 19, 2021 at 08:31:03PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
Microsoft Azure Blob storage service exposes a REST API to applications for data access.
(https://docs.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api)This patchset implements a VSC (Virtualization Service Consumer) that communicates with a VSP (Virtualization Service Provider) on the Hyper-V host to execute Blob storage access via native network stack on the host. This VSC doesn't implement the semantics of REST API. Those are implemented in user-space. The VSC provides a fast data path to VSP.
Answers to some previous questions discussing the driver:
Quoted 7 lines
Q: Why this driver doesn't use the block layer
A: The Azure Blob is based on a model of object oriented storage. The storage object is not modeled in block sectors. While it's possible to present the storage object as a block device (assuming it makes sense to fake all the block device attributes), we lose the ability to express functionality that are defined in the REST API.
What "REST API"?
Why doesn't object oriented storage map to a file handle to read from? No need to mess with "blocks", why would you care about them?
And again, you loose all caching, this thing has got to be really slow, why add a slow storage interface? What workload requires this type of slow block storage?
Quoted 7 lines
Q: You also just abandoned the POSIX model and forced people to use a random-custom-library just to access their storage devices, breaking all existing programs in the world?
A: The existing Blob applications access storage via HTTP (REST API). They don't use POSIX interface. The interface for Azure Blob is not designed on POSIX.
I do not see a HTTP interface here, what does that have to do with the kernel?
I see a single ioctl interface, that's all.
Quoted 4 lines
Q: What programs today use this new API?
A: Currently none is released. But per above, there are also none using POSIX.
Great, so no one uses this, so who is asking for and requiring this thing?
What about just doing it all from userspace using FUSE? Why does this HAVE to be a kernel driver?
thanks,
greg k-h
Quoted 42 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 5 lines · On Mon, Jul 19, 2021 at 08:31:03PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
Microsoft Azure Blob storage service exposes a REST API to applications for data access.
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc
Quoted 1 line
s.microsoft.com%2Fen-us%2Frest%2Fapi%2Fstorageservices%2Fblob-
service-
Quoted 1 line
rest-
api&data=04%7C01%7Clongli%40microsoft.com%7Ce499fbe161554232e
Quoted 1 line
b1608d94b96a772%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637 623932
Quoted 1 line
843247787%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj oiV2luMzIi
Quoted 1 line
LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=9CKNHAmurdBWp ZeLfkiC18
Quoted 5 lines
CXNg66UhKsSZzzHZkzf0Y%3D&reserved=0)
This patchset implements a VSC (Virtualization Service Consumer) that communicates with a VSP (Virtualization Service Provider) on the Hyper-V host to execute Blob storage access via native network stack on
the host.
Quoted 5 lines
This VSC doesn't implement the semantics of REST API. Those are implemented in user-space. The VSC provides a fast data path to VSP.
Answers to some previous questions discussing the driver:
Quoted 7 lines
Q: Why this driver doesn't use the block layer
A: The Azure Blob is based on a model of object oriented storage. The storage object is not modeled in block sectors. While it's possible to present the storage object as a block device (assuming it makes sense to fake all the block device attributes), we lose the ability to express functionality that are defined in the REST API.
Quoted 8 lines
What "REST API"?
Why doesn't object oriented storage map to a file handle to read from? No need to mess with "blocks", why would you care about them?
And again, you loose all caching, this thing has got to be really slow, why add a slow storage interface? What workload requires this type of slow block storage?
"Blob REST API" expresses storage request semantics through HTTP. In Blob REST API, each request is associated with a context meta data expressed in HTTP headers. The ability to express those semantics is rich, it's only limited by HTTP protocol.
There are attempts to implement the Blob as a file system. Here is an example filesystem (BlobFuse) implemented for Blob:
(https://github.com/Azure/azure-storage-fuse).
It's doable, but at the same time you lose all the performance and shareable/security features presented in the REST API for Blob. A POSIX interface cannot express same functionality as the REST API for Blob.
For example, The Blob API for read (Get Blob,
https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob)
has rich meta data context that cannot easily be mapped to POSIX. The same goes to other Blob API to manage security tokens and the life cycle of shareable objects.
BlobFuse (above) filesystem demonstrated why Blob should not be implemented on a filesystem. It's useable for data management purposes. It's not usable for an I/O intensive workload. It's not usable for managing sharable objects and security tokens.
Blob is designed not to use file system caching and block layer I/O scheduling. There are many solutions existing today, that access raw disk for I/O, bypassing filesystem and block layer. For example, many database applications access raw disks for I/O. When the application knows the I/O pattern and its intended behavior, it doesn't get much benefit from filesystem or block.
Quoted 5 lines
Quoted 5 lines
Q: You also just abandoned the POSIX model and forced people to use a random-custom-library just to access their storage devices, breaking all existing programs in the world?
Quoted 7 lines
Quoted 3 lines
A: The existing Blob applications access storage via HTTP (REST API). They don't use POSIX interface. The interface for Azure Blob is not designed on POSIX.
I do not see a HTTP interface here, what does that have to do with the kernel?
I see a single ioctl interface, that's all.
The driver doesn't attempt to implement Blob API or HTTP. It presents a fast data path to pass Blob requests to hosts, so the guest VM doesn't need to assemble a HTTP packet for each Blob REST requests. This also eliminates additional overhead in guest network stack to send the HTTP packets over TCP/IP.
Once the request reaches the Azure host, it knows the best way to reach to the backend storage and serving the Blob request, while at the same time all the security and integrity features are preserved.
Quoted 10 lines
Quoted 5 lines
Q: What programs today use this new API?
A: Currently none is released. But per above, there are also none using POSIX.
Great, so no one uses this, so who is asking for and requiring this thing?
What about just doing it all from userspace using FUSE? Why does this HAVE to be a kernel driver?
We have a major workload nearing the end of development. Compared with REST over HTTP, this device model presented faster data access and CPU savings in that there is no overhead of sending HTTP over network.
Eventually, all the existing Blob REST API applications can use this new API, once it gets to their Blob transport libraries.
I have explained why BlobFuse is not suitable for production workloads. There are people using BlobFuse but mostly for management purposes.
Thanks,
Long
Quoted 4 lines
thanks,
greg k-h
Quoted 47 lines · Long Li
Quoted 47 lines
Subject: Re: [Patch v4 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob
Quoted 5 lines · On Mon, Jul 19, 2021 at 08:31:03PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
Microsoft Azure Blob storage service exposes a REST API to applications for data access.
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc
Quoted 1 line
s.microsoft.com%2Fen-us%2Frest%2Fapi%2Fstorageservices%2Fblob-
service-
Quoted 1 line
rest-
api&data=04%7C01%7Clongli%40microsoft.com%7Ce499fbe161554232e
Quoted 1 line
b1608d94b96a772%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637 623932
Quoted 1 line
843247787%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj oiV2luMzIi
Quoted 1 line
LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=9CKNHAmurdBWp ZeLfkiC18
Quoted 5 lines
CXNg66UhKsSZzzHZkzf0Y%3D&reserved=0)
This patchset implements a VSC (Virtualization Service Consumer) that communicates with a VSP (Virtualization Service Provider) on the Hyper-V host to execute Blob storage access via native network stack on
the host.
Quoted 5 lines
This VSC doesn't implement the semantics of REST API. Those are implemented in user-space. The VSC provides a fast data path to VSP.
Answers to some previous questions discussing the driver:
Quoted 7 lines
Q: Why this driver doesn't use the block layer
A: The Azure Blob is based on a model of object oriented storage. The storage object is not modeled in block sectors. While it's possible to present the storage object as a block device (assuming it makes sense to fake all the block device attributes), we lose the ability to express functionality that are defined in the REST API.
What "REST API"?
Why doesn't object oriented storage map to a file handle to read from? No need to mess with "blocks", why would you care about them?
Quoted 8 lines
Quoted 3 lines
And again, you loose all caching, this thing has got to be really slow, why add a slow storage interface? What workload requires this type of slow block storage?
"Blob REST API" expresses storage request semantics through HTTP. In Blob REST API, each request is associated with a context meta data expressed in HTTP headers. The ability to express those semantics is rich, it's only limited by HTTP protocol.
HTTP has nothing to do with the kernel, so I do not understand what you are talking about here.
Quoted 6 lines
There are attempts to implement the Blob as a file system. Here is an example filesystem (BlobFuse) implemented for Blob:
(https://github.com/Azure/azure-storage-fuse).It's doable, but at the same time you lose all the performance and shareable/security features presented in the REST API for Blob.
What sharable/security features are in this driver instead? I saw none.
Quoted 2 lines
A POSIX
interface cannot express same functionality as the REST API for Blob.
But you are not putting a REST api into this kernel driver, so I fail to understand this.
Quoted 5 lines
For example, The Blob API for read (Get Blob,
https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob)
has rich meta data context that cannot easily be mapped to POSIX. The same goes to other Blob API to manage security tokens and the life cycle of shareable objects.
How can you have sharable objects in this ioctl interface instead?
Quoted 3 lines
BlobFuse (above) filesystem demonstrated why Blob should not be implemented on a filesystem. It's useable for data management purposes. It's not usable for an I/O intensive workload. It's not usable for managing sharable objects and security tokens.
What is the bottleneck for the throughput/performance issues involved?
Quoted 5 lines
Blob is designed not to use file system caching and block layer I/O scheduling. There are many solutions existing today, that access raw disk for I/O, bypassing filesystem and block layer. For example, many database applications access raw disks for I/O. When the application knows the I/O pattern and its intended behavior, it doesn't get much benefit from filesystem or block.
Databases that use raw i/o "know what they are doing" and are constantly fighting with the kernel. Don't expect kernel developers to just think that this is ok.
But this is not what you are doing here at all, this is an object storage, you are still being forced to open/ioctl/close to get an object instead of just doing open/read/close, so I fail to understand where the performance issues are.
And if they are in the FUSE layer, why not just write a filesystem layer in the kernel instead to resolve that? Who is insisting that you do this through a character device driver to get filesystem data?
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Q: You also just abandoned the POSIX model and forced people to use a random-custom-library just to access their storage devices, breaking all existing programs in the world?
A: The existing Blob applications access storage via HTTP (REST API). They don't use POSIX interface. The interface for Azure Blob is not designed on POSIX.
Quoted 8 lines
Quoted 3 lines
I do not see a HTTP interface here, what does that have to do with the kernel?
I see a single ioctl interface, that's all.
The driver doesn't attempt to implement Blob API or HTTP. It presents a fast data path to pass Blob requests to hosts, so the guest VM doesn't need to assemble a HTTP packet for each Blob REST requests. This also eliminates additional overhead in guest network stack to send the HTTP packets over TCP/IP.
Again, I fail to understand how http or tcp/ip comes into play here at all, that's not what this driver does.
Quoted 3 lines
Once the request reaches the Azure host, it knows the best way to reach to the backend storage and serving the Blob request, while at the same time all the security and integrity features are preserved.
I do not understand this statement at all.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Q: What programs today use this new API?
A: Currently none is released. But per above, there are also none using POSIX.
Quoted 8 lines
Quoted 4 lines
Great, so no one uses this, so who is asking for and requiring this thing?
What about just doing it all from userspace using FUSE? Why does this HAVE to be a kernel driver?
We have a major workload nearing the end of development. Compared with REST over HTTP, this device model presented faster data access and CPU savings in that there is no overhead of sending HTTP over network.
Your development cycle means nothing to us, sorry. Please realize that if you submit something that is not acceptable to us, there is no requirement that we take it just because we feel this is implemented in totally the wrong way.
And you have not, again, proven that there is any performance
improvement anywhere due to lack of numbers and data. And again, what
does HTTP have to do with this driver.
Quoted 2 lines
Eventually, all the existing Blob REST API applications can use this new API, once it gets to their Blob transport libraries.
What applications? What libraries? Who is using any of this?
Quoted 2 lines
I have explained why BlobFuse is not suitable for production workloads. There are people using BlobFuse but mostly for management purposes.
I fail to see why it is not usable as you have not provided any real information, sorry.
Please step back and write up a document that explains the problem you are trying to solve and then we can go from there. Right now you are throwing a driver at us and expecting us to just accept it, despite it looking like it is completly wrong for the problem space it is attempting to interact with.
Please work with some experienced Linux kernel developers on your team to do all of this _BEFORE_ submitting it to the community again. Based on the mistakes made so far, it looks like you could use some guidance in turning this into something that might be acceptable. As it is, you are a long way off.
good luck!
greg k-h