Skip to contentSkip to conversation

Introduce a driver to support host accelerated access to Microsoft Azure Blob

PATCH Superseded [email protected] 9 participants 29 messages Jun 26, 2021 – Jul 12, 2021

Subsystems (3)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover Introduce a driver to support host accelerated access to Microsoft Azure Blob

From: Long Li <[email protected]>

Microsoft Azure Blob storage service exposes a REST API to applications for data access. While it's flexible and works on most platforms, it's not as efficient as native network stack.

This patchset implements a VSC that communicates with a VSP on the host to execute blob storage access via native network stack on the host.

Reference:
https://azure.microsoft.com/en-us/services/storage/blobs/#overview

Long Li (3):
Drivers: hv: vmbus: add support to ignore certain PCIE devices
Drivers: hv: add Azure Blob driver
Drivers: hv: add to maintainer

Changes:

v2:
Refactored the code in vmbus to scan devices
Reworked Azure Blob driver and moved user-mode interfaces to uapi

 Documentation/userspace-api/ioctl/ioctl-number.rst |   2 +
 MAINTAINERS                                        |   1 +
 drivers/hv/Kconfig                                 |  10 +
 drivers/hv/Makefile                                |   1 +
 drivers/hv/azure_blob.c                            | 621 +++++++++++++++++++++
 drivers/hv/channel_mgmt.c                          |  55 +-
 include/linux/hyperv.h                             |   9 +
 include/uapi/misc/azure_blob.h                     |  31 +
 8 files changed, 724 insertions(+), 6 deletions(-)
 create mode 100644 drivers/hv/azure_blob.c
 create mode 100644 include/uapi/misc/azure_blob.h
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v2 1/3 Drivers: hv: vmbus: add support to ignore certain PCIE devices

From: Long Li <[email protected]>

Under certain configurations when Hyper-v presents a device to VMBUS, it may have a VMBUS channel and a PCIe channel. The PCIe channel is not used in Linux and does not have a backing PCIE device on Hyper-v. For such devices, ignore those PCIe channels so they are not getting probed by the PCI subsystem.

Cc K. Y. Srinivasan <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
---
 drivers/hv/channel_mgmt.c | 48 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 42 insertions(+), 6 deletions(-)
drivers/hv/channel_mgmt.c +42 -6
index caf6d0c..0c75662 100644
@@ -26,6 +26,21 @@
 
 static void init_vp_index(struct vmbus_channel *channel);
 
+/*
+ * For some VMBUS devices, Hyper-v also presents certain PCIE devices as
+ * part of the host device implementation. Those devices have no real
+ * PCI implementation in Hyper-V, and should be ignored and not presented
+ * to the PCI layer.
+ */
+static const guid_t vpci_ignore_instances[] = {
+	/*
+	 * Azure Blob instance ID in VPCI
+	 * {d4573da2-2caa-4711-a8f9-bbabf4ee9685}
+	 */
+	GUID_INIT(0xd4573da2, 0x2caa, 0x4711, 0xa8, 0xf9,
+		0xbb, 0xab, 0xf4, 0xee, 0x96, 0x85),
+};
+
 const struct vmbus_device vmbus_devs[] = {
 	/* IDE */
 	{ .dev_type = HV_IDE,
@@ -187,20 +202,19 @@ static bool is_unsupported_vmbus_devs(const guid_t *guid)
 	return false;
 }
 
-static u16 hv_get_dev_type(const struct vmbus_channel *channel)
+static u16 hv_get_dev_type(const guid_t *guid)
 {
-	const guid_t *guid = &channel->offermsg.offer.if_type;
 	u16 i;
 
-	if (is_hvsock_channel(channel) || is_unsupported_vmbus_devs(guid))
+	if (is_unsupported_vmbus_devs(guid))
 		return HV_UNKNOWN;
 
 	for (i = HV_IDE; i < HV_UNKNOWN; i++) {
 		if (guid_equal(guid, &vmbus_devs[i].guid))
-			return i;
+			return vmbus_devs[i].dev_type;
 	}
 	pr_info("Unknown GUID: %pUl\n", guid);
-	return i;
+	return HV_UNKNOWN;
 }
 
 /**
@@ -487,6 +501,16 @@ void vmbus_free_channels(void)
 	}
 }
 
+static bool ignore_pcie_device(guid_t *if_instance)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(vpci_ignore_instances); i++)
+		if (guid_equal(&vpci_ignore_instances[i], if_instance))
+			return true;
+	return false;
+}
+
 /* Note: the function can run concurrently for primary/sub channels. */
 static void vmbus_add_channel_work(struct work_struct *work)
 {
@@ -910,7 +934,11 @@ static void vmbus_setup_channel_state(struct vmbus_channel *channel,
 	       sizeof(struct vmbus_channel_offer_channel));
 	channel->monitor_grp = (u8)offer->monitorid / 32;
 	channel->monitor_bit = (u8)offer->monitorid % 32;
-	channel->device_id = hv_get_dev_type(channel);
+	if (is_hvsock_channel(channel))
+		channel->device_id = HV_UNKNOWN;
+	else
+		channel->device_id =
+			hv_get_dev_type(&channel->offermsg.offer.if_type);
 }
 
 /*
@@ -972,6 +1000,14 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr)
 
 	trace_vmbus_onoffer(offer);
 
+	/* Check to see if we should ignore this PCIe channel */
+	if (hv_get_dev_type(&offer->offer.if_type) == HV_PCIE &&
+	    ignore_pcie_device(&offer->offer.if_instance)) {
+		pr_debug("Ignore instance %pUl over VPCI\n",
+			&offer->offer.if_instance);
+		return;
+	}
+
 	if (!vmbus_is_valid_device(&offer->offer.if_type)) {
 		pr_err_ratelimited("Invalid offer %d from the host supporting isolation\n",
 				   offer->child_relid);
Signature · 1 line
1.8.3.1
Reviewed-by on v2 1/3
#2 lore raw

RE: [Patch v2 1/3] Drivers: hv: vmbus: add support to ignore certain PCIE devices

From: [email protected] <[email protected]> Sent: Friday, June 25, 2021 11:30 PM

Quoted 13 lines

Under certain configurations when Hyper-v presents a device to VMBUS, it may have a VMBUS channel and a PCIe channel. The PCIe channel is not used in Linux and does not have a backing PCIE device on Hyper-v. For such devices, ignore those PCIe channels so they are not getting probed by the PCI subsystem.

Cc K. Y. Srinivasan <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
Quoted 4 lines
---
 drivers/hv/channel_mgmt.c | 48 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 42 insertions(+), 6 deletions(-)
#3 lore raw

PATCH v2 2/3 Drivers: hv: add Azure Blob driver

In reply to [email protected]

From: Long Li <[email protected]>

Azure Blob storage provides scalable and durable data storage for Azure.
(https://azure.microsoft.com/en-us/services/storage/blobs/)

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server access.

Cc Jonathan Corbet <[email protected]>
Cc "K. Y. Srinivasan" <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Bjorn Andersson <[email protected]>
Cc Hans de Goede <[email protected]>
Cc Dan Williams <[email protected]>
Cc Maximilian Luz <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Ben Widawsky <[email protected]>
Cc Jiri Slaby <[email protected]>
Cc Andra Paraschiv <[email protected]>
Cc Siddharth Gupta <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 Documentation/userspace-api/ioctl/ioctl-number.rst |   2 +
 drivers/hv/Kconfig                                 |  10 +
 drivers/hv/Makefile                                |   1 +
 drivers/hv/azure_blob.c                            | 655 +++++++++++++++++++++
 drivers/hv/channel_mgmt.c                          |   7 +
 include/linux/hyperv.h                             |   9 +
 include/uapi/misc/azure_blob.h                     |  31 +
 7 files changed, 715 insertions(+)
 create mode 100644 drivers/hv/azure_blob.c
 create mode 100644 include/uapi/misc/azure_blob.h
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
index 9bfc2b5..d3c2a90 100644
@@ -180,6 +180,8 @@ Code  Seq#    Include File                                           Comments
 'R'   01     linux/rfkill.h                                          conflict!
 'R'   C0-DF  net/bluetooth/rfcomm.h
 'R'   E0     uapi/linux/fsl_mc.h
+'R'   F0-FF  uapi/misc/azure_blob.h                                  Microsoft Azure Blob driver
+                                                                     <mailto:[email protected]>
 'S'   all    linux/cdrom.h                                           conflict!
 'S'   80-81  scsi/scsi_ioctl.h                                       conflict!
 'S'   82-FF  scsi/scsi.h                                             conflict!
drivers/hv/Kconfig +10 -0
index 66c794d..e08b8d3 100644
@@ -27,4 +27,14 @@ config HYPERV_BALLOON
 	help
 	  Select this option to enable Hyper-V Balloon driver.
 
+config HYPERV_AZURE_BLOB
+	tristate "Microsoft Azure Blob driver"
+	depends on HYPERV && X86_64
+	help
+	  Select this option to enable Microsoft Azure Blob driver.
+
+	  This driver supports accelerated Microsoft Azure Blob access.
+	  To compile this driver as a module, choose M here. The module will be
+	  called azure_blob.
+
 endmenu
drivers/hv/Makefile +1 -0
index 94daf82..a322575 100644
@@ -2,6 +2,7 @@
 obj-$(CONFIG_HYPERV)		+= hv_vmbus.o
 obj-$(CONFIG_HYPERV_UTILS)	+= hv_utils.o
 obj-$(CONFIG_HYPERV_BALLOON)	+= hv_balloon.o
+obj-$(CONFIG_HYPERV_AZURE_BLOB)	+= azure_blob.o
 
 CFLAGS_hv_trace.o = -I$(src)
 CFLAGS_hv_balloon.o = -I$(src)
drivers/hv/azure_blob.c new file +655 -0

655 lines not on this page · raw message

drivers/hv/channel_mgmt.c +7 -0
index 0c75662..436fdeb 100644
@@ -154,6 +154,13 @@
 	  .allowed_in_isolated = false,
 	},
 
+	/* Azure Blob */
+	{ .dev_type = HV_AZURE_BLOB,
+	  HV_AZURE_BLOB_GUID,
+	  .perf_device = false,
+	  .allowed_in_isolated = false,
+	},
+
 	/* Unknown GUID */
 	{ .dev_type = HV_UNKNOWN,
 	  .perf_device = false,
include/linux/hyperv.h +9 -0
index d1e59db..ac31362 100644
@@ -772,6 +772,7 @@ enum vmbus_device_type {
 	HV_FCOPY,
 	HV_BACKUP,
 	HV_DM,
+	HV_AZURE_BLOB,
 	HV_UNKNOWN,
 };
 
@@ -1350,6 +1351,14 @@ int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,
 			  0x72, 0xe2, 0xff, 0xb1, 0xdc, 0x7f)
 
 /*
+ * Azure Blob GUID
+ * {0590b792-db64-45cc-81db-b8d70c577c9e}
+ */
+#define HV_AZURE_BLOB_GUID \
+	.guid = GUID_INIT(0x0590b792, 0xdb64, 0x45cc, 0x81, 0xdb, \
+			  0xb8, 0xd7, 0x0c, 0x57, 0x7c, 0x9e)
+
+/*
  * Shutdown GUID
  * {0e0b6031-5213-4934-818b-38d90ced39db}
  */
include/uapi/misc/azure_blob.h new file +31 -0
new file mode 100644
index 0000000..29413fc
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* Copyright (c) 2021, Microsoft Corporation. */
+
+#ifndef _AZ_BLOB_H
+#define _AZ_BLOB_H
+
+/* user-mode sync request sent through ioctl */
+struct az_blob_request_sync_response {
+	__u32 status;
+	__u32 response_len;
+};
+
+struct az_blob_request_sync {
+	guid_t guid;
+	__u32 timeout;
+	__u32 request_len;
+	__u32 response_len;
+	__u32 data_len;
+	__u32 data_valid;
+	__aligned_u64 request_buffer;
+	__aligned_u64 response_buffer;
+	__aligned_u64 data_buffer;
+	struct az_blob_request_sync_response response;
+};
+
+#define AZ_BLOB_MAGIC_NUMBER	'R'
+#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \
+		_IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \
+			struct az_blob_request_sync)
+
+#endif /* define _AZ_BLOB_H */
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Thank you for the patch! Yet something to improve:

Hi,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linux/master]
[also build test ERROR on linus/master v5.13-rc7 next-20210625] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/longli-linuxonhyperv-com/Introduce-a-driver-to-support-host-accelerated-access-to-Microsoft-Azure-Blob/20210626-143213
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 0c18f29aae7ce3dadd26d8ee3505d07cc982df75
config: sh-buildonly-randconfig-r006-20210625 (attached as .config)
compiler: sh4-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/76fc89da311e853f8ecc323ad52b367eced9c730
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review longli-linuxonhyperv-com/Introduce-a-driver-to-support-host-accelerated-access-to-Microsoft-Azure-Blob/20210626-143213 git checkout 76fc89da311e853f8ecc323ad52b367eced9c730
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sh

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> error: include/uapi/misc/azure_blob.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier
--
>> error: include/uapi/misc/azure_blob.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier

make[2]: *** [scripts/Makefile.headersinst:63: usr/include/misc/azure_blob.h] Error 1 make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1341: headers] Error 2
make[1]: Target 'headers_install' not remade because of errors.
make: *** [Makefile:215: __sub-make] Error 2
make: Target 'headers_install' not remade because of errors.
--

>> error: include/uapi/misc/azure_blob.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier

make[2]: *** [scripts/Makefile.headersinst:63: usr/include/misc/azure_blob.h] Error 1 make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1341: headers] Error 2
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported by this configuration
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported by this configuration make[2]: *** [scripts/Makefile.build:117: scripts/mod/devicetable-offsets.s] Error 1
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported by this configuration
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported by this configuration make[2]: *** [scripts/Makefile.build:272: scripts/mod/empty.o] Error 1 make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1234: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:215: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#5 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to [email protected]

On 26.06.21 08:30, [email protected] wrote:

Hi,

Quoted 4 lines

From: Long Li <[email protected]>

Azure Blob storage provides scalable and durable data storage for Azure.
(https://azure.microsoft.com/en-us/services/storage/blobs/)

It's a bit hard to quickly find out the exact semantics of that (w/o fully reading the library code ...). :(

Quoted 3 lines

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server access.

I don't think that yet another misc device with some very specific API is a good thing. We should have a more generic userland interface.

Maybe a file system ?

OTOH, I've been considering adding other kind of blob stores (eg. venti and similar things) and inventing an own subsys for that, so we'd have some more universal interface. Unfortunately always been busy with other things.

Maybe we should go that route instead of introducing yet another device specific uapi.

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
maintainer
#6 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 12 lines · metux IT consult

On 26.06.21 08:30, [email protected] wrote:

Hi,

Quoted 4 lines

From: Long Li <[email protected]>

Azure Blob storage provides scalable and durable data storage for Azure.
(https://azure.microsoft.com/en-us/services/storage/blobs/)

It's a bit hard to quickly find out the exact semantics of that (w/o fully reading the library code ...). :(

Quoted 8 lines
Quoted 3 lines

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server access.

I don't think that yet another misc device with some very specific API is a good thing. We should have a more generic userland interface.

Maybe a file system ?

This can already be done if I'm not mistaken:

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux
https://docs.microsoft.com/en-us/azure/storage/blobs/network-file-system-protocol-support-how-to

Quoted 8 lines

OTOH, I've been considering adding other kind of blob stores (eg. venti and similar things) and inventing an own subsys for that, so we'd have some more universal interface. Unfortunately always been busy with other things.

Maybe we should go that route instead of introducing yet another device specific uapi.

This work of course does not preclude anyone from going that route.

Wei.

Quoted 13 lines

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#7 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to [email protected]

From: [email protected] <[email protected]> Sent: Friday, June 25, 2021 11:30 PM

Quoted 246 lines

Azure Blob storage provides scalable and durable data storage for Azure.
(https://azure.microsoft.com/en-us/services/storage/blobs/)

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server access.

Cc Jonathan Corbet <[email protected]>
Cc "K. Y. Srinivasan" <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Bjorn Andersson <[email protected]>
Cc Hans de Goede <[email protected]>
Cc Dan Williams <[email protected]>
Cc Maximilian Luz <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Ben Widawsky <[email protected]>
Cc Jiri Slaby <[email protected]>
Cc Andra Paraschiv <[email protected]>
Cc Siddharth Gupta <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 Documentation/userspace-api/ioctl/ioctl-number.rst |   2 +
 drivers/hv/Kconfig                                 |  10 +
 drivers/hv/Makefile                                |   1 +
 drivers/hv/azure_blob.c                            | 655 +++++++++++++++++++++
 drivers/hv/channel_mgmt.c                          |   7 +
 include/linux/hyperv.h                             |   9 +
 include/uapi/misc/azure_blob.h                     |  31 +
 7 files changed, 715 insertions(+)
 create mode 100644 drivers/hv/azure_blob.c
 create mode 100644 include/uapi/misc/azure_blob.h
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
index 9bfc2b5..d3c2a90 100644
@@ -180,6 +180,8 @@ Code  Seq#    Include File                                           Comments
 'R'   01     linux/rfkill.h                                          conflict!
 'R'   C0-DF  net/bluetooth/rfcomm.h
 'R'   E0     uapi/linux/fsl_mc.h
+'R'   F0-FF  uapi/misc/azure_blob.h                                  Microsoft Azure Blob driver
+                                                                     <mailto:[email protected]>
 'S'   all    linux/cdrom.h                                           conflict!
 'S'   80-81  scsi/scsi_ioctl.h                                       conflict!
 'S'   82-FF  scsi/scsi.h                                             conflict!
drivers/hv/Kconfig +10 -0
index 66c794d..e08b8d3 100644
@@ -27,4 +27,14 @@ config HYPERV_BALLOON
 	help
 	  Select this option to enable Hyper-V Balloon driver.
 
+config HYPERV_AZURE_BLOB
+	tristate "Microsoft Azure Blob driver"
+	depends on HYPERV && X86_64
+	help
+	  Select this option to enable Microsoft Azure Blob driver.
+
+	  This driver supports accelerated Microsoft Azure Blob access.
+	  To compile this driver as a module, choose M here. The module will be
+	  called azure_blob.
+
 endmenu
drivers/hv/Makefile +1 -0
index 94daf82..a322575 100644
@@ -2,6 +2,7 @@
 obj-$(CONFIG_HYPERV)		+= hv_vmbus.o
 obj-$(CONFIG_HYPERV_UTILS)	+= hv_utils.o
 obj-$(CONFIG_HYPERV_BALLOON)	+= hv_balloon.o
+obj-$(CONFIG_HYPERV_AZURE_BLOB)	+= azure_blob.o
 
 CFLAGS_hv_trace.o = -I$(src)
 CFLAGS_hv_balloon.o = -I$(src)
drivers/hv/azure_blob.c new file +157 -0
new file mode 100644
index 0000000..6c8f957
@@ -0,0 +1,655 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/* Copyright (c) 2021, Microsoft Corporation. */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/cred.h>
+#include <linux/debugfs.h>
+#include <linux/pagemap.h>
+#include <linux/hyperv.h>
+#include <linux/miscdevice.h>
+#include <linux/uio.h>
+#include <uapi/misc/azure_blob.h>
+
+struct az_blob_device {
+	struct hv_device *device;
+
+	/* Opened files maintained by this device */
+	struct list_head file_list;
+	spinlock_t file_lock;
+	wait_queue_head_t file_wait;
+
+	bool removing;
+};
+
+/* VSP messages */
+enum az_blob_vsp_request_type {
+	AZ_BLOB_DRIVER_REQUEST_FIRST     = 0x100,
+	AZ_BLOB_DRIVER_USER_REQUEST      = 0x100,
+	AZ_BLOB_DRIVER_REGISTER_BUFFER   = 0x101,
+	AZ_BLOB_DRIVER_DEREGISTER_BUFFER = 0x102,
+};
+
+/* VSC->VSP request */
+struct az_blob_vsp_request {
+	u32 version;
+	u32 timeout_ms;
+	u32 data_buffer_offset;
+	u32 data_buffer_length;
+	u32 data_buffer_valid;
+	u32 operation_type;
+	u32 request_buffer_offset;
+	u32 request_buffer_length;
+	u32 response_buffer_offset;
+	u32 response_buffer_length;
+	guid_t transaction_id;
+} __packed;
+
+/* VSP->VSC response */
+struct az_blob_vsp_response {
+	u32 length;
+	u32 error;
+	u32 response_len;
+} __packed;
+
+struct az_blob_vsp_request_ctx {
+	struct list_head list;
+	struct completion wait_vsp;
+	struct az_blob_request_sync *request;
+};
+
+struct az_blob_file_ctx {
+	struct list_head list;
+
+	/* List of pending requests to VSP */
+	struct list_head vsp_pending_requests;
+	spinlock_t vsp_pending_lock;
+	wait_queue_head_t wait_vsp_pending;
+};
+
+/* The maximum number of pages we can pass to VSP in a single packet */
+#define AZ_BLOB_MAX_PAGES 8192
+
+#ifdef CONFIG_DEBUG_FS
+struct dentry *az_blob_debugfs_root;
+#endif
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size (bytes)");
+
+static const struct hv_vmbus_device_id id_table[] = {
+	{ HV_AZURE_BLOB_GUID,
+	  .driver_data = 0
+	},
+	{ },
+};
+
+#define AZ_ERR 0
+#define AZ_WARN 1
+#define AZ_DBG 2
+static int log_level = AZ_ERR;
+module_param(log_level, int, 0644);
+MODULE_PARM_DESC(log_level,
+	"Log level: 0 - Error (default), 1 - Warning, 2 - Debug.");
+
+static uint device_queue_depth = 1024;
+module_param(device_queue_depth, uint, 0444);
+MODULE_PARM_DESC(device_queue_depth,
+	"System level max queue depth for this device");
+
+#define az_blob_log(level, fmt, args...)	\
+do {	\
+	if (level <= log_level)	\
+		pr_err("%s:%d " fmt, __func__, __LINE__, ##args);	\
+} while (0)
+
+#define az_blob_dbg(fmt, args...) az_blob_log(AZ_DBG, fmt, ##args)
+#define az_blob_warn(fmt, args...) az_blob_log(AZ_WARN, fmt, ##args)
+#define az_blob_err(fmt, args...) az_blob_log(AZ_ERR, fmt, ##args)
+
+static void az_blob_on_channel_callback(void *context)
+{
+	struct vmbus_channel *channel = (struct vmbus_channel *)context;
+	const struct vmpacket_descriptor *desc;
+
+	az_blob_dbg("entering interrupt from vmbus\n");
+	foreach_vmbus_pkt(desc, channel) {
+		struct az_blob_vsp_request_ctx *request_ctx;
+		struct az_blob_vsp_response *response;
+		u64 cmd_rqst = vmbus_request_addr(&channel->requestor,
+					desc->trans_id);
+		if (cmd_rqst == VMBUS_RQST_ERROR) {
+			az_blob_err("incorrect transaction id %llu\n",
+				desc->trans_id);
+			continue;
+		}
+		request_ctx = (struct az_blob_vsp_request_ctx *) cmd_rqst;
+		response = hv_pkt_data(desc);
+
+		az_blob_dbg("got response for request %pUb status %u "
+			"response_len %u\n",
+			&request_ctx->request->guid, response->error,
+			response->response_len);
+		request_ctx->request->response.status = response->error;
+		request_ctx->request->response.response_len =
+			response->response_len;
+		complete(&request_ctx->wait_vsp);
+	}
+
+}
+
+static int az_blob_fop_open(struct inode *inode, struct file *file)
+{
+	struct az_blob_file_ctx *file_ctx;
+	unsigned long flags;
+
+
+	file_ctx = kzalloc(sizeof(*file_ctx), GFP_KERNEL);
+	if (!file_ctx)
+		return -ENOMEM;
+
+
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);

drivers/hv/azure_blob.c line 167

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	if (az_blob_dev.removing) {
+		spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+		kfree(file_ctx);
+		return -ENODEV;
+	}
+	list_add_tail(&file_ctx->list, &az_blob_dev.file_list);
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	INIT_LIST_HEAD(&file_ctx->vsp_pending_requests);
+	init_waitqueue_head(&file_ctx->wait_vsp_pending);

It feels a little weird to be initializing fields in the file_ctx *after* it has been added to the file list. But I guess it's OK since no requests can be issued and the file can't be closed until after this open call completes.

Quoted 147 lines
drivers/hv/azure_blob.c new file +147 -0
+	file->private_data = file_ctx;
+
+	return 0;
+}
+
+static int az_blob_fop_release(struct inode *inode, struct file *file)
+{
+	struct az_blob_file_ctx *file_ctx = file->private_data;
+	unsigned long flags;
+
+	wait_event(file_ctx->wait_vsp_pending,
+		list_empty(&file_ctx->vsp_pending_requests));
+
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);
+	list_del(&file_ctx->list);
+	if (list_empty(&az_blob_dev.file_list))
+		wake_up(&az_blob_dev.file_wait);
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	return 0;
+}
+
+static inline bool az_blob_safe_file_access(struct file *file)
+{
+	return file->f_cred == current_cred() && !uaccess_kernel();
+}
+
+static int get_buffer_pages(int rw, void __user *buffer, u32 buffer_len,
+	struct page ***ppages, size_t *start, size_t *num_pages)
+{
+	struct iovec iov;
+	struct iov_iter iter;
+	int ret;
+	ssize_t result;
+	struct page **pages;
+
+	ret = import_single_range(rw, buffer, buffer_len, &iov, &iter);
+	if (ret) {
+		az_blob_dbg("request buffer access error %d\n", ret);
+		return ret;
+	}
+	az_blob_dbg("iov_iter type %d offset %lu count %lu nr_segs %lu\n",
+		iter.type, iter.iov_offset, iter.count, iter.nr_segs);
+
+	result = iov_iter_get_pages_alloc(&iter, &pages, buffer_len, start);
+	if (result < 0) {
+		az_blob_dbg("failed to pin user pages result=%ld\n", result);
+		return result;
+	}
+	if (result != buffer_len) {
+		az_blob_dbg("can't pin user pages requested %d got %ld\n",
+			buffer_len, result);
+		return -EFAULT;
+	}
+
+	*ppages = pages;
+	*num_pages = (result + *start + PAGE_SIZE - 1) / PAGE_SIZE;
+	return 0;
+}
+
+static void fill_in_page_buffer(u64 *pfn_array,
+	int *index, struct page **pages, unsigned long num_pages)
+{
+	int i, page_idx = *index;
+
+	for (i = 0; i < num_pages; i++)
+		pfn_array[page_idx++] = page_to_pfn(pages[i]);
+	*index = page_idx;
+}
+
+static void free_buffer_pages(size_t num_pages, struct page **pages)
+{
+	unsigned long i;
+
+	for (i = 0; i < num_pages; i++)
+		if (pages[i])
+			put_page(pages[i]);
+	kvfree(pages);
+}
+
+static long az_blob_ioctl_user_request(struct file *filp, unsigned long arg)
+{
+	struct az_blob_device *dev = &az_blob_dev;
+	struct az_blob_file_ctx *file_ctx = filp->private_data;
+	char __user *argp = (char __user *) arg;
+	struct az_blob_request_sync request;
+	struct az_blob_vsp_request_ctx request_ctx;
+	unsigned long flags;
+	int ret;
+	size_t request_start, request_num_pages = 0;
+	size_t response_start, response_num_pages = 0;
+	size_t data_start, data_num_pages = 0, total_num_pages;
+	struct page **request_pages = NULL, **response_pages = NULL;
+	struct page **data_pages = NULL;
+	struct vmbus_packet_mpb_array *desc;
+	u64 *pfn_array;
+	int desc_size;
+	int page_idx;
+	struct az_blob_vsp_request *vsp_request;
+
+	/* Fast fail if device is being removed */
+	if (dev->removing)
+		return -ENODEV;
+
+	if (!az_blob_safe_file_access(filp)) {
+		az_blob_dbg("process %d(%s) changed security contexts after"
+			" opening file descriptor\n",
+			task_tgid_vnr(current), current->comm);
+		return -EACCES;
+	}
+
+	if (copy_from_user(&request, argp, sizeof(request))) {
+		az_blob_dbg("don't have permission to user provided buffer\n");
+		return -EFAULT;
+	}
+
+	az_blob_dbg("az_blob ioctl request guid %pUb timeout %u request_len %u"
+		" response_len %u data_len %u request_buffer %llx "
+		"response_buffer %llx data_buffer %llx\n",
+		&request.guid, request.timeout, request.request_len,
+		request.response_len, request.data_len, request.request_buffer,
+		request.response_buffer, request.data_buffer);
+
+	if (!request.request_len || !request.response_len)
+		return -EINVAL;
+
+	if (request.data_len && request.data_len < request.data_valid)
+		return -EINVAL;
+
+	init_completion(&request_ctx.wait_vsp);
+	request_ctx.request = &request;
+
+	/*
+	 * Need to set rw to READ to have page table set up for passing to VSP.
+	 * Setting it to WRITE will cause the page table entry not allocated
+	 * as it's waiting on Copy-On-Write on next page fault. This doesn't
+	 * work in this scenario because VSP wants all the pages to be present.
+	 */
+	ret = get_buffer_pages(READ, (void __user *) request.request_buffer,
+		request.request_len, &request_pages, &request_start,
+		&request_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	ret = get_buffer_pages(READ, (void __user *) request.response_buffer,
+		request.response_len, &response_pages, &response_start,
+		&response_num_pages);

drivers/hv/azure_blob.c line 324

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	if (ret)
+		goto get_user_page_failed;
+
+	if (request.data_len) {
+		ret = get_buffer_pages(READ,
+			(void __user *) request.data_buffer, request.data_len,
+			&data_pages, &data_start, &data_num_pages);
+		if (ret)
+			goto get_user_page_failed;
+	}

It still seems to me that request.request_len, request.response_len, and request.data_len should be validated *before* get_buffer_pages() is called. While the total number of pages that pinned is validated below against AZ_BLOB_MAX_PAGES, bad values in the *_len fields could pin up to 12 Gbytes of memory before the check against AZ_BLOB_MAX_PAGES
happens. The pre-validation would make sure that each *_len value
is less than AZ_BLOB_MAX_PAGES, so that the max amount of memory that could get pinned is 96 Mbytes.

Also, are the *_len values required to be a multiple of PAGE_SIZE? I'm assuming not, and that any number of bytes is acceptable. If there is some requirement such as being a multiple of 512 or something like that, it should also be part of the pre-validation.

Quoted 14 lines
drivers/hv/azure_blob.c new file +14 -0
+
+	total_num_pages = request_num_pages + response_num_pages +
+				data_num_pages;
+	if (total_num_pages > AZ_BLOB_MAX_PAGES) {
+		az_blob_dbg("number of DMA pages %lu buffer exceeding %u\n",
+			total_num_pages, AZ_BLOB_MAX_PAGES);
+		ret = -EINVAL;
+		goto get_user_page_failed;
+	}
+
+	/* Construct a VMBUS packet and send it over to VSP */
+	desc_size = sizeof(struct vmbus_packet_mpb_array) +
+			sizeof(u64) * total_num_pages;
+	desc = kzalloc(desc_size, GFP_KERNEL);

drivers/hv/azure_blob.c line 348

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	vsp_request = kzalloc(sizeof(*vsp_request), GFP_KERNEL);
+	if (!desc || !vsp_request) {
+		kfree(desc);
+		kfree(vsp_request);
+		ret = -ENOMEM;
+		goto get_user_page_failed;
+	}
+
+	desc->range.offset = 0;
+	desc->range.len = total_num_pages * PAGE_SIZE;

The range.len value is always a multiple of PAGE_SIZE. Assuming that the request.*_len values are not required to be multiples of PAGE_SIZE, then the sum of the buffer_length fields in the vsp_request may not add up to range.len. Presumably the VSP is OK with that ....

Quoted 12 lines
drivers/hv/azure_blob.c new file +12 -0
+	pfn_array = desc->range.pfn_array;
+	page_idx = 0;
+
+	if (request.data_len) {
+		fill_in_page_buffer(pfn_array, &page_idx, data_pages,
+			data_num_pages);
+		vsp_request->data_buffer_offset = data_start;
+		vsp_request->data_buffer_length = request.data_len;
+		vsp_request->data_buffer_valid = request.data_valid;
+	}
+
+	fill_in_page_buffer(pfn_array, &page_idx, request_pages,

drivers/hv/azure_blob.c line 370

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		request_num_pages);
+	vsp_request->request_buffer_offset = request_start +
+						data_num_pages * PAGE_SIZE;
+	vsp_request->request_buffer_length = request.request_len;
+
+	fill_in_page_buffer(pfn_array, &page_idx, response_pages,
+		response_num_pages);
+	vsp_request->response_buffer_offset = response_start +
+		(data_num_pages + request_num_pages) * PAGE_SIZE;
+	vsp_request->response_buffer_length = request.response_len;

Interestingly, if any of the data or request or response are adjacent in user memory, but the boundary is not on a page boundary, the same page may appear in two successive pfn_array entries. Again, I'm assuming that's OK with the VSP. It appears that the VSP just treats the three memory areas as completely independent of each other but defined by sequential entries in the pfn_array.

Quoted 182 lines
drivers/hv/azure_blob.c new file +182 -0
+
+	vsp_request->version = 0;
+	vsp_request->timeout_ms = request.timeout;
+	vsp_request->operation_type = AZ_BLOB_DRIVER_USER_REQUEST;
+	guid_copy(&vsp_request->transaction_id, &request.guid);
+
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_add_tail(&request_ctx.list, &file_ctx->vsp_pending_requests);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+	az_blob_dbg("sending request to VSP\n");
+	az_blob_dbg("desc_size %u desc->range.len %u desc->range.offset %u\n",
+		desc_size, desc->range.len, desc->range.offset);
+	az_blob_dbg("vsp_request data_buffer_offset %u data_buffer_length %u "
+		"data_buffer_valid %u request_buffer_offset %u "
+		"request_buffer_length %u response_buffer_offset %u "
+		"response_buffer_length %u\n",
+		vsp_request->data_buffer_offset,
+		vsp_request->data_buffer_length,
+		vsp_request->data_buffer_valid,
+		vsp_request->request_buffer_offset,
+		vsp_request->request_buffer_length,
+		vsp_request->response_buffer_offset,
+		vsp_request->response_buffer_length);
+
+	ret = vmbus_sendpacket_mpb_desc(dev->device->channel, desc, desc_size,
+		vsp_request, sizeof(*vsp_request), (u64) &request_ctx);
+
+	kfree(desc);
+	kfree(vsp_request);
+	if (ret)
+		goto vmbus_send_failed;
+
+	wait_for_completion(&request_ctx.wait_vsp);
+
+	/*
+	 * At this point, the response is already written to request
+	 * by VMBUS completion handler, copy them to user-mode buffers
+	 * and return to user-mode
+	 */
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.status),
+			&request.response.status,
+			sizeof(request.response.status))) {
+		ret = -EFAULT;
+		goto vmbus_send_failed;
+	}
+
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.response_len),
+			&request.response.response_len,
+			sizeof(request.response.response_len)))
+		ret = -EFAULT;
+
+vmbus_send_failed:
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_del(&request_ctx.list);
+	if (list_empty(&file_ctx->vsp_pending_requests))
+		wake_up(&file_ctx->wait_vsp_pending);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+get_user_page_failed:
+	free_buffer_pages(request_num_pages, request_pages);
+	free_buffer_pages(response_num_pages, response_pages);
+	free_buffer_pages(data_num_pages, data_pages);
+
+	return ret;
+}
+
+static long az_blob_fop_ioctl(struct file *filp, unsigned int cmd,
+	unsigned long arg)
+{
+	long ret = -EIO;
+
+	switch (cmd) {
+	case IOCTL_AZ_BLOB_DRIVER_USER_REQUEST:
+		if (_IOC_SIZE(cmd) != sizeof(struct az_blob_request_sync))
+			return -EINVAL;
+		ret = az_blob_ioctl_user_request(filp, arg);
+		break;
+
+	default:
+		az_blob_dbg("unrecognized IOCTL code %u\n", cmd);
+	}
+
+	return ret;
+}
+
+static const struct file_operations az_blob_client_fops = {
+	.owner	= THIS_MODULE,
+	.open	= az_blob_fop_open,
+	.unlocked_ioctl = az_blob_fop_ioctl,
+	.release = az_blob_fop_release,
+};
+
+static struct miscdevice az_blob_misc_device = {
+	MISC_DYNAMIC_MINOR,
+	"azure_blob",
+	&az_blob_client_fops,
+};
+
+static int az_blob_show_pending_requests(struct seq_file *m, void *v)
+{
+	unsigned long flags, flags2;
+	struct az_blob_vsp_request_ctx *request_ctx;
+	struct az_blob_file_ctx *file_ctx;
+
+	seq_puts(m, "List of pending requests\n");
+	seq_puts(m, "UUID request_len response_len data_len "
+		"request_buffer response_buffer data_buffer\n");
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);
+	list_for_each_entry(file_ctx, &az_blob_dev.file_list, list) {
+		spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags2);
+		list_for_each_entry(request_ctx,
+				&file_ctx->vsp_pending_requests, list) {
+			seq_printf(m, "%pUb ", &request_ctx->request->guid);
+			seq_printf(m, "%u ", request_ctx->request->request_len);
+			seq_printf(m,
+				"%u ", request_ctx->request->response_len);
+			seq_printf(m, "%u ", request_ctx->request->data_len);
+			seq_printf(m,
+				"%llx ", request_ctx->request->request_buffer);
+			seq_printf(m,
+				"%llx ", request_ctx->request->response_buffer);
+			seq_printf(m,
+				"%llx\n", request_ctx->request->data_buffer);
+		}
+		spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags2);
+	}
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	return 0;
+}
+
+static int az_blob_debugfs_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, az_blob_show_pending_requests, NULL);
+}
+
+static const struct file_operations az_blob_debugfs_fops = {
+	.open		= az_blob_debugfs_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release
+};
+
+static void az_blob_remove_device(struct az_blob_device *dev)
+{
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */
+}
+
+static int az_blob_create_device(struct az_blob_device *dev)
+{
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else
+		az_blob_warn("failed to create debugfs root\n");

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+#endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32 ring_size)
+{
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin lock.
6) Before az_blob_fop_open() releases the spin lock, az block_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re-initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in az_blob_drv_init().

Quoted 144 lines
drivers/hv/azure_blob.c new file +93 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size, NULL, 0,
+			az_blob_on_channel_callback, device->channel);
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device)
+{
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel);
+}
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id)
+{
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device, az_blob_ringbuffer_size);
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev)
+{
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void)
+{
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void)
+{
+	vmbus_driver_unregister(&az_blob_drv);
+}
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Microsoft Azure Blob driver");
+module_init(az_blob_drv_init);
+module_exit(az_blob_drv_exit);
drivers/hv/channel_mgmt.c +7 -0
index 0c75662..436fdeb 100644
@@ -154,6 +154,13 @@
 	  .allowed_in_isolated = false,
 	},
 
+	/* Azure Blob */
+	{ .dev_type = HV_AZURE_BLOB,
+	  HV_AZURE_BLOB_GUID,
+	  .perf_device = false,
+	  .allowed_in_isolated = false,
+	},
+
 	/* Unknown GUID */
 	{ .dev_type = HV_UNKNOWN,
 	  .perf_device = false,
include/linux/hyperv.h +9 -0
index d1e59db..ac31362 100644
@@ -772,6 +772,7 @@ enum vmbus_device_type {
 	HV_FCOPY,
 	HV_BACKUP,
 	HV_DM,
+	HV_AZURE_BLOB,
 	HV_UNKNOWN,
 };
 
@@ -1350,6 +1351,14 @@ int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,
 			  0x72, 0xe2, 0xff, 0xb1, 0xdc, 0x7f)
 
 /*
+ * Azure Blob GUID
+ * {0590b792-db64-45cc-81db-b8d70c577c9e}
+ */
+#define HV_AZURE_BLOB_GUID \
+	.guid = GUID_INIT(0x0590b792, 0xdb64, 0x45cc, 0x81, 0xdb, \
+			  0xb8, 0xd7, 0x0c, 0x57, 0x7c, 0x9e)
+
+/*
  * Shutdown GUID
  * {0e0b6031-5213-4934-818b-38d90ced39db}
  */
include/uapi/misc/azure_blob.h new file +0 -0
new file mode 100644
index 0000000..29413fc

include/uapi/misc/azure_blob.h line 5

Quoted 5 lines
include/uapi/misc/azure_blob.h new file +5 -0
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* Copyright (c) 2021, Microsoft Corporation. */
+
+#ifndef _AZ_BLOB_H
+#define _AZ_BLOB_H

Seems like a #include of asm/ioctl.h (or something similar) is needed so that _IOWR is defined. Also, a #include is needed for __u32, __aligned_u64, guid_t, etc.

Quoted 28 lines
include/uapi/misc/azure_blob.h new file +26 -1
+
+/* user-mode sync request sent through ioctl */
+struct az_blob_request_sync_response {
+	__u32 status;
+	__u32 response_len;
+};
+
+struct az_blob_request_sync {
+	guid_t guid;
+	__u32 timeout;
+	__u32 request_len;
+	__u32 response_len;
+	__u32 data_len;
+	__u32 data_valid;
+	__aligned_u64 request_buffer;
+	__aligned_u64 response_buffer;
+	__aligned_u64 data_buffer;
+	struct az_blob_request_sync_response response;
+};
+
+#define AZ_BLOB_MAGIC_NUMBER	'R'
+#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \
+		_IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \
+			struct az_blob_request_sync)
+
+#endif /* define _AZ_BLOB_H */
--

1.8.3.1

#8 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 450 lines

Subject: RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: [email protected] <[email protected]> Sent: Friday, June 25, 2021 11:30 PM

Quoted 4 lines

Azure Blob storage provides scalable and durable data storage for Azure.
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fazu
re.microsoft.com%2Fen-

us%2Fservices%2Fstorage%2Fblobs%2F&amp;data=04%7

Quoted 1 line

C01%7Clongli%40microsoft.com%7Cd8cabb55b9f24e98fefe08d93a7651e2% 7C72f9

Quoted 1 line

88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637605102139607593%7C Unknown%7C

Quoted 1 line

TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ XVC

Quoted 1 line

I6Mn0%3D%7C1000&amp;sdata=GfpN0Gb5IkkdeSe4e0zaW3y7iRYAoRFVicYIy L4F9T8%

Quoted 5 lines

3D&amp;reserved=0)

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server

access.

Documentation/userspace-api/ioctl/ioctl-number.rst Code Seq# Include File

Quoted 38 lines
Cc Jonathan Corbet <[email protected]>
Cc "K. Y. Srinivasan" <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Bjorn Andersson <[email protected]>
Cc Hans de Goede <[email protected]>
Cc Dan Williams <[email protected]>
Cc Maximilian Luz <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Ben Widawsky <[email protected]>
Cc Jiri Slaby <[email protected]>
Cc Andra Paraschiv <[email protected]>
Cc Siddharth Gupta <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 Documentation/userspace-api/ioctl/ioctl-number.rst |   2 +
 drivers/hv/Kconfig                                 |  10 +
 drivers/hv/Makefile                                |   1 +
 drivers/hv/azure_blob.c                            | 655 +++++++++++++++++++++
 drivers/hv/channel_mgmt.c                          |   7 +
 include/linux/hyperv.h                             |   9 +
 include/uapi/misc/azure_blob.h                     |  31 +
 7 files changed, 715 insertions(+)
 create mode 100644 drivers/hv/azure_blob.c  create mode 100644

include/uapi/misc/azure_blob.h

Documentation/userspace-api/ioctl/ioctl-number.rst +0 -0
index 9bfc2b5..d3c2a90 100644
@@ -180,6 +180,8 @@ Code  Seq#    Include File

Comments

Documentation/userspace-api/ioctl/ioctl-number.rst Code Seq# Include File Comments

Quoted 4 lines
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
 'R'   01     linux/rfkill.h                                          conflict!
 'R'   C0-DF  net/bluetooth/rfcomm.h
 'R'   E0     uapi/linux/fsl_mc.h
+'R'   F0-FF  uapi/misc/azure_blob.h                                  Microsoft Azure Blob

driver

Quoted 11 lines
Documentation/userspace-api/ioctl/ioctl-number.rst +2 -0
+
+<mailto:[email protected]>
 'S'   all    linux/cdrom.h                                           conflict!
 'S'   80-81  scsi/scsi_ioctl.h                                       conflict!
 'S'   82-FF  scsi/scsi.h                                             conflict!
drivers/hv/Kconfig index renamed from drivers/hv/Kconfig +0 -0

66c794d..e08b8d3 100644

drivers/hv/Kconfig +0 -0
@@ -27,4 +27,14 @@ config HYPERV_BALLOON
 	help

drivers/hv/Kconfig config HYPERV_BALLOON · line 37

Quoted 10 lines
drivers/hv/Kconfig +8 -0
 	  Select this option to enable Hyper-V Balloon driver.
 
+config HYPERV_AZURE_BLOB
+	tristate "Microsoft Azure Blob driver"
+	depends on HYPERV && X86_64
+	help
+	  Select this option to enable Microsoft Azure Blob driver.
+
+	  This driver supports accelerated Microsoft Azure Blob access.
+	  To compile this driver as a module, choose M here. The module will

be

Quoted 174 lines
drivers/hv/Kconfig +2 -0
+	  called azure_blob.
+
 endmenu
drivers/hv/Makefile index renamed from drivers/hv/Makefile +0 -0

94daf82..a322575 100644

drivers/hv/Makefile +1 -0
@@ -2,6 +2,7 @@
 obj-$(CONFIG_HYPERV)		+= hv_vmbus.o
 obj-$(CONFIG_HYPERV_UTILS)	+= hv_utils.o
 obj-$(CONFIG_HYPERV_BALLOON)	+= hv_balloon.o
+obj-$(CONFIG_HYPERV_AZURE_BLOB)	+= azure_blob.o
 
 CFLAGS_hv_trace.o = -I$(src)
 CFLAGS_hv_balloon.o = -I$(src)
drivers/hv/azure_blob.c new renamed from drivers/hv/azure_blob.c +0 -0

file mode 100644 index 0000000..6c8f957

drivers/hv/azure_blob.c new file +154 -0
@@ -0,0 +1,655 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/* Copyright (c) 2021, Microsoft Corporation. */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/cred.h>
+#include <linux/debugfs.h>
+#include <linux/pagemap.h>
+#include <linux/hyperv.h>
+#include <linux/miscdevice.h>
+#include <linux/uio.h>
+#include <uapi/misc/azure_blob.h>
+
+struct az_blob_device {
+	struct hv_device *device;
+
+	/* Opened files maintained by this device */
+	struct list_head file_list;
+	spinlock_t file_lock;
+	wait_queue_head_t file_wait;
+
+	bool removing;
+};
+
+/* VSP messages */
+enum az_blob_vsp_request_type {
+	AZ_BLOB_DRIVER_REQUEST_FIRST     = 0x100,
+	AZ_BLOB_DRIVER_USER_REQUEST      = 0x100,
+	AZ_BLOB_DRIVER_REGISTER_BUFFER   = 0x101,
+	AZ_BLOB_DRIVER_DEREGISTER_BUFFER = 0x102, };
+
+/* VSC->VSP request */
+struct az_blob_vsp_request {
+	u32 version;
+	u32 timeout_ms;
+	u32 data_buffer_offset;
+	u32 data_buffer_length;
+	u32 data_buffer_valid;
+	u32 operation_type;
+	u32 request_buffer_offset;
+	u32 request_buffer_length;
+	u32 response_buffer_offset;
+	u32 response_buffer_length;
+	guid_t transaction_id;
+} __packed;
+
+/* VSP->VSC response */
+struct az_blob_vsp_response {
+	u32 length;
+	u32 error;
+	u32 response_len;
+} __packed;
+
+struct az_blob_vsp_request_ctx {
+	struct list_head list;
+	struct completion wait_vsp;
+	struct az_blob_request_sync *request; };
+
+struct az_blob_file_ctx {
+	struct list_head list;
+
+	/* List of pending requests to VSP */
+	struct list_head vsp_pending_requests;
+	spinlock_t vsp_pending_lock;
+	wait_queue_head_t wait_vsp_pending;
+};
+
+/* The maximum number of pages we can pass to VSP in a single packet
+*/ #define AZ_BLOB_MAX_PAGES 8192
+
+#ifdef CONFIG_DEBUG_FS
+struct dentry *az_blob_debugfs_root;
+#endif
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size
+(bytes)");
+
+static const struct hv_vmbus_device_id id_table[] = {
+	{ HV_AZURE_BLOB_GUID,
+	  .driver_data = 0
+	},
+	{ },
+};
+
+#define AZ_ERR 0
+#define AZ_WARN 1
+#define AZ_DBG 2
+static int log_level = AZ_ERR;
+module_param(log_level, int, 0644);
+MODULE_PARM_DESC(log_level,
+	"Log level: 0 - Error (default), 1 - Warning, 2 - Debug.");
+
+static uint device_queue_depth = 1024;
+module_param(device_queue_depth, uint, 0444);
+MODULE_PARM_DESC(device_queue_depth,
+	"System level max queue depth for this device");
+
+#define az_blob_log(level, fmt, args...)	\
+do {	\
+	if (level <= log_level)	\
+		pr_err("%s:%d " fmt, __func__, __LINE__, ##args);	\
+} while (0)
+
+#define az_blob_dbg(fmt, args...) az_blob_log(AZ_DBG, fmt, ##args)
+#define az_blob_warn(fmt, args...) az_blob_log(AZ_WARN, fmt, ##args)
+#define az_blob_err(fmt, args...) az_blob_log(AZ_ERR, fmt, ##args)
+
+static void az_blob_on_channel_callback(void *context) {
+	struct vmbus_channel *channel = (struct vmbus_channel *)context;
+	const struct vmpacket_descriptor *desc;
+
+	az_blob_dbg("entering interrupt from vmbus\n");
+	foreach_vmbus_pkt(desc, channel) {
+		struct az_blob_vsp_request_ctx *request_ctx;
+		struct az_blob_vsp_response *response;
+		u64 cmd_rqst = vmbus_request_addr(&channel->requestor,
+					desc->trans_id);
+		if (cmd_rqst == VMBUS_RQST_ERROR) {
+			az_blob_err("incorrect transaction id %llu\n",
+				desc->trans_id);
+			continue;
+		}
+		request_ctx = (struct az_blob_vsp_request_ctx *) cmd_rqst;
+		response = hv_pkt_data(desc);
+
+		az_blob_dbg("got response for request %pUb status %u "
+			"response_len %u\n",
+			&request_ctx->request->guid, response->error,
+			response->response_len);
+		request_ctx->request->response.status = response->error;
+		request_ctx->request->response.response_len =
+			response->response_len;
+		complete(&request_ctx->wait_vsp);
+	}
+
+}
+
+static int az_blob_fop_open(struct inode *inode, struct file *file) {
+	struct az_blob_file_ctx *file_ctx;
+	unsigned long flags;
+
+
+	file_ctx = kzalloc(sizeof(*file_ctx), GFP_KERNEL);
+	if (!file_ctx)
+		return -ENOMEM;
+
+
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);

drivers/hv/azure_blob.c line 164

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	if (az_blob_dev.removing) {
+		spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+		kfree(file_ctx);
+		return -ENODEV;
+	}
+	list_add_tail(&file_ctx->list, &az_blob_dev.file_list);
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	INIT_LIST_HEAD(&file_ctx->vsp_pending_requests);
+	init_waitqueue_head(&file_ctx->wait_vsp_pending);

It feels a little weird to be initializing fields in the file_ctx *after* it has been added to the file list. But I guess it's OK since no requests can be issued and the file can't be closed until after this open call completes.

Quoted 92 lines
drivers/hv/azure_blob.c new file +92 -0
+	file->private_data = file_ctx;
+
+	return 0;
+}
+
+static int az_blob_fop_release(struct inode *inode, struct file
+*file) {
+	struct az_blob_file_ctx *file_ctx = file->private_data;
+	unsigned long flags;
+
+	wait_event(file_ctx->wait_vsp_pending,
+		list_empty(&file_ctx->vsp_pending_requests));
+
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);
+	list_del(&file_ctx->list);
+	if (list_empty(&az_blob_dev.file_list))
+		wake_up(&az_blob_dev.file_wait);
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	return 0;
+}
+
+static inline bool az_blob_safe_file_access(struct file *file) {
+	return file->f_cred == current_cred() && !uaccess_kernel(); }
+
+static int get_buffer_pages(int rw, void __user *buffer, u32 buffer_len,
+	struct page ***ppages, size_t *start, size_t *num_pages) {
+	struct iovec iov;
+	struct iov_iter iter;
+	int ret;
+	ssize_t result;
+	struct page **pages;
+
+	ret = import_single_range(rw, buffer, buffer_len, &iov, &iter);
+	if (ret) {
+		az_blob_dbg("request buffer access error %d\n", ret);
+		return ret;
+	}
+	az_blob_dbg("iov_iter type %d offset %lu count %lu nr_segs %lu\n",
+		iter.type, iter.iov_offset, iter.count, iter.nr_segs);
+
+	result = iov_iter_get_pages_alloc(&iter, &pages, buffer_len, start);
+	if (result < 0) {
+		az_blob_dbg("failed to pin user pages result=%ld\n", result);
+		return result;
+	}
+	if (result != buffer_len) {
+		az_blob_dbg("can't pin user pages requested %d got %ld\n",
+			buffer_len, result);
+		return -EFAULT;
+	}
+
+	*ppages = pages;
+	*num_pages = (result + *start + PAGE_SIZE - 1) / PAGE_SIZE;
+	return 0;
+}
+
+static void fill_in_page_buffer(u64 *pfn_array,
+	int *index, struct page **pages, unsigned long num_pages) {
+	int i, page_idx = *index;
+
+	for (i = 0; i < num_pages; i++)
+		pfn_array[page_idx++] = page_to_pfn(pages[i]);
+	*index = page_idx;
+}
+
+static void free_buffer_pages(size_t num_pages, struct page **pages)
+{
+	unsigned long i;
+
+	for (i = 0; i < num_pages; i++)
+		if (pages[i])
+			put_page(pages[i]);
+	kvfree(pages);
+}
+
+static long az_blob_ioctl_user_request(struct file *filp, unsigned
+long arg) {
+	struct az_blob_device *dev = &az_blob_dev;
+	struct az_blob_file_ctx *file_ctx = filp->private_data;
+	char __user *argp = (char __user *) arg;
+	struct az_blob_request_sync request;
+	struct az_blob_vsp_request_ctx request_ctx;
+	unsigned long flags;
+	int ret;
+	size_t request_start, request_num_pages = 0;
+	size_t response_start, response_num_pages = 0;
+	size_t data_start, data_num_pages = 0, total_num_pages;
+	struct page **request_pages = NULL, **response_pages = NULL;
+	struct page **data_pages = NULL;
+	struct vmbus_packet_mpb_array *desc;
+	u64 *pfn_array;

drivers/hv/azure_blob.c

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	int desc_size;
+	int page_idx;
+	struct az_blob_vsp_request *vsp_request;
+
+	/* Fast fail if device is being removed */
+	if (dev->removing)
+		return -ENODEV;
+
+	if (!az_blob_safe_file_access(filp)) {
+		az_blob_dbg("process %d(%s) changed security contexts

after"

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+			" opening file descriptor\n",
+			task_tgid_vnr(current), current->comm);
+		return -EACCES;
+	}
+
+	if (copy_from_user(&request, argp, sizeof(request))) {
+		az_blob_dbg("don't have permission to user provided

buffer\n");

drivers/hv/azure_blob.c

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		return -EFAULT;
+	}
+
+	az_blob_dbg("az_blob ioctl request guid %pUb timeout %u

request_len %u"

drivers/hv/azure_blob.c

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		" response_len %u data_len %u request_buffer %llx "
+		"response_buffer %llx data_buffer %llx\n",
+		&request.guid, request.timeout, request.request_len,
+		request.response_len, request.data_len,

request.request_buffer,

Quoted 14 lines
drivers/hv/azure_blob.c new file +14 -0
+		request.response_buffer, request.data_buffer);
+
+	if (!request.request_len || !request.response_len)
+		return -EINVAL;
+
+	if (request.data_len && request.data_len < request.data_valid)
+		return -EINVAL;
+
+	init_completion(&request_ctx.wait_vsp);
+	request_ctx.request = &request;
+
+	/*
+	 * Need to set rw to READ to have page table set up for passing to VSP.
+	 * Setting it to WRITE will cause the page table entry not allocated

drivers/hv/azure_blob.c

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	 * as it's waiting on Copy-On-Write on next page fault. This doesn't
+	 * work in this scenario because VSP wants all the pages to be present.
+	 */
+	ret = get_buffer_pages(READ, (void __user *) request.request_buffer,
+		request.request_len, &request_pages, &request_start,
+		&request_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	ret = get_buffer_pages(READ, (void __user *)

request.response_buffer,

drivers/hv/azure_blob.c line 324

Quoted 12 lines
drivers/hv/azure_blob.c new file +12 -0
+		request.response_len, &response_pages, &response_start,
+		&response_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	if (request.data_len) {
+		ret = get_buffer_pages(READ,
+			(void __user *) request.data_buffer, request.data_len,
+			&data_pages, &data_start, &data_num_pages);
+		if (ret)
+			goto get_user_page_failed;
+	}

It still seems to me that request.request_len, request.response_len, and request.data_len should be validated *before* get_buffer_pages() is called. While the total number of pages that pinned is validated below against AZ_BLOB_MAX_PAGES, bad values in the *_len fields could pin up to 12 Gbytes of memory before the check against AZ_BLOB_MAX_PAGES
happens. The pre-validation would make sure that each *_len value
is less than AZ_BLOB_MAX_PAGES, so that the max amount of memory that could get pinned is 96 Mbytes.

Quoted 4 lines

Also, are the *_len values required to be a multiple of PAGE_SIZE? I'm assuming not, and that any number of bytes is acceptable. If there is some requirement such as being a multiple of 512 or something like that, it should also be part of the pre-validation.

*len values are not required to be aligned to page size or 512.

Adding pre-validation to those *len is a good idea. I'll make changes.

Quoted 23 lines

drivers/hv/azure_blob.c

Quoted 6 lines
drivers/hv/azure_blob.c new file +5 -0
+
+	total_num_pages = request_num_pages + response_num_pages +
+				data_num_pages;
+	if (total_num_pages > AZ_BLOB_MAX_PAGES) {
+		az_blob_dbg("number of DMA pages %lu buffer

exceeding %u\n",

Quoted 16 lines
drivers/hv/azure_blob.c new file +16 -0
+			total_num_pages, AZ_BLOB_MAX_PAGES);
+		ret = -EINVAL;
+		goto get_user_page_failed;
+	}
+
+	/* Construct a VMBUS packet and send it over to VSP */
+	desc_size = sizeof(struct vmbus_packet_mpb_array) +
+			sizeof(u64) * total_num_pages;
+	desc = kzalloc(desc_size, GFP_KERNEL);
+	vsp_request = kzalloc(sizeof(*vsp_request), GFP_KERNEL);
+	if (!desc || !vsp_request) {
+		kfree(desc);
+		kfree(vsp_request);
+		ret = -ENOMEM;
+		goto get_user_page_failed;
+	}
Quoted 8 lines

drivers/hv/azure_blob.c line 348

Quoted 3 lines
drivers/hv/azure_blob.c new file +3 -0
+
+	desc->range.offset = 0;
+	desc->range.len = total_num_pages * PAGE_SIZE;

The range.len value is always a multiple of PAGE_SIZE. Assuming that the request.*_len values are not required to be multiples of PAGE_SIZE, then the sum of the buffer_length fields in the vsp_request may not add up to range.len. Presumably the VSP is OK with that ....

range.len describes the length of whole array of page buffers sent to the host, the individual buffers within the page buffers are described by *_offset and *_length in the VSP packet.

Quoted 22 lines
Quoted 22 lines
drivers/hv/azure_blob.c new file +21 -0
+	pfn_array = desc->range.pfn_array;
+	page_idx = 0;
+
+	if (request.data_len) {
+		fill_in_page_buffer(pfn_array, &page_idx, data_pages,
+			data_num_pages);
+		vsp_request->data_buffer_offset = data_start;
+		vsp_request->data_buffer_length = request.data_len;
+		vsp_request->data_buffer_valid = request.data_valid;
+	}
+
+	fill_in_page_buffer(pfn_array, &page_idx, request_pages,
+		request_num_pages);
+	vsp_request->request_buffer_offset = request_start +
+						data_num_pages * PAGE_SIZE;
+	vsp_request->request_buffer_length = request.request_len;
+
+	fill_in_page_buffer(pfn_array, &page_idx, response_pages,
+		response_num_pages);
+	vsp_request->response_buffer_offset = response_start +
+		(data_num_pages + request_num_pages) * PAGE_SIZE;
Quoted 8 lines

drivers/hv/azure_blob.c line 370

Quoted 1 line
drivers/hv/azure_blob.c new file +1 -0
+	vsp_request->response_buffer_length = request.response_len;

Interestingly, if any of the data or request or response are adjacent in user memory, but the boundary is not on a page boundary, the same page may appear in two successive pfn_array entries. Again, I'm assuming that's OK with the VSP. It appears that the VSP just treats the three memory areas as completely independent of each other but defined by sequential entries in the pfn_array.

Yes the VSP is okay with that. We can send duplicate page numbers to the VSP if the buffers don't end with page boundary, and are next to each other.

Quoted 211 lines
Quoted 14 lines
drivers/hv/azure_blob.c new file +12 -0
+
+	vsp_request->version = 0;
+	vsp_request->timeout_ms = request.timeout;
+	vsp_request->operation_type = AZ_BLOB_DRIVER_USER_REQUEST;
+	guid_copy(&vsp_request->transaction_id, &request.guid);
+
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_add_tail(&request_ctx.list, &file_ctx->vsp_pending_requests);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+	az_blob_dbg("sending request to VSP\n");
+	az_blob_dbg("desc_size %u desc->range.len %u desc-

range.offset %u\n",

drivers/hv/azure_blob.c

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+		desc_size, desc->range.len, desc->range.offset);
+	az_blob_dbg("vsp_request data_buffer_offset %u

data_buffer_length %u "

drivers/hv/azure_blob.c

Quoted 12 lines
drivers/hv/azure_blob.c new file +12 -0
+		"data_buffer_valid %u request_buffer_offset %u "
+		"request_buffer_length %u response_buffer_offset %u "
+		"response_buffer_length %u\n",
+		vsp_request->data_buffer_offset,
+		vsp_request->data_buffer_length,
+		vsp_request->data_buffer_valid,
+		vsp_request->request_buffer_offset,
+		vsp_request->request_buffer_length,
+		vsp_request->response_buffer_offset,
+		vsp_request->response_buffer_length);
+
+	ret = vmbus_sendpacket_mpb_desc(dev->device->channel, desc,

desc_size,

Quoted 154 lines
drivers/hv/azure_blob.c new file +93 -0
+		vsp_request, sizeof(*vsp_request), (u64) &request_ctx);
+
+	kfree(desc);
+	kfree(vsp_request);
+	if (ret)
+		goto vmbus_send_failed;
+
+	wait_for_completion(&request_ctx.wait_vsp);
+
+	/*
+	 * At this point, the response is already written to request
+	 * by VMBUS completion handler, copy them to user-mode buffers
+	 * and return to user-mode
+	 */
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.status),
+			&request.response.status,
+			sizeof(request.response.status))) {
+		ret = -EFAULT;
+		goto vmbus_send_failed;
+	}
+
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.response_len),
+			&request.response.response_len,
+			sizeof(request.response.response_len)))
+		ret = -EFAULT;
+
+vmbus_send_failed:
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_del(&request_ctx.list);
+	if (list_empty(&file_ctx->vsp_pending_requests))
+		wake_up(&file_ctx->wait_vsp_pending);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+get_user_page_failed:
+	free_buffer_pages(request_num_pages, request_pages);
+	free_buffer_pages(response_num_pages, response_pages);
+	free_buffer_pages(data_num_pages, data_pages);
+
+	return ret;
+}
+
+static long az_blob_fop_ioctl(struct file *filp, unsigned int cmd,
+	unsigned long arg)
+{
+	long ret = -EIO;
+
+	switch (cmd) {
+	case IOCTL_AZ_BLOB_DRIVER_USER_REQUEST:
+		if (_IOC_SIZE(cmd) != sizeof(struct az_blob_request_sync))
+			return -EINVAL;
+		ret = az_blob_ioctl_user_request(filp, arg);
+		break;
+
+	default:
+		az_blob_dbg("unrecognized IOCTL code %u\n", cmd);
+	}
+
+	return ret;
+}
+
+static const struct file_operations az_blob_client_fops = {
+	.owner	= THIS_MODULE,
+	.open	= az_blob_fop_open,
+	.unlocked_ioctl = az_blob_fop_ioctl,
+	.release = az_blob_fop_release,
+};
+
+static struct miscdevice az_blob_misc_device = {
+	MISC_DYNAMIC_MINOR,
+	"azure_blob",
+	&az_blob_client_fops,
+};
+
+static int az_blob_show_pending_requests(struct seq_file *m, void *v)
+{
+	unsigned long flags, flags2;
+	struct az_blob_vsp_request_ctx *request_ctx;
+	struct az_blob_file_ctx *file_ctx;
+
+	seq_puts(m, "List of pending requests\n");
+	seq_puts(m, "UUID request_len response_len data_len "
+		"request_buffer response_buffer data_buffer\n");
+	spin_lock_irqsave(&az_blob_dev.file_lock, flags);
+	list_for_each_entry(file_ctx, &az_blob_dev.file_list, list) {
+		spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags2);
+		list_for_each_entry(request_ctx,
+				&file_ctx->vsp_pending_requests, list) {
+			seq_printf(m, "%pUb ", &request_ctx->request->guid);
+			seq_printf(m, "%u ", request_ctx->request-

request_len);

drivers/hv/azure_blob.c new file +5 -0
+			seq_printf(m,
+				"%u ", request_ctx->request->response_len);
+			seq_printf(m, "%u ", request_ctx->request->data_len);
+			seq_printf(m,
+				"%llx ", request_ctx->request-

request_buffer);

drivers/hv/azure_blob.c new file +2 -0
+			seq_printf(m,
+				"%llx ", request_ctx->request-

response_buffer);

drivers/hv/azure_blob.c new file +51 -0
+			seq_printf(m,
+				"%llx\n", request_ctx->request->data_buffer);
+		}
+		spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags2);
+	}
+	spin_unlock_irqrestore(&az_blob_dev.file_lock, flags);
+
+	return 0;
+}
+
+static int az_blob_debugfs_open(struct inode *inode, struct file
+*file) {
+	return single_open(file, az_blob_show_pending_requests, NULL); }
+
+static const struct file_operations az_blob_debugfs_fops = {
+	.open		= az_blob_debugfs_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release
+};
+
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_warn("failed to create debugfs root\n"); #endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin lock.
6) Before az_blob_fop_open() releases the spin lock, az block_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

Quoted 4 lines

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in az_blob_drv_init().

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in az_blob_remove(). Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

Quoted 141 lines

include/linux/hyperv.h int vmbus_allocate_mmio(struct resource

Quoted 119 lines
drivers/hv/azure_blob.c new file +87 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size, NULL, 0,
+			az_blob_on_channel_callback, device->channel);
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel);
+}
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device, az_blob_ringbuffer_size);
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Microsoft Azure Blob driver");
+module_init(az_blob_drv_init); module_exit(az_blob_drv_exit);
drivers/hv/channel_mgmt.c +7 -0
index 0c75662..436fdeb 100644
@@ -154,6 +154,13 @@
 	  .allowed_in_isolated = false,
 	},
 
+	/* Azure Blob */
+	{ .dev_type = HV_AZURE_BLOB,
+	  HV_AZURE_BLOB_GUID,
+	  .perf_device = false,
+	  .allowed_in_isolated = false,
+	},
+
 	/* Unknown GUID */
 	{ .dev_type = HV_UNKNOWN,
 	  .perf_device = false,
include/linux/hyperv.h index renamed from include/linux/hyperv.h +0 -0

d1e59db..ac31362 100644

include/linux/hyperv.h +1 -0
@@ -772,6 +772,7 @@ enum vmbus_device_type {
 	HV_FCOPY,
 	HV_BACKUP,
 	HV_DM,
+	HV_AZURE_BLOB,
 	HV_UNKNOWN,
 };
 
@@ -1350,6 +1351,14 @@ int vmbus_allocate_mmio(struct resource

**new, struct hv_device *device_obj,

Quoted 21 lines
include/linux/hyperv.h +8 -0
 			  0x72, 0xe2, 0xff, 0xb1, 0xdc, 0x7f)
 
 /*
+ * Azure Blob GUID
+ * {0590b792-db64-45cc-81db-b8d70c577c9e}
+ */
+#define HV_AZURE_BLOB_GUID \
+	.guid = GUID_INIT(0x0590b792, 0xdb64, 0x45cc, 0x81, 0xdb, \
+			  0xb8, 0xd7, 0x0c, 0x57, 0x7c, 0x9e)
+
+/*
  * Shutdown GUID
  * {0e0b6031-5213-4934-818b-38d90ced39db}
  */
(quoted diff) +0 -0

b/include/uapi/misc/azure_blob.h new file mode 100644 index 0000000..29413fc

include/uapi/misc/azure_blob.h new file +1 -0
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
Quoted 8 lines

include/uapi/misc/azure_blob.h line 5

Quoted 4 lines
include/uapi/misc/azure_blob.h new file +4 -0
+/* Copyright (c) 2021, Microsoft Corporation. */
+
+#ifndef _AZ_BLOB_H
+#define _AZ_BLOB_H

Seems like a #include of asm/ioctl.h (or something similar) is needed so that _IOWR is defined. Also, a #include is needed for __u32, __aligned_u64, guid_t, etc.

Yes, somehow it can still compile because they are defined in azure_blob.c. Will fix this.

Quoted 28 lines
Quoted 28 lines
include/uapi/misc/azure_blob.h new file +25 -1
+
+/* user-mode sync request sent through ioctl */ struct
+az_blob_request_sync_response {
+	__u32 status;
+	__u32 response_len;
+};
+
+struct az_blob_request_sync {
+	guid_t guid;
+	__u32 timeout;
+	__u32 request_len;
+	__u32 response_len;
+	__u32 data_len;
+	__u32 data_valid;
+	__aligned_u64 request_buffer;
+	__aligned_u64 response_buffer;
+	__aligned_u64 data_buffer;
+	struct az_blob_request_sync_response response; };
+
+#define AZ_BLOB_MAGIC_NUMBER	'R'
+#define IOCTL_AZ_BLOB_DRIVER_USER_REQUEST \
+		_IOWR(AZ_BLOB_MAGIC_NUMBER, 0xf0, \
+			struct az_blob_request_sync)
+
+#endif /* define _AZ_BLOB_H */
--

1.8.3.1

#9 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: Long Li <[email protected]> Sent: Wednesday, June 30, 2021 11:51 PM

[snip]

Quoted 62 lines
Quoted 57 lines
Quoted 30 lines
drivers/hv/azure_blob.c new file +30 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_warn("failed to create debugfs root\n"); #endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin lock.
6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

Quoted 4 lines

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in az_blob_drv_init().

Quoted 5 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in az_blob_remove(). Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Michael

Quoted 85 lines
Quoted 85 lines
Quoted 85 lines
drivers/hv/azure_blob.c new file +83 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size, NULL, 0,
+			az_blob_on_channel_callback, device->channel);
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel);
+}
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device, az_blob_ringbuffer_size);
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
#10 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 78 lines

Subject: RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: Long Li <[email protected]> Sent: Wednesday, June 30, 2021 11:51 PM

[snip]

Quoted 44 lines
Quoted 44 lines
Quoted 30 lines
drivers/hv/azure_blob.c new file +30 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_warn("failed to create debugfs root\n"); #endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

Quoted 7 lines
Quoted 7 lines

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin

lock.

Quoted 10 lines
Quoted 10 lines

6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in

az_blob_drv_init().

Quoted 3 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in

az_blob_remove().

Quoted 3 lines

Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

Quoted 8 lines

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an
open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Michael

I think it's a valid scenario. The return value of devtmpfs_delete_node() is not checked in misc_deregister(). It decreases the refcount on inodes but it's not guaranteed that someone else is still using it (in the middle of opening a file).

However, this works fine for "rmmod" that causes device to be removed. Before file is opened the refcount on the module is increased so it can't be removed when file is being opened. The scenario you described can't happen.

But during VMBUS rescind, it can happen. It's possible that the driver is using the spinlock that has been re-initialized, when the next VMBUS offer on the same channel comes before all the attempting open file calls exit.

This is a very rare. I agree things happen that we should make sure the driver can handle this. I'll update the driver.

Long

Quoted 86 lines
Quoted 86 lines
Quoted 86 lines
Quoted 86 lines
drivers/hv/azure_blob.c new file +83 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size, NULL, 0,
+			az_blob_on_channel_callback, device->channel);
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel);
+}
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device, az_blob_ringbuffer_size);
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
#11 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: Long Li <[email protected]> Sent: Friday, July 2, 2021 4:59 PM

Quoted 87 lines
Quoted 74 lines

[snip]

Quoted 44 lines
Quoted 44 lines
Quoted 30 lines
drivers/hv/azure_blob.c new file +30 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_warn("failed to create debugfs root\n"); #endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

Quoted 7 lines
Quoted 7 lines

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin

lock.

Quoted 10 lines
Quoted 10 lines

6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in

az_blob_drv_init().

Quoted 3 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in

az_blob_remove().

Quoted 3 lines

Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

Quoted 8 lines

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an
open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Michael

I think it's a valid scenario. The return value of devtmpfs_delete_node() is not checked in misc_deregister(). It decreases the refcount on inodes but it's not guaranteed that someone else is still using it (in the middle of opening a file).

Quoted 7 lines

However, this works fine for "rmmod" that causes device to be removed. Before file is opened the refcount on the module is increased so it can't be removed when file is being opened. The scenario you described can't happen.

But during VMBUS rescind, it can happen. It's possible that the driver is using the spinlock that has been re-initialized, when the next VMBUS offer on the
same channel comes before all the attempting open file calls exit.

In my scenario, Step #1 is an unbind operation, not a module removal. But you make a valid point about VMbus rescind, which has the same effect as unbind.

Quoted 3 lines

This is a very rare. I agree things happen that we should make sure the driver can handle this. I'll update the driver.

Sounds good.

Michael

#12 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to Long Li

From: Long Li <[email protected]> Sent: Friday, July 2, 2021 4:59 PM

Quoted 84 lines
Quoted 75 lines

PM

[snip]

Quoted 44 lines
Quoted 44 lines
Quoted 30 lines
drivers/hv/azure_blob.c new file +30 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs file\n");
+			debugfs_remove_recursive(az_blob_debugfs_root);
+			az_blob_debugfs_root = NULL;
+		}
+	} else

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_warn("failed to create debugfs root\n"); #endif
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

Quoted 7 lines
Quoted 7 lines

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin

lock.

Quoted 10 lines
Quoted 10 lines

6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in

az_blob_drv_init().

Quoted 3 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in

az_blob_remove().

Quoted 3 lines

Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

Quoted 8 lines

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an
open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Michael

Quoted 8 lines

I think it's a valid scenario. The return value of devtmpfs_delete_node() is not checked in misc_deregister(). It decreases the refcount on inodes but it's not guaranteed that someone else is still using it (in the middle of opening a file).

However, this works fine for "rmmod" that causes device to be removed. Before file is opened the refcount on the module is increased so it can't be removed when file is being opened. The scenario you described can't happen.

But during VMBUS rescind, it can happen. It's possible that the driver is using the spinlock that has been re-initialized, when the next VMBUS offer on the same channel comes before all the attempting open file calls exit.

I was thinking about the rescind scenario. vmbus_onoffer_rescind() will run on the global workqueue. If it eventually calls az_blob_remove() and then az_blob_remove_device(), it will wait until the file_list is empty, which essentially means waiting until user space processes decide to close the instances they have open. This seems like a problem that could block the global workqueue for a long time and
thereby hang the kernel. Is my reasoning valid? If so, I haven't
thought about what the solution might be. It seems like we do need to wait until any in-progress requests to Hyper-V are complete because Hyper-V has references to guest physical memory. But waiting for all open instances to be closed seems to be problematic.

Michael

Quoted 91 lines

This is a very rare. I agree things happen that we should make sure the driver can handle this. I'll update the driver.

Long

Quoted 86 lines
Quoted 86 lines
Quoted 86 lines
Quoted 86 lines
drivers/hv/azure_blob.c new file +83 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size, NULL, 0,
+			az_blob_on_channel_callback, device->channel);
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel);
+}
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device, az_blob_ringbuffer_size);
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
#13 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 110 lines

Subject: RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: Long Li <[email protected]> Sent: Friday, July 2, 2021 4:59 PM

Quoted 17 lines
Quoted 17 lines

PM

[snip]

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
drivers/hv/azure_blob.c new file +13 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode
+*/ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/hv/azure_blob.c

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob",

NULL);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/hv/azure_blob.c

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs

file\n");

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+

	debugfs_remove_recursive(az_blob_debugfs_root);
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/hv/azure_blob.c line 552

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+			az_blob_debugfs_root = NULL;
+		}
+	} else
+		az_blob_warn("failed to create debugfs root\n");

#endif

Quoted 50 lines
Quoted 50 lines
Quoted 14 lines
Quoted 14 lines

drivers/hv/azure_blob.c line 562

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device,
+u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

Quoted 8 lines
Quoted 8 lines

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin

lock.

Quoted 11 lines
Quoted 11 lines

6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in

az_blob_drv_init().

Quoted 3 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in

az_blob_remove().

Quoted 3 lines

Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an
open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Quoted 4 lines
Quoted 1 line

Michael

I think it's a valid scenario. The return value of devtmpfs_delete_node() is not checked in misc_deregister(). It decreases

the refcount on inodes but it's not guaranteed that someone else is still using it (in the middle of opening a file).

Quoted 3 lines

However, this works fine for "rmmod" that causes device to be removed. Before file is opened the refcount on the module is increased so it can't be

removed when file is being opened. The scenario you described can't happen.

Quoted 3 lines

But during VMBUS rescind, it can happen. It's possible that the driver is using the spinlock that has been re-initialized, when the next VMBUS

offer on the same channel comes before all the attempting open file calls exit.

Quoted 11 lines

I was thinking about the rescind scenario. vmbus_onoffer_rescind() will run on the global workqueue. If it eventually calls az_blob_remove() and then az_blob_remove_device(), it will wait until the file_list is empty, which essentially means waiting until user space processes decide to close the instances they have open. This seems like a problem that could block the global workqueue for a long time and
thereby hang the kernel. Is my reasoning valid? If so, I haven't
thought about what the solution might be. It seems like we do need to wait until any in-progress requests to Hyper-V are complete because Hyper-V has references to guest physical memory. But waiting for all open instances to be closed seems to be problematic.

My tests showed that misc_deregister() caused all opened files to be released if there are no pending I/O waiting in the driver.

If there are pending I/O, we must wait as the VSP owns the memory of the I/O. The correct VSP behavior is to return all the pending I/O along with rescind. This is the same to what storvsc does for rescind.

It looks to me waiting for opened files after the call to misc_deregister(), but before removing the vmbus channel is a safe approach.

If the VSP is behaving correctly, the rescind process should not block for too long. If we want to deal with a buggy VSP that takes forever to release a resource, we want to create a work queue for rescind handling.

Quoted 97 lines

Michael

Quoted 2 lines

This is a very rare. I agree things happen that we should make sure the

driver can handle this. I'll update the driver.

Quoted 6 lines

Long

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

drivers/hv/azure_blob.c

Quoted 9 lines
drivers/hv/azure_blob.c new file +9 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size,

NULL, 0,

Quoted 17 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+			az_blob_on_channel_callback, device-

channel);

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
drivers/hv/azure_blob.c new file +15 -0
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel); }
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/hv/azure_blob.c

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device,

az_blob_ringbuffer_size);

Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
drivers/hv/azure_blob.c new file +49 -0
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
#14 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

From: Long Li <[email protected]> Sent: Friday, July 9, 2021 12:44 PM

Quoted 121 lines
Quoted 109 lines

From: Long Li <[email protected]> Sent: Friday, July 2, 2021 4:59 PM

Quoted 17 lines
Quoted 17 lines

PM

[snip]

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +13 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode
+*/ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {
+	int rc;
+	struct dentry *d;
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	az_blob_debugfs_root = debugfs_create_dir("az_blob",

NULL);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {
+			az_blob_warn("failed to create debugfs

file\n");

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+

	debugfs_remove_recursive(az_blob_debugfs_root);
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+			az_blob_debugfs_root = NULL;
+		}
+	} else
+		az_blob_warn("failed to create debugfs root\n");

#endif

Quoted 50 lines
Quoted 50 lines
Quoted 14 lines
Quoted 14 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	return 0;
+}
+
+static int az_blob_connect_to_vsp(struct hv_device *device,
+u32
+ring_size) {
+	int ret;
+
+	spin_lock_init(&az_blob_dev.file_lock);

I'd argue that the spin lock should not be re-initialized here. Here's the sequence where things go wrong:

Quoted 8 lines
Quoted 8 lines

1) The driver is unbound, so az_blob_remove() is called.
2) az_blob_remove() sets the "removing" flag to true, and calls az_blob_remove_device().
3) az_blob_remove_device() waits for the file_list to become empty.
4) After the file_list becomes empty, but before misc_deregister() is called, a separate thread opens the device again.
5) In the separate thread, az_blob_fop_open() obtains the file_lock spin

lock.

Quoted 11 lines
Quoted 11 lines

6) Before az_blob_fop_open() releases the spin lock, az_blob_remove_device() completes, along with az_blob_remove().
7) Then the device gets rebound, and az_blob_connect_to_vsp() gets called, all while az_blob_fop_open() still holds the spin lock. So the spin lock get re- initialized while it is held.

This is admittedly a far-fetched scenario, but stranger things have happened. :-) The issue is that you are counting on the az_blob_dev structure to persist and have a valid file_lock, even while the device is unbound. So any initialization should only happen in

az_blob_drv_init().

Quoted 3 lines

I'm not sure if az_blob_probe() and az_blob_remove() can be called at the same time, as az_blob_remove_vmbus() is called the last in

az_blob_remove().

Quoted 3 lines

Is it possible for vmbus asking the driver to probe a new channel before the old channel is closed? I expect the vmbus provide guarantee that those calls are made in sequence.

In my scenario above, az_blob_remove_vmbus() and az_blob_remove() run to completion in Step #6, all while some other thread is still in the middle of an
open() call and holding the file_lock spin lock. Then in Step #7 az_blob_probe() runs. So az_blob_remove() and az_blob_probe() execute sequentially, not at the same time.

Quoted 4 lines
Quoted 1 line

Michael

I think it's a valid scenario. The return value of devtmpfs_delete_node() is not checked in misc_deregister(). It decreases

the refcount on inodes but it's not guaranteed that someone else is still using it (in the middle of opening a file).

Quoted 3 lines

However, this works fine for "rmmod" that causes device to be removed. Before file is opened the refcount on the module is increased so it can't be

removed when file is being opened. The scenario you described can't happen.

Quoted 3 lines

But during VMBUS rescind, it can happen. It's possible that the driver is using the spinlock that has been re-initialized, when the next VMBUS

offer on the same channel comes before all the attempting open file calls exit.

Quoted 11 lines

I was thinking about the rescind scenario. vmbus_onoffer_rescind() will run on the global workqueue. If it eventually calls az_blob_remove() and then az_blob_remove_device(), it will wait until the file_list is empty, which essentially means waiting until user space processes decide to close the instances they have open. This seems like a problem that could block the global workqueue for a long time and
thereby hang the kernel. Is my reasoning valid? If so, I haven't
thought about what the solution might be. It seems like we do need to wait until any in-progress requests to Hyper-V are complete because Hyper-V has references to guest physical memory. But waiting for all open instances to be closed seems to be problematic.

Quoted 5 lines

My tests showed that misc_deregister() caused all opened files to be released if there are no pending I/O waiting in the driver.

If there are pending I/O, we must wait as the VSP owns the memory of the I/O. The correct VSP behavior is to return all the pending I/O along with rescind. This is the same to what storvsc does for rescind.

Yes, we have to wait since the VSP owns the memory. But you make a good point that the VSP behavior should be to return all the pending I/Os at roughly the same time as the rescind.

Quoted 3 lines

It looks to me waiting for opened files after the call to misc_deregister(), but before removing the vmbus channel is a safe approach.

To me, this would be a great solution. And it closes the original timing window I pointed out where some other thread could open the device after having waited for the file_list to be empty, but before misc_deregister() is called. With the order swapped, once the file_list is empty, there's no way for the device to be opened again. So it solves the problem with the spin_lock initialization.

Quoted 3 lines

If the VSP is behaving correctly, the rescind process should not block for too long. If we want to deal with a buggy VSP that takes forever to release a resource, we want to create a work queue for rescind handling.

I don't think we need to deal with a buggy VSP holding memory for an excessively long time.

Michael

Quoted 94 lines
Quoted 94 lines
Quoted 2 lines

This is a very rare. I agree things happen that we should make sure the

driver can handle this. I'll update the driver.

Quoted 6 lines

Long

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+	INIT_LIST_HEAD(&az_blob_dev.file_list);
+	init_waitqueue_head(&az_blob_dev.file_wait);
+
+	az_blob_dev.removing = false;
+
+	az_blob_dev.device = device;
+	device->channel->rqstor_size = device_queue_depth;
+
+	ret = vmbus_open(device->channel, ring_size, ring_size,

NULL, 0,

Quoted 17 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +1 -0
+			az_blob_on_channel_callback, device-

channel);

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+
+	if (ret) {
+		az_blob_err("failed to connect to VSP ret %d\n", ret);
+		return ret;
+	}
+
+	hv_set_drvdata(device, &az_blob_dev);
+
+	return ret;
+}
+
+static void az_blob_remove_vmbus(struct hv_device *device) {
+	/* At this point, no VSC/VSP traffic is possible over vmbus */
+	hv_set_drvdata(device, NULL);
+	vmbus_close(device->channel); }
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+static int az_blob_probe(struct hv_device *device,
+			const struct hv_vmbus_device_id *dev_id) {
+	int rc;
+
+	az_blob_dbg("probing device\n");
+
+	rc = az_blob_connect_to_vsp(device,

az_blob_ringbuffer_size);

Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
(quoted diff) +49 -0
+	if (rc) {
+		az_blob_err("error connecting to VSP rc %d\n", rc);
+		return rc;
+	}
+
+	// create user-mode client library facing device
+	rc = az_blob_create_device(&az_blob_dev);
+	if (rc) {
+		az_blob_remove_vmbus(device);
+		return rc;
+	}
+
+	az_blob_dbg("successfully probed device\n");
+	return 0;
+}
+
+static int az_blob_remove(struct hv_device *dev) {
+	struct az_blob_device *device = hv_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&device->file_lock, flags);
+	device->removing = true;
+	spin_unlock_irqrestore(&device->file_lock, flags);
+
+	az_blob_remove_device(device);
+	az_blob_remove_vmbus(dev);
+	return 0;
+}
+
+static struct hv_driver az_blob_drv = {
+	.name = KBUILD_MODNAME,
+	.id_table = id_table,
+	.probe = az_blob_probe,
+	.remove = az_blob_remove,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+};
+
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;
+}
+
+static void __exit az_blob_drv_exit(void) {
+	vmbus_driver_unregister(&az_blob_drv);
+}
#15 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to [email protected]

drivers/hv/azure_blob.c line 77

Quoted 3 lines · On Fri, Jun 25, 2021 at 11:30:19PM -0700, [email protected] wrote:
drivers/hv/azure_blob.c new file +3 -0
+#ifdef CONFIG_DEBUG_FS
+struct dentry *az_blob_debugfs_root;
+#endif

No need to keep this dentry, just look it up if you need it.

drivers/hv/azure_blob.c line 83

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size (bytes)");

This is NOT the 1990's, please do not create new module parameters. Just make this work properly for everyone.

drivers/hv/azure_blob.c line 98

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+#define AZ_ERR 0
+#define AZ_WARN 1
+#define AZ_DBG 2
+static int log_level = AZ_ERR;
+module_param(log_level, int, 0644);
+MODULE_PARM_DESC(log_level,
+	"Log level: 0 - Error (default), 1 - Warning, 2 - Debug.");

A single driver does not need a special debug/log level, use the system-wide functions and all will "just work"

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+
+static uint device_queue_depth = 1024;
+module_param(device_queue_depth, uint, 0444);
+MODULE_PARM_DESC(device_queue_depth,
+	"System level max queue depth for this device");

drivers/hv/azure_blob.c line 113

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+
+#define az_blob_log(level, fmt, args...)	\
+do {	\
+	if (level <= log_level)	\
+		pr_err("%s:%d " fmt, __func__, __LINE__, ##args);	\
+} while (0)
+
+#define az_blob_dbg(fmt, args...) az_blob_log(AZ_DBG, fmt, ##args)
+#define az_blob_warn(fmt, args...) az_blob_log(AZ_WARN, fmt, ##args)
+#define az_blob_err(fmt, args...) az_blob_log(AZ_ERR, fmt, ##args)

Again, no.

Just use dev_dbg(), dev_warn(), and dev_err() and there is no need for anything "special". This is just one tiny driver, do not rewrite logic like this for no reason.

drivers/hv/azure_blob.c line 523

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+static void az_blob_remove_device(struct az_blob_device *dev)
+{
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS

No need for the #ifdef.

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */
+}
+
+static int az_blob_create_device(struct az_blob_device *dev)
+{

drivers/hv/azure_blob.c line 540

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS

No need for the #ifdef

drivers/hv/azure_blob.c line 542

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {

No need to check this.

drivers/hv/azure_blob.c line 546

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {

How can that be NULL?

No need to ever check any debugfs calls, please just make them and move on.

thanks,

greg k-h

#16 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

On 29.06.21 08:24, Greg Kroah-Hartman wrote:

Hi folks,

Quoted 5 lines

Again, no.

Just use dev_dbg(), dev_warn(), and dev_err() and there is no need for anything "special". This is just one tiny driver, do not rewrite logic like this for no reason.

Maybe worth mentioning that we have the pr_fmt macro that can be defined if some wants to do things like adding some prefix.

Quoted 2 lines

No need to ever check any debugfs calls, please just make them and move on.

In this case a failed attempt to create debugfs files/dirs should not be treated as some actual error, but a normal situation, therefore there shouldn't be an error message. (see debugfs.h)

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#17 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 4 lines · metux IT consult

On 29.06.21 08:24, Greg Kroah-Hartman wrote:

Hi folks,

Quoted 8 lines
Quoted 5 lines

Again, no.

Just use dev_dbg(), dev_warn(), and dev_err() and there is no need for anything "special". This is just one tiny driver, do not rewrite logic like this for no reason.

Maybe worth mentioning that we have the pr_fmt macro that can be defined if some wants to do things like adding some prefix.

No, no driver should mess with that, just use dev_*() calls instead please.

greg k-h

#18 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 11 lines · Greg Kroah-Hartman
Quoted 10 lines · metux IT consult

On 29.06.21 08:24, Greg Kroah-Hartman wrote:

Hi folks,

Quoted 5 lines

Again, no.

Just use dev_dbg(), dev_warn(), and dev_err() and there is no need for anything "special". This is just one tiny driver, do not rewrite logic like this for no reason.

Quoted 5 lines
Quoted 2 lines

Maybe worth mentioning that we have the pr_fmt macro that can be defined if some wants to do things like adding some prefix.

No, no driver should mess with that, just use dev_*() calls instead please.

Well, I've been referring to some scenario where you *really* want some *additional* prefix over the whole file. But you're right, the good use cases for that are very rare - dev_*() usually should already put enough information into the message.

--mtx

Signature · 8 lines
---
Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schlüssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[email protected] -- +49-151-27565287
#19 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to Greg Kroah-Hartman
Quoted 8 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

drivers/hv/azure_blob.c line 77

Quoted 3 lines · On Fri, Jun 25, 2021 at 11:30:19PM -0700, [email protected] wrote:
drivers/hv/azure_blob.c new file +3 -0
+#ifdef CONFIG_DEBUG_FS
+struct dentry *az_blob_debugfs_root;
+#endif

No need to keep this dentry, just look it up if you need it.

Will fix this.

Quoted 11 lines

drivers/hv/azure_blob.c

Quoted 8 lines
drivers/hv/azure_blob.c new file +7 -0
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size
+(bytes)");

This is NOT the 1990's, please do not create new module parameters. Just make this work properly for everyone.

The default value is chosen so that it works best for most workloads while not taking too much system resources. It should work out of box for nearly all customers.

But what we see is that from time to time, some customers still want to change those values to work best for their workloads. It's hard to predict their workload. They have to recompile the kernel if there is no module parameter to do it. So the intent of this module parameter is that if the default value works for you, don't mess up with it.

Quoted 11 lines

drivers/hv/azure_blob.c line 98

Quoted 8 lines
drivers/hv/azure_blob.c new file +7 -0
+#define AZ_ERR 0
+#define AZ_WARN 1
+#define AZ_DBG 2
+static int log_level = AZ_ERR;
+module_param(log_level, int, 0644);
+MODULE_PARM_DESC(log_level,
+	"Log level: 0 - Error (default), 1 - Warning, 2 - Debug.");

A single driver does not need a special debug/log level, use the system-wide functions and all will "just work"

Will fix this.

Quoted 53 lines
Quoted 6 lines
drivers/hv/azure_blob.c new file +5 -0
+
+static uint device_queue_depth = 1024;
+module_param(device_queue_depth, uint, 0444);
+MODULE_PARM_DESC(device_queue_depth,
+	"System level max queue depth for this device");

drivers/hv/azure_blob.c line 113

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+
+#define az_blob_log(level, fmt, args...)	\
+do {	\
+	if (level <= log_level)	\
+		pr_err("%s:%d " fmt, __func__, __LINE__, ##args);	\
+} while (0)
+
+#define az_blob_dbg(fmt, args...) az_blob_log(AZ_DBG, fmt, ##args)
+#define az_blob_warn(fmt, args...) az_blob_log(AZ_WARN, fmt, ##args)
+#define az_blob_err(fmt, args...) az_blob_log(AZ_ERR, fmt, ##args)

Again, no.

Just use dev_dbg(), dev_warn(), and dev_err() and there is no need for anything "special". This is just one tiny driver, do not rewrite logic like this for no reason.

drivers/hv/azure_blob.c line 523

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+static void az_blob_remove_device(struct az_blob_device *dev) {
+	wait_event(dev->file_wait, list_empty(&dev->file_list));
+	misc_deregister(&az_blob_misc_device);
+#ifdef CONFIG_DEBUG_FS

No need for the #ifdef.

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+	debugfs_remove_recursive(az_blob_debugfs_root);
+#endif
+	/* At this point, we won't get any requests from user-mode */ }
+
+static int az_blob_create_device(struct az_blob_device *dev) {

drivers/hv/azure_blob.c line 540

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	int rc;
+	struct dentry *d;
+
+	rc = misc_register(&az_blob_misc_device);
+	if (rc) {
+		az_blob_err("misc_register failed rc %d\n", rc);
+		return rc;
+	}
+
+#ifdef CONFIG_DEBUG_FS

No need for the #ifdef

drivers/hv/azure_blob.c line 542

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+	az_blob_debugfs_root = debugfs_create_dir("az_blob", NULL);
+	if (!IS_ERR_OR_NULL(az_blob_debugfs_root)) {

No need to check this.

Quoted 8 lines

drivers/hv/azure_blob.c line 546

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		d = debugfs_create_file("pending_requests", 0400,
+			az_blob_debugfs_root, NULL,
+			&az_blob_debugfs_fops);
+		if (IS_ERR_OR_NULL(d)) {

How can that be NULL?

No need to ever check any debugfs calls, please just make them and move on.

Will fix this.

Thank you,

Long

Quoted 4 lines

thanks,

greg k-h

#20 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 12 lines · Long Li
Quoted 11 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size
+(bytes)");

This is NOT the 1990's, please do not create new module parameters. Just make this work properly for everyone.

Quoted 1 line

The default value is chosen so that it works best for most workloads while not taking too much system resources. It should work out of box for nearly all customers.

Please wrap your email lines :(

Anyway, great, it will work for everyone, no need for this to be changed. Then just drop it.

Quoted 1 line

But what we see is that from time to time, some customers still want to change those values to work best for their workloads. It's hard to predict their workload. They have to recompile the kernel if there is no module parameter to do it. So the intent of this module parameter is that if the default value works for you, don't mess up with it.

Can you not just determine at runtime that these workloads are overflowing the buffer and increase the size of it? That would be best otherwise you are forced to try to deal with a module parameter that is for code, not for devices (which is why we do not like module parameters for drivers.)

Please remove this for now and if you _REALLY_ need it in the future, make it auto-tunable. Or if that is somehow impossible, then make it a per-device option, through something like sysfs so that it will work correctly per-device.

thanks,

greg k-h

#21 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 35 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 12 lines · Long Li
Quoted 11 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+
+static struct az_blob_device az_blob_dev;
+
+static int az_blob_ringbuffer_size = (128 * 1024);
+module_param(az_blob_ringbuffer_size, int, 0444);
+MODULE_PARM_DESC(az_blob_ringbuffer_size, "Ring buffer size
+(bytes)");

This is NOT the 1990's, please do not create new module parameters. Just make this work properly for everyone.

Quoted 1 line

The default value is chosen so that it works best for most workloads while

not taking too much system resources. It should work out of box for nearly all customers.

Please wrap your email lines :(

Anyway, great, it will work for everyone, no need for this to be changed. Then just drop it.

Quoted 1 line

But what we see is that from time to time, some customers still want to

change those values to work best for their workloads. It's hard to predict their workload. They have to recompile the kernel if there is no module parameter to do it. So the intent of this module parameter is that if the default value works for you, don't mess up with it.

Can you not just determine at runtime that these workloads are overflowing the buffer and increase the size of it? That would be best otherwise you are forced to try to deal with a module parameter that is for code, not for devices (which is why we do not like module parameters for drivers.)

Quoted 7 lines

Please remove this for now and if you _REALLY_ need it in the future, make it auto-tunable. Or if that is somehow impossible, then make it a per-device option, through something like sysfs so that it will work correctly per-device.

thanks,

greg k-h

I got your point, will be removing this.

Thanks,
Long

#22 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to [email protected]

drivers/hv/azure_blob.c line 1

Quoted 2 lines · On Fri, Jun 25, 2021 at 11:30:19PM -0700, [email protected] wrote:
drivers/hv/azure_blob.c new file +1 -0
@@ -0,0 +1,655 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause

If you _REALLY_ want to do this, please document _WHY_ you are doing this in the changelog comment as dual-licensed Linux kernel code will just cause you a lot of problems in the long run and we want to be sure you understand all of the issues here and your managers agree with it.

thanks,

greg k-h

#23 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 11 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

drivers/hv/azure_blob.c line 1

Quoted 2 lines · On Fri, Jun 25, 2021 at 11:30:19PM -0700, [email protected] wrote:
drivers/hv/azure_blob.c new file +1 -0
@@ -0,0 +1,655 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause

If you _REALLY_ want to do this, please document _WHY_ you are doing this in the changelog comment as dual-licensed Linux kernel code will just cause you a lot of problems in the long run and we want to be sure you understand all of the issues here and your managers agree with it.

Quoted 3 lines

thanks,

greg k-h

Will fix this.

Thanks,
Long

#24 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

In reply to [email protected]
Quoted 7 lines · On 26. 06. 21, 8:30, [email protected] wrote:

Azure Blob storage provides scalable and durable data storage for Azure.
(https://azure.microsoft.com/en-us/services/storage/blobs/)

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server access.

...

Quoted 1 line
Cc Jiri Slaby <[email protected]>

I am just curious, why am I CCed? Anyway, some comments below:

Quoted 3 lines
drivers/hv/azure_blob.c new file +0 -0
@@ -0,0 +1,655 @@

...

drivers/hv/azure_blob.c line 117

Quoted 3 lines
drivers/hv/azure_blob.c new file +3 -0
+static void az_blob_on_channel_callback(void *context)
+{
+	struct vmbus_channel *channel = (struct vmbus_channel *)context;

Have you fed this patch through checkpatch?

Quoted 27 lines
drivers/hv/azure_blob.c new file +27 -0
+	const struct vmpacket_descriptor *desc;
+
+	az_blob_dbg("entering interrupt from vmbus\n");
+	foreach_vmbus_pkt(desc, channel) {
+		struct az_blob_vsp_request_ctx *request_ctx;
+		struct az_blob_vsp_response *response;
+		u64 cmd_rqst = vmbus_request_addr(&channel->requestor,
+					desc->trans_id);
+		if (cmd_rqst == VMBUS_RQST_ERROR) {
+			az_blob_err("incorrect transaction id %llu\n",
+				desc->trans_id);
+			continue;
+		}
+		request_ctx = (struct az_blob_vsp_request_ctx *) cmd_rqst;
+		response = hv_pkt_data(desc);
+
+		az_blob_dbg("got response for request %pUb status %u "
+			"response_len %u\n",
+			&request_ctx->request->guid, response->error,
+			response->response_len);
+		request_ctx->request->response.status = response->error;
+		request_ctx->request->response.response_len =
+			response->response_len;
+		complete(&request_ctx->wait_vsp);
+	}
+
+}

...

Quoted 9 lines
drivers/hv/azure_blob.c new file +9 -0
+static long az_blob_ioctl_user_request(struct file *filp, unsigned long arg)
+{
+	struct az_blob_device *dev = &az_blob_dev;
+	struct az_blob_file_ctx *file_ctx = filp->private_data;
+	char __user *argp = (char __user *) arg;
+	struct az_blob_request_sync request;
+	struct az_blob_vsp_request_ctx request_ctx;
+	unsigned long flags;
+	int ret;

drivers/hv/azure_blob.c line 266

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	size_t request_start, request_num_pages = 0;
+	size_t response_start, response_num_pages = 0;
+	size_t data_start, data_num_pages = 0, total_num_pages;
+	struct page **request_pages = NULL, **response_pages = NULL;
+	struct page **data_pages = NULL;
+	struct vmbus_packet_mpb_array *desc;
+	u64 *pfn_array;
+	int desc_size;
+	int page_idx;
+	struct az_blob_vsp_request *vsp_request;

Ugh, see further.

Quoted 30 lines
drivers/hv/azure_blob.c new file +30 -0
+
+	/* Fast fail if device is being removed */
+	if (dev->removing)
+		return -ENODEV;
+
+	if (!az_blob_safe_file_access(filp)) {
+		az_blob_dbg("process %d(%s) changed security contexts after"
+			" opening file descriptor\n",
+			task_tgid_vnr(current), current->comm);
+		return -EACCES;
+	}
+
+	if (copy_from_user(&request, argp, sizeof(request))) {
+		az_blob_dbg("don't have permission to user provided buffer\n");
+		return -EFAULT;
+	}
+
+	az_blob_dbg("az_blob ioctl request guid %pUb timeout %u request_len %u"
+		" response_len %u data_len %u request_buffer %llx "
+		"response_buffer %llx data_buffer %llx\n",
+		&request.guid, request.timeout, request.request_len,
+		request.response_len, request.data_len, request.request_buffer,
+		request.response_buffer, request.data_buffer);
+
+	if (!request.request_len || !request.response_len)
+		return -EINVAL;
+
+	if (request.data_len && request.data_len < request.data_valid)
+		return -EINVAL;
+

drivers/hv/azure_blob.c line 306

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	init_completion(&request_ctx.wait_vsp);
+	request_ctx.request = &request;
+
+	/*
+	 * Need to set rw to READ to have page table set up for passing to VSP.
+	 * Setting it to WRITE will cause the page table entry not allocated
+	 * as it's waiting on Copy-On-Write on next page fault. This doesn't
+	 * work in this scenario because VSP wants all the pages to be present.
+	 */
+	ret = get_buffer_pages(READ, (void __user *) request.request_buffer,

Does this need __force for sparse not to complain?

Quoted 22 lines
drivers/hv/azure_blob.c new file +22 -0
+		request.request_len, &request_pages, &request_start,
+		&request_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	ret = get_buffer_pages(READ, (void __user *) request.response_buffer,
+		request.response_len, &response_pages, &response_start,
+		&response_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	if (request.data_len) {
+		ret = get_buffer_pages(READ,
+			(void __user *) request.data_buffer, request.data_len,
+			&data_pages, &data_start, &data_num_pages);
+		if (ret)
+			goto get_user_page_failed;
+	}
+
+	total_num_pages = request_num_pages + response_num_pages +
+				data_num_pages;
+	if (total_num_pages > AZ_BLOB_MAX_PAGES) {

drivers/hv/azure_blob.c line 338

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		az_blob_dbg("number of DMA pages %lu buffer exceeding %u\n",
+			total_num_pages, AZ_BLOB_MAX_PAGES);
+		ret = -EINVAL;
+		goto get_user_page_failed;
+	}
+
+	/* Construct a VMBUS packet and send it over to VSP */
+	desc_size = sizeof(struct vmbus_packet_mpb_array) +
+			sizeof(u64) * total_num_pages;
+	desc = kzalloc(desc_size, GFP_KERNEL);

Smells like a call for struct_size().

Quoted 56 lines
drivers/hv/azure_blob.c new file +56 -0
+	vsp_request = kzalloc(sizeof(*vsp_request), GFP_KERNEL);
+	if (!desc || !vsp_request) {
+		kfree(desc);
+		kfree(vsp_request);
+		ret = -ENOMEM;
+		goto get_user_page_failed;
+	}
+
+	desc->range.offset = 0;
+	desc->range.len = total_num_pages * PAGE_SIZE;
+	pfn_array = desc->range.pfn_array;
+	page_idx = 0;
+
+	if (request.data_len) {
+		fill_in_page_buffer(pfn_array, &page_idx, data_pages,
+			data_num_pages);
+		vsp_request->data_buffer_offset = data_start;
+		vsp_request->data_buffer_length = request.data_len;
+		vsp_request->data_buffer_valid = request.data_valid;
+	}
+
+	fill_in_page_buffer(pfn_array, &page_idx, request_pages,
+		request_num_pages);
+	vsp_request->request_buffer_offset = request_start +
+						data_num_pages * PAGE_SIZE;
+	vsp_request->request_buffer_length = request.request_len;
+
+	fill_in_page_buffer(pfn_array, &page_idx, response_pages,
+		response_num_pages);
+	vsp_request->response_buffer_offset = response_start +
+		(data_num_pages + request_num_pages) * PAGE_SIZE;
+	vsp_request->response_buffer_length = request.response_len;
+
+	vsp_request->version = 0;
+	vsp_request->timeout_ms = request.timeout;
+	vsp_request->operation_type = AZ_BLOB_DRIVER_USER_REQUEST;
+	guid_copy(&vsp_request->transaction_id, &request.guid);
+
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_add_tail(&request_ctx.list, &file_ctx->vsp_pending_requests);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+	az_blob_dbg("sending request to VSP\n");
+	az_blob_dbg("desc_size %u desc->range.len %u desc->range.offset %u\n",
+		desc_size, desc->range.len, desc->range.offset);
+	az_blob_dbg("vsp_request data_buffer_offset %u data_buffer_length %u "
+		"data_buffer_valid %u request_buffer_offset %u "
+		"request_buffer_length %u response_buffer_offset %u "
+		"response_buffer_length %u\n",
+		vsp_request->data_buffer_offset,
+		vsp_request->data_buffer_length,
+		vsp_request->data_buffer_valid,
+		vsp_request->request_buffer_offset,
+		vsp_request->request_buffer_length,
+		vsp_request->response_buffer_offset,
+		vsp_request->response_buffer_length);

drivers/hv/azure_blob.c line 404

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+
+	ret = vmbus_sendpacket_mpb_desc(dev->device->channel, desc, desc_size,
+		vsp_request, sizeof(*vsp_request), (u64) &request_ctx);
+
+	kfree(desc);
+	kfree(vsp_request);
+	if (ret)
+		goto vmbus_send_failed;
+
+	wait_for_completion(&request_ctx.wait_vsp);

Provided this is ioctl, this should likely be interruptible. You don't want to account to I/O load. The same likely for az_blob_fop_release.

drivers/hv/azure_blob.c line 414

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+
+	/*
+	 * At this point, the response is already written to request
+	 * by VMBUS completion handler, copy them to user-mode buffers
+	 * and return to user-mode
+	 */
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.status),
+			&request.response.status,

This is ugly, why don't you make argp the appropriate pointer instead of char *? You'd need not do copy_to_user twice then, right?

drivers/hv/azure_blob.c line 422

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+			sizeof(request.response.status))) {
+		ret = -EFAULT;
+		goto vmbus_send_failed;
+	}
+
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.response_len),

The same here.

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+			&request.response.response_len,
+			sizeof(request.response.response_len)))
+		ret = -EFAULT;
+
+vmbus_send_failed:
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_del(&request_ctx.list);
+	if (list_empty(&file_ctx->vsp_pending_requests))

drivers/hv/azure_blob.c line 440

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+		wake_up(&file_ctx->wait_vsp_pending);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+get_user_page_failed:
+	free_buffer_pages(request_num_pages, request_pages);
+	free_buffer_pages(response_num_pages, response_pages);
+	free_buffer_pages(data_num_pages, data_pages);
+
+	return ret;
+}

This function is overly long. Care to split it (e.g. moving away the initialization of the structs and the debug stuff)?

drivers/hv/azure_blob.c line 445

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+
+static long az_blob_fop_ioctl(struct file *filp, unsigned int cmd,
+	unsigned long arg)
+{
+	long ret = -EIO;

EINVAL would be more appropriate.

drivers/hv/azure_blob.c line 450

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+
+	switch (cmd) {
+	case IOCTL_AZ_BLOB_DRIVER_USER_REQUEST:
+		if (_IOC_SIZE(cmd) != sizeof(struct az_blob_request_sync))
+			return -EINVAL;

How can that happen, provided the switch (cmd) and case?

drivers/hv/azure_blob.c line 452

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+		ret = az_blob_ioctl_user_request(filp, arg);
+		break;

Simply:
return az_blob_ioctl_user_request(filp, arg);

drivers/hv/azure_blob.c line 458

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+
+	default:
+		az_blob_dbg("unrecognized IOCTL code %u\n", cmd);
+	}
+
+	return ret;

So return -EINVAL here directly now.

Quoted 1 line

+}

...

Quoted 3 lines
drivers/hv/azure_blob.c new file +3 -0
+static int az_blob_connect_to_vsp(struct hv_device *device, u32 ring_size)
+{
+	int ret;

...

drivers/hv/azure_blob.c

Quoted 1 line
drivers/hv/azure_blob.c new file +1 -0
+	int rc;

Sometimes ret, sometimes rc, could you unify the two?

drivers/hv/azure_blob.c line 644

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+static int __init az_blob_drv_init(void)
+{
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;

Simply:
return vmbus_driver_register(&az_blob_drv);

regards,

Signature · 3 lines
-- 
js
suse labs
#25 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 31 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 3 lines · On 26. 06. 21, 8:30, [email protected] wrote:

Azure Blob storage provides scalable and durable data storage for Azure.
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fazu
re.microsoft.com%2Fen-

us%2Fservices%2Fstorage%2Fblobs%2F&amp;data=04%7

Quoted 1 line

C01%7Clongli%40microsoft.com%7Cf5787b443bd04a8a2db708d93ad9ec59 %7C72f9

Quoted 1 line

88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637605529942807730%7C Unknown%7C

Quoted 1 line

TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ XVC

Quoted 1 line

I6Mn0%3D%7C1000&amp;sdata=bu0G8il7D%2F3emZkPn8FXIHWWff0WPORF 5CBfHAsIOI

Quoted 5 lines

4%3D&amp;reserved=0)

This driver adds support for accelerated access to Azure Blob storage. As an alternative to REST APIs, it provides a fast data path that uses host native network stack and secure direct data link for storage server

access.

Quoted 1 line

...

Quoted 1 line
Cc Jiri Slaby <[email protected]>

I am just curious, why am I CCed? Anyway, some comments below:

Quoted 8 lines
Quoted 3 lines
drivers/hv/azure_blob.c new file +0 -0
@@ -0,0 +1,655 @@

...

drivers/hv/azure_blob.c

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+static void az_blob_on_channel_callback(void *context) {
+	struct vmbus_channel *channel = (struct vmbus_channel *)context;

Have you fed this patch through checkpatch?

Yes, it didn't throw out any errors.

Quoted 89 lines
Quoted 28 lines
drivers/hv/azure_blob.c new file +27 -0
+	const struct vmpacket_descriptor *desc;
+
+	az_blob_dbg("entering interrupt from vmbus\n");
+	foreach_vmbus_pkt(desc, channel) {
+		struct az_blob_vsp_request_ctx *request_ctx;
+		struct az_blob_vsp_response *response;
+		u64 cmd_rqst = vmbus_request_addr(&channel->requestor,
+					desc->trans_id);
+		if (cmd_rqst == VMBUS_RQST_ERROR) {
+			az_blob_err("incorrect transaction id %llu\n",
+				desc->trans_id);
+			continue;
+		}
+		request_ctx = (struct az_blob_vsp_request_ctx *) cmd_rqst;
+		response = hv_pkt_data(desc);
+
+		az_blob_dbg("got response for request %pUb status %u "
+			"response_len %u\n",
+			&request_ctx->request->guid, response->error,
+			response->response_len);
+		request_ctx->request->response.status = response->error;
+		request_ctx->request->response.response_len =
+			response->response_len;
+		complete(&request_ctx->wait_vsp);
+	}
+
+}

...

Quoted 9 lines
drivers/hv/azure_blob.c new file +9 -0
+static long az_blob_ioctl_user_request(struct file *filp, unsigned
+long arg) {
+	struct az_blob_device *dev = &az_blob_dev;
+	struct az_blob_file_ctx *file_ctx = filp->private_data;
+	char __user *argp = (char __user *) arg;
+	struct az_blob_request_sync request;
+	struct az_blob_vsp_request_ctx request_ctx;
+	unsigned long flags;
+	int ret;

drivers/hv/azure_blob.c line 266

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+	size_t request_start, request_num_pages = 0;
+	size_t response_start, response_num_pages = 0;
+	size_t data_start, data_num_pages = 0, total_num_pages;
+	struct page **request_pages = NULL, **response_pages = NULL;
+	struct page **data_pages = NULL;
+	struct vmbus_packet_mpb_array *desc;
+	u64 *pfn_array;
+	int desc_size;
+	int page_idx;
+	struct az_blob_vsp_request *vsp_request;

Ugh, see further.

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+
+	/* Fast fail if device is being removed */
+	if (dev->removing)
+		return -ENODEV;
+
+	if (!az_blob_safe_file_access(filp)) {
+		az_blob_dbg("process %d(%s) changed security contexts

after"

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+			" opening file descriptor\n",
+			task_tgid_vnr(current), current->comm);
+		return -EACCES;
+	}
+
+	if (copy_from_user(&request, argp, sizeof(request))) {
+		az_blob_dbg("don't have permission to user provided

buffer\n");

drivers/hv/azure_blob.c

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		return -EFAULT;
+	}
+
+	az_blob_dbg("az_blob ioctl request guid %pUb timeout %u

request_len %u"

drivers/hv/azure_blob.c

Quoted 4 lines
drivers/hv/azure_blob.c new file +4 -0
+		" response_len %u data_len %u request_buffer %llx "
+		"response_buffer %llx data_buffer %llx\n",
+		&request.guid, request.timeout, request.request_len,
+		request.response_len, request.data_len,

request.request_buffer,

Quoted 12 lines
drivers/hv/azure_blob.c new file +12 -0
+		request.response_buffer, request.data_buffer);
+
+	if (!request.request_len || !request.response_len)
+		return -EINVAL;
+
+	if (request.data_len && request.data_len < request.data_valid)
+		return -EINVAL;
+
+	init_completion(&request_ctx.wait_vsp);
+	request_ctx.request = &request;
+
+	/*
Quoted 8 lines

drivers/hv/azure_blob.c line 306

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+	 * Need to set rw to READ to have page table set up for passing to VSP.
+	 * Setting it to WRITE will cause the page table entry not allocated
+	 * as it's waiting on Copy-On-Write on next page fault. This doesn't
+	 * work in this scenario because VSP wants all the pages to be present.
+	 */
+	ret = get_buffer_pages(READ, (void __user *) request.request_buffer,

Does this need __force for sparse not to complain?

I didn't run sparse for the patch. Will look into it. Thanks for the pointer.

Quoted 26 lines

drivers/hv/azure_blob.c

Quoted 7 lines
drivers/hv/azure_blob.c new file +6 -0
+		request.request_len, &request_pages, &request_start,
+		&request_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	ret = get_buffer_pages(READ, (void __user *)

request.response_buffer,

Quoted 7 lines
drivers/hv/azure_blob.c new file +7 -0
+		request.response_len, &response_pages, &response_start,
+		&response_num_pages);
+	if (ret)
+		goto get_user_page_failed;
+
+	if (request.data_len) {
+		ret = get_buffer_pages(READ,

drivers/hv/azure_blob.c

Quoted 10 lines
drivers/hv/azure_blob.c new file +10 -0
+			(void __user *) request.data_buffer, request.data_len,
+			&data_pages, &data_start, &data_num_pages);
+		if (ret)
+			goto get_user_page_failed;
+	}
+
+	total_num_pages = request_num_pages + response_num_pages +
+				data_num_pages;
+	if (total_num_pages > AZ_BLOB_MAX_PAGES) {
+		az_blob_dbg("number of DMA pages %lu buffer

exceeding %u\n",

Quoted 11 lines

drivers/hv/azure_blob.c line 338

Quoted 9 lines
drivers/hv/azure_blob.c new file +9 -0
+			total_num_pages, AZ_BLOB_MAX_PAGES);
+		ret = -EINVAL;
+		goto get_user_page_failed;
+	}
+
+	/* Construct a VMBUS packet and send it over to VSP */
+	desc_size = sizeof(struct vmbus_packet_mpb_array) +
+			sizeof(u64) * total_num_pages;
+	desc = kzalloc(desc_size, GFP_KERNEL);

Smells like a call for struct_size().

Yes it's a good idea to use struct_size. Will change this.

Quoted 62 lines
Quoted 46 lines
drivers/hv/azure_blob.c new file +44 -0
+	vsp_request = kzalloc(sizeof(*vsp_request), GFP_KERNEL);
+	if (!desc || !vsp_request) {
+		kfree(desc);
+		kfree(vsp_request);
+		ret = -ENOMEM;
+		goto get_user_page_failed;
+	}
+
+	desc->range.offset = 0;
+	desc->range.len = total_num_pages * PAGE_SIZE;
+	pfn_array = desc->range.pfn_array;
+	page_idx = 0;
+
+	if (request.data_len) {
+		fill_in_page_buffer(pfn_array, &page_idx, data_pages,
+			data_num_pages);
+		vsp_request->data_buffer_offset = data_start;
+		vsp_request->data_buffer_length = request.data_len;
+		vsp_request->data_buffer_valid = request.data_valid;
+	}
+
+	fill_in_page_buffer(pfn_array, &page_idx, request_pages,
+		request_num_pages);
+	vsp_request->request_buffer_offset = request_start +
+						data_num_pages * PAGE_SIZE;
+	vsp_request->request_buffer_length = request.request_len;
+
+	fill_in_page_buffer(pfn_array, &page_idx, response_pages,
+		response_num_pages);
+	vsp_request->response_buffer_offset = response_start +
+		(data_num_pages + request_num_pages) * PAGE_SIZE;
+	vsp_request->response_buffer_length = request.response_len;
+
+	vsp_request->version = 0;
+	vsp_request->timeout_ms = request.timeout;
+	vsp_request->operation_type = AZ_BLOB_DRIVER_USER_REQUEST;
+	guid_copy(&vsp_request->transaction_id, &request.guid);
+
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_add_tail(&request_ctx.list, &file_ctx->vsp_pending_requests);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+	az_blob_dbg("sending request to VSP\n");
+	az_blob_dbg("desc_size %u desc->range.len %u desc-

range.offset %u\n",

drivers/hv/azure_blob.c

Quoted 2 lines
drivers/hv/azure_blob.c new file +2 -0
+		desc_size, desc->range.len, desc->range.offset);
+	az_blob_dbg("vsp_request data_buffer_offset %u

data_buffer_length %u "

drivers/hv/azure_blob.c

Quoted 12 lines
drivers/hv/azure_blob.c new file +12 -0
+		"data_buffer_valid %u request_buffer_offset %u "
+		"request_buffer_length %u response_buffer_offset %u "
+		"response_buffer_length %u\n",
+		vsp_request->data_buffer_offset,
+		vsp_request->data_buffer_length,
+		vsp_request->data_buffer_valid,
+		vsp_request->request_buffer_offset,
+		vsp_request->request_buffer_length,
+		vsp_request->response_buffer_offset,
+		vsp_request->response_buffer_length);
+
+	ret = vmbus_sendpacket_mpb_desc(dev->device->channel, desc,

desc_size,

Quoted 11 lines

drivers/hv/azure_blob.c line 404

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+		vsp_request, sizeof(*vsp_request), (u64) &request_ctx);
+
+	kfree(desc);
+	kfree(vsp_request);
+	if (ret)
+		goto vmbus_send_failed;
+
+	wait_for_completion(&request_ctx.wait_vsp);

Provided this is ioctl, this should likely be interruptible. You don't want to account to I/O load. The same likely for az_blob_fop_release.

The reason for wait is that the memory for I/O is pinned and passed to the host for direct I/O. Now the host owns those memory, and we can't do anything to those memory until the host releases ownership.

Quoted 15 lines

drivers/hv/azure_blob.c line 414

Quoted 11 lines
drivers/hv/azure_blob.c new file +10 -0
+
+	/*
+	 * At this point, the response is already written to request
+	 * by VMBUS completion handler, copy them to user-mode buffers
+	 * and return to user-mode
+	 */
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.status),
+			&request.response.status,

This is ugly, why don't you make argp the appropriate pointer instead of char *? You'd need not do copy_to_user twice then, right?

Quoted 10 lines

drivers/hv/azure_blob.c line 422

Quoted 8 lines
drivers/hv/azure_blob.c new file +8 -0
+			sizeof(request.response.status))) {
+		ret = -EFAULT;
+		goto vmbus_send_failed;
+	}
+
+	if (copy_to_user(argp +
+			offsetof(struct az_blob_request_sync,
+				response.response_len),

The same here.

I'll fix the pointers.

Quoted 14 lines
Quoted 14 lines
drivers/hv/azure_blob.c new file +13 -0
+			&request.response.response_len,
+			sizeof(request.response.response_len)))
+		ret = -EFAULT;
+
+vmbus_send_failed:
+	spin_lock_irqsave(&file_ctx->vsp_pending_lock, flags);
+	list_del(&request_ctx.list);
+	if (list_empty(&file_ctx->vsp_pending_requests))
+		wake_up(&file_ctx->wait_vsp_pending);
+	spin_unlock_irqrestore(&file_ctx->vsp_pending_lock, flags);
+
+get_user_page_failed:
+	free_buffer_pages(request_num_pages, request_pages);
Quoted 8 lines

drivers/hv/azure_blob.c line 440

Quoted 5 lines
drivers/hv/azure_blob.c new file +5 -0
+	free_buffer_pages(response_num_pages, response_pages);
+	free_buffer_pages(data_num_pages, data_pages);
+
+	return ret;
+}

This function is overly long. Care to split it (e.g. moving away the initialization of the structs and the debug stuff)?

This is true. I'm looking for ways to refactor it.

Quoted 8 lines

drivers/hv/azure_blob.c line 445

Quoted 6 lines
drivers/hv/azure_blob.c new file +5 -0
+
+static long az_blob_fop_ioctl(struct file *filp, unsigned int cmd,
+	unsigned long arg)
+{
+	long ret = -EIO;

EINVAL would be more appropriate.

Good point. I'll fix it.

Quoted 8 lines

drivers/hv/azure_blob.c line 450

Quoted 6 lines
drivers/hv/azure_blob.c new file +5 -0
+
+	switch (cmd) {
+	case IOCTL_AZ_BLOB_DRIVER_USER_REQUEST:
+		if (_IOC_SIZE(cmd) != sizeof(struct az_blob_request_sync))
+			return -EINVAL;

How can that happen, provided the switch (cmd) and case?

I see some other drivers checking it. It's probably not needed. Will remove it.

Quoted 18 lines

drivers/hv/azure_blob.c line 452

Quoted 3 lines
drivers/hv/azure_blob.c new file +2 -0
+		ret = az_blob_ioctl_user_request(filp, arg);
+		break;

Simply:
return az_blob_ioctl_user_request(filp, arg);

drivers/hv/azure_blob.c line 458

Quoted 6 lines
drivers/hv/azure_blob.c new file +6 -0
+
+	default:
+		az_blob_dbg("unrecognized IOCTL code %u\n", cmd);
+	}
+
+	return ret;

So return -EINVAL here directly now.

Quoted 1 line

+}

...

Quoted 8 lines
Quoted 3 lines
drivers/hv/azure_blob.c new file +3 -0
+static int az_blob_connect_to_vsp(struct hv_device *device, u32
+ring_size) {
+	int ret;

...

drivers/hv/azure_blob.c

Quoted 1 line
drivers/hv/azure_blob.c new file +1 -0
+	int rc;

Sometimes ret, sometimes rc, could you unify the two?

Will fix this.

Quoted 10 lines

drivers/hv/azure_blob.c line 644

Quoted 6 lines
drivers/hv/azure_blob.c new file +5 -0
+static int __init az_blob_drv_init(void) {
+	int ret;
+
+	ret = vmbus_driver_register(&az_blob_drv);
+	return ret;

Simply:
return vmbus_driver_register(&az_blob_drv);

Quoted 5 lines

regards,
--
--
js
suse labs

Thank you!

Long

#26 lore raw

Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 5 lines · Long Li
Quoted 3 lines · On 26. 06. 21, 8:30, [email protected] wrote:

Have you fed this patch through checkpatch?

Yes, it didn't throw out any errors.

Several warnings and checks though.

$ ./scripts/checkpatch.pl 2.patch --strict --terse
2.patch:68: WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line)
2.patch:148: WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
2.patch:173: CHECK: spinlock_t definition without comment
2.patch:220: CHECK: spinlock_t definition without comment
2.patch:250: CHECK: Alignment should match open parenthesis
2.patch:255: CHECK: Alignment should match open parenthesis
2.patch:257: CHECK: Macro argument 'level' may be better as '(level)' to avoid precedence issues
2.patch:280: CHECK: Alignment should match open parenthesis
2.patch:283: CHECK: No space is necessary after a cast
2.patch:287: WARNING: quoted string split across lines
2.patch:296: CHECK: Blank lines aren't necessary before a close brace '}'
2.patch:303: CHECK: Please don't use multiple blank lines
2.patch:308: CHECK: Please don't use multiple blank lines
2.patch:331: CHECK: Alignment should match open parenthesis
2.patch:348: CHECK: Alignment should match open parenthesis
2.patch:362: CHECK: Alignment should match open parenthesis
2.patch:371: CHECK: Alignment should match open parenthesis
2.patch:381: CHECK: Alignment should match open parenthesis
2.patch:404: CHECK: No space is necessary after a cast
2.patch:426: WARNING: quoted string split across lines
2.patch:437: WARNING: quoted string split across lines
2.patch:438: WARNING: quoted string split across lines
2.patch:458: CHECK: No space is necessary after a cast
2.patch:459: CHECK: Alignment should match open parenthesis
2.patch:464: CHECK: No space is necessary after a cast
2.patch:465: CHECK: Alignment should match open parenthesis
2.patch:472: CHECK: Alignment should match open parenthesis
2.patch:472: CHECK: No space is necessary after a cast
2.patch:482: CHECK: Alignment should match open parenthesis
2.patch:506: CHECK: Alignment should match open parenthesis
2.patch:513: CHECK: Alignment should match open parenthesis
2.patch:519: CHECK: Alignment should match open parenthesis
2.patch:535: CHECK: Alignment should match open parenthesis
2.patch:537: WARNING: quoted string split across lines
2.patch:538: WARNING: quoted string split across lines
2.patch:539: WARNING: quoted string split across lines
2.patch:549: CHECK: Alignment should match open parenthesis
2.patch:549: CHECK: No space is necessary after a cast
2.patch:565: CHECK: Alignment should match open parenthesis
2.patch:574: CHECK: Alignment should match open parenthesis
2.patch:595: CHECK: Alignment should match open parenthesis
2.patch:634: WARNING: quoted string split across lines
2.patch:639: CHECK: Alignment should match open parenthesis
2.patch:643: CHECK: Alignment should match open parenthesis
2.patch:646: CHECK: Alignment should match open parenthesis
2.patch:648: CHECK: Alignment should match open parenthesis
2.patch:650: CHECK: Alignment should match open parenthesis
2.patch:694: CHECK: braces {} should be used on all arms of this statement
2.patch:696: CHECK: Alignment should match open parenthesis
2.patch:703: CHECK: Unbalanced braces around else statement
2.patch:724: CHECK: Alignment should match open parenthesis
2.patch:744: CHECK: Alignment should match open parenthesis
total: 0 errors, 10 warnings, 42 checks, 749 lines checked
#27 lore raw

RE: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 5 lines

Subject: Re: [Patch v2 2/3] Drivers: hv: add Azure Blob driver

Quoted 1 line · Long Li
Quoted 1 line · On 26. 06. 21, 8:30, [email protected] wrote:
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Have you fed this patch through checkpatch?

Yes, it didn't throw out any errors.

Several warnings and checks though.

$ ./scripts/checkpatch.pl 2.patch --strict --terse

Thanks for the pointer. I didn't check with "--strict".

Will fix this.

Quoted 57 lines

2.patch:68: WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line)
2.patch:148: WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
2.patch:173: CHECK: spinlock_t definition without comment 2.patch:220: CHECK: spinlock_t definition without comment 2.patch:250: CHECK: Alignment should match open parenthesis 2.patch:255: CHECK: Alignment should match open parenthesis 2.patch:257: CHECK: Macro argument 'level' may be better as '(level)' to avoid precedence issues
2.patch:280: CHECK: Alignment should match open parenthesis 2.patch:283: CHECK: No space is necessary after a cast 2.patch:287: WARNING: quoted string split across lines 2.patch:296: CHECK: Blank lines aren't necessary before a close brace '}' 2.patch:303: CHECK: Please don't use multiple blank lines 2.patch:308: CHECK: Please don't use multiple blank lines 2.patch:331: CHECK: Alignment should match open parenthesis 2.patch:348: CHECK: Alignment should match open parenthesis 2.patch:362: CHECK: Alignment should match open parenthesis 2.patch:371: CHECK: Alignment should match open parenthesis 2.patch:381: CHECK: Alignment should match open parenthesis 2.patch:404: CHECK: No space is necessary after a cast 2.patch:426: WARNING: quoted string split across lines 2.patch:437: WARNING: quoted string split across lines 2.patch:438: WARNING: quoted string split across lines 2.patch:458: CHECK: No space is necessary after a cast 2.patch:459: CHECK: Alignment should match open parenthesis 2.patch:464: CHECK: No space is necessary after a cast 2.patch:465: CHECK: Alignment should match open parenthesis 2.patch:472: CHECK: Alignment should match open parenthesis 2.patch:472: CHECK: No space is necessary after a cast 2.patch:482: CHECK: Alignment should match open parenthesis 2.patch:506: CHECK: Alignment should match open parenthesis 2.patch:513: CHECK: Alignment should match open parenthesis 2.patch:519: CHECK: Alignment should match open parenthesis 2.patch:535: CHECK: Alignment should match open parenthesis 2.patch:537: WARNING: quoted string split across lines 2.patch:538: WARNING: quoted string split across lines 2.patch:539: WARNING: quoted string split across lines 2.patch:549: CHECK: Alignment should match open parenthesis 2.patch:549: CHECK: No space is necessary after a cast 2.patch:565: CHECK: Alignment should match open parenthesis 2.patch:574: CHECK: Alignment should match open parenthesis 2.patch:595: CHECK: Alignment should match open parenthesis 2.patch:634: WARNING: quoted string split across lines 2.patch:639: CHECK: Alignment should match open parenthesis 2.patch:643: CHECK: Alignment should match open parenthesis 2.patch:646: CHECK: Alignment should match open parenthesis 2.patch:648: CHECK: Alignment should match open parenthesis 2.patch:650: CHECK: Alignment should match open parenthesis 2.patch:694: CHECK: braces {} should be used on all arms of this statement 2.patch:696: CHECK: Alignment should match open parenthesis 2.patch:703: CHECK: Unbalanced braces around else statement 2.patch:724: CHECK: Alignment should match open parenthesis 2.patch:744: CHECK: Alignment should match open parenthesis
total: 0 errors, 10 warnings, 42 checks, 749 lines checked

#28 lore raw

PATCH v2 3/3 Drivers: hv: Add to maintainer for Azure Blob driver

In reply to [email protected]

From: Long Li <[email protected]>

Add [email protected] to maintainer list for Azure Blob driver.

Cc K. Y. Srinivasan <[email protected]>
Cc Haiyang Zhang <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Wei Liu <[email protected]>
Cc Dexuan Cui <[email protected]>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)
MAINTAINERS +1 -0
index 9487061..b547eb9 100644
@@ -8440,6 +8440,7 @@ M:	Haiyang Zhang <[email protected]>
 M:	Stephen Hemminger <[email protected]>
 M:	Wei Liu <[email protected]>
 M:	Dexuan Cui <[email protected]>
+M:	Long Li <[email protected]>
 L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
Signature · 1 line
1.8.3.1