Skip to contentSkip to conversation

myriad-ma24xx-vsc: Helper fw driver for AI inference accelerator

RFC PATCH In review Rhys Kidd 3 participants 4 messages Sep 16, 2019 – Sep 17, 2019 USB

lore
Share & tools
lore

Rhys Kidd

author
#0 lore raw

cover myriad-ma24xx-vsc: Helper fw driver for AI inference accelerator

So this series is an effort to start a conversation in the community, with the intent to free the Intel(R) Myriad(TM) ma24xx stack, as found within:

* USB Intel Neural Compute Stick (ma2450)
* USB Intel Neural Compute Stick 2 (ma2485)

What this patch series is:
* RFC for a helper driver that loads a (currently) non-redistributable firmware binary blob for this hotswappable AI inference hardware.

What this patch series is NOT:
* Kernel driver that provides clean API access to this AI accelerator.
* Any usermode support.

Hardware:
=======================================================================
The Myriad ma24xx in the USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.

This VPU chip is positioned as a specialised low-power processor chip for computer vision inference applications. It contains a dedicated hardware accelerator for neural network deep-learning inferences.

It was originally designed by Movidius Ltd, who announced they were to be acquired by Intel in September 2016.

Boot process:
=======================================================================
The core chip is housed within a USB enclosure (there is a different variant embedded on a SoC over a local bus, but that's out of scope).

When physically connected to a host system, the accelerator card first needs firmware to be loaded for the device to be useable. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.

Current software stack:
=======================================================================

+ *
+ * The firmware is non-free and must be extracted by the user.
+ */

Intel packages and distributes a downstream vendor Intel(R) Distribution of OpenVINO(TM) toolkit [0] (under a limited proprietary license) for these devices which, amongst other binary packages, contains the Deep Learning Deployment Toolkit [1] (Apache License Version 2.0).

However, the non-redistributable firmware binary blob that must be loaded onto the Myriad ma24xx is not available under a suitable FOSS license.

There have been periods of time during which support for non-x86 host platforms (e.g. Raspberry Pi ARMv8 64-bit) were missing or delayed in the Intel-distributed downstream vendor fork OpenVINO.

Whilst Movidius was an independent company, a separate (also non-FOSS) "Myriad Development Kit" was made available for developers, although it is no longer actively supported nor new features developed for it.

Let's try to open more of this stack to FOSS mainline, on any host!

This patch series would provide mainline Linux kernel support to load the Myriad ma24xx firmware, were it to become freely licensed.

As the linux-accelerators list is forming as somewhat of a home for Linux AI accelerators (training and inference) subsystem, I have copied that list as a heads-up to the larger effort to integrate this hardware into a fully open stack.

May the RFC comments flood in!

[0] https://software.intel.com/en-us/openvino-toolkit
[1] https://github.com/opencv/dldt

Note:
Intel, the Intel logo, Movidius, Myriad, OpenVINO and the OpenVINO logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

Rhys Kidd (1):
USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx

 MAINTAINERS                          |   6 +
 drivers/usb/misc/Kconfig             |   8 ++
 drivers/usb/misc/Makefile            |   1 +
 drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++
 4 files changed, 186 insertions(+)
 create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.c
Signature · 1 line
2.20.1
author
#1 lore raw

RFC PATCH 1/1 USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx

The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.

However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.

---
 MAINTAINERS                          |   6 +
 drivers/usb/misc/Kconfig             |   8 ++
 drivers/usb/misc/Makefile            |   1 +
 drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++
 4 files changed, 186 insertions(+)
 create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.c
MAINTAINERS +6 -0
index a50e97a63bc8..2c3ab39ac26a 100644
@@ -16682,6 +16682,12 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
 S:	Maintained
 F:	sound/usb/midi.*
 
+USB MYRIAD MA24XX FIRMWARE DRIVER
+M:	Rhys Kidd <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/usb/misc/myriad-ma24xx-vsc.c
+
 USB NETWORKING DRIVERS
 L:	[email protected]
 S:	Odd Fixes
drivers/usb/misc/Kconfig +8 -0
index bdae62b2ffe0..5d600fb8ac50 100644
@@ -275,3 +275,11 @@ config USB_CHAOSKEY
 
 	  To compile this driver as a module, choose M here: the
 	  module will be called chaoskey.
+
+config USB_MYRIAD_MA24XX_VSC
+	tristate "USB Intel Myriad MA24xx firmware loading support"
+	select FW_LOADER
+	help
+	  This driver loads firmware for Myriad ma24xx AI inference accelerators, as
+	  found in the USB Intel Neural Compute Stick (ma2450) and Intel Neural
+	  Compute Stick 2 (ma2485).
\ No newline at end of file
drivers/usb/misc/Makefile +1 -0
index 109f54f5b9aa..e19d1348c5b6 100644
@@ -17,6 +17,7 @@ obj-$(CONFIG_USB_ISIGHTFW)		+= isight_firmware.o
 obj-$(CONFIG_USB_LCD)			+= usblcd.o
 obj-$(CONFIG_USB_LD)			+= ldusb.o
 obj-$(CONFIG_USB_LEGOTOWER)		+= legousbtower.o
+obj-$(CONFIG_USB_MYRIAD_MA24XX_VSC)    += myriad-ma24xx-vsc.o
 obj-$(CONFIG_USB_RIO500)		+= rio500.o
 obj-$(CONFIG_USB_TEST)			+= usbtest.o
 obj-$(CONFIG_USB_EHSET_TEST_FIXTURE)    += ehset.o
drivers/usb/misc/myriad-ma24xx-vsc.c new file +171 -0

171 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw

Re: [RFC PATCH 1/1] USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx

Quoted 9 lines · Rhys Kidd

The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.

However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.

Is there any particular reason you need to have a kernel driver load the firmware? Why can't a user program take care of it? The usb_modeswitch program does that sort of thing all the time.

Alan Stern

#3 lore raw

Re: [RFC PATCH 1/1] USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx

In reply to Rhys Kidd

Am Montag, den 16.09.2019, 21:03 +1000 schrieb Rhys Kidd:

Quoted 9 lines

The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.

However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.

Why are you using a kernel space driver for this?

Quoted 82 lines
---
 MAINTAINERS                          |   6 +
 drivers/usb/misc/Kconfig             |   8 ++
 drivers/usb/misc/Makefile            |   1 +
 drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++
 4 files changed, 186 insertions(+)
 create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.c
MAINTAINERS +6 -0
index a50e97a63bc8..2c3ab39ac26a 100644
@@ -16682,6 +16682,12 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
 S:	Maintained
 F:	sound/usb/midi.*
 
+USB MYRIAD MA24XX FIRMWARE DRIVER
+M:	Rhys Kidd <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/usb/misc/myriad-ma24xx-vsc.c
+
 USB NETWORKING DRIVERS
 L:	[email protected]
 S:	Odd Fixes
drivers/usb/misc/Kconfig +8 -0
index bdae62b2ffe0..5d600fb8ac50 100644
@@ -275,3 +275,11 @@ config USB_CHAOSKEY
 
 	  To compile this driver as a module, choose M here: the
 	  module will be called chaoskey.
+
+config USB_MYRIAD_MA24XX_VSC
+	tristate "USB Intel Myriad MA24xx firmware loading support"
+	select FW_LOADER
+	help
+	  This driver loads firmware for Myriad ma24xx AI inference accelerators, as
+	  found in the USB Intel Neural Compute Stick (ma2450) and Intel Neural
+	  Compute Stick 2 (ma2485).
\ No newline at end of file
drivers/usb/misc/Makefile +1 -0
index 109f54f5b9aa..e19d1348c5b6 100644
@@ -17,6 +17,7 @@ obj-$(CONFIG_USB_ISIGHTFW)		+= isight_firmware.o
 obj-$(CONFIG_USB_LCD)			+= usblcd.o
 obj-$(CONFIG_USB_LD)			+= ldusb.o
 obj-$(CONFIG_USB_LEGOTOWER)		+= legousbtower.o
+obj-$(CONFIG_USB_MYRIAD_MA24XX_VSC)    += myriad-ma24xx-vsc.o
 obj-$(CONFIG_USB_RIO500)		+= rio500.o
 obj-$(CONFIG_USB_TEST)			+= usbtest.o
 obj-$(CONFIG_USB_EHSET_TEST_FIXTURE)    += ehset.o
drivers/usb/misc/myriad-ma24xx-vsc.c new file +21 -0
new file mode 100644
index 000000000000..f516c236a8f5
@@ -0,0 +1,171 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Driver for loading USB Myriad ma24xx firmware
+ *
+ * Copyright (C) 2018 Rhys Kidd <[email protected]>
+ *
+ * The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural
+ * Compute Stick 2 provides an API to accelerate AI inference calculations
+ * on the dedicated SHAVE VLIW vector co-processors, which are orchestrated
+ * by one or more LEON SPARC v8 real time cores.
+ *
+ * However, they need firmware to be loaded beforehand. An uninitialised
+ * Myriad ma24xx presents with a distinctive USB ID. After firmware
+ * loading, the device detaches from the USB bus and reattaches with a new
+ * device ID. It can then be claimed by the usermode driver.
+ *
+ * The firmware is non-free and must be extracted by the user.
+ */
+
+/* Standard include files */
+#include <linux/errno.h>

drivers/usb/misc/myriad-ma24xx-vsc.c line 31

Quoted 10 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +10 -0
+#include <linux/firmware.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/usb.h>
+
+#define usb_dbg(usb_if, format, arg...) \
+	dev_dbg(&(usb_if)->dev, format, ## arg)
+
+#define usb_err(usb_if, format, arg...) \
+	dev_err(&(usb_if)->dev, format, ## arg)

Why?

Quoted 79 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +79 -0
+/* Version Information */
+#define DRIVER_AUTHOR "Rhys Kidd <[email protected]>"
+#define DRIVER_DESC   "Driver for loading USB Myriad ma24xx firmware"
+#define DRIVER_SHORT  "myriad_ma24xx_vsc"
+
+MODULE_AUTHOR(DRIVER_AUTHOR);
+MODULE_DESCRIPTION(DRIVER_DESC);
+MODULE_LICENSE("GPL");
+
+#define FW_DIR "myriad/"
+#define MA2450_FIRMWARE FW_DIR "mvncapi-2450.mvcmd"
+#define MA2480_FIRMWARE FW_DIR "mvncapi-2480.mvcmd"
+
+MODULE_FIRMWARE(MA2450_FIRMWARE);
+MODULE_FIRMWARE(MA2480_FIRMWARE);
+
+enum {
+	MA2450FW = 0,
+	MA2480FW
+};
+
+/* macros for struct usb_device_id */
+#define MYRIAD_CHIP_VERSION(x) \
+	((x)->driver_info & 0xf)
+
+#define MYRIAD_VID  0x03e7   /* Movidius Ltd, an Intel company */
+#define MA2450_PID  0x2150   /* ma2450 VSC loopback device, without fw */
+#define MA2485_PID  0x2485   /* ma2485 VSC loopback device, without fw */
+
+#define MYRIAD_BUF_LEN 512   /* max size of USB_SPEED_HIGH packet */
+#define WRITE_TIMEOUT  2000  /* milliseconds */
+
+static const struct usb_device_id id_table[] = {
+	{ USB_DEVICE(MYRIAD_VID, MA2450_PID), .driver_info = MA2450FW },
+	{ USB_DEVICE(MYRIAD_VID, MA2485_PID), .driver_info = MA2480FW },
+	{ },
+};
+MODULE_DEVICE_TABLE(usb, id_table);
+
+static int myriad_ma24xx_vsc_probe(struct usb_interface *intf,
+				    const struct usb_device_id *id)
+{
+	struct usb_device *dev = interface_to_usbdev(intf);
+	struct usb_host_interface *altsetting = intf->cur_altsetting;
+	struct usb_endpoint_descriptor *epd;
+	int out_ep, res, size;
+	const struct firmware *firmware = NULL;
+	const u8 *ptr;
+	int offset, ret = 0;
+	char *buf;
+	char *fw_family_name;
+
+	usb_dbg(intf, "probe %s-%s", dev->product, dev->serial);
+
+	/* Find the first bulk OUT endpoint and its packet size */
+	res = usb_find_bulk_out_endpoint(altsetting, &epd);
+	if (res) {
+		usb_err(intf, "no OUT endpoint found");
+		return res;
+	}
+
+	out_ep = usb_endpoint_num(epd);
+	size = usb_endpoint_maxp(epd);
+
+	/* Validate endpoint and size */
+	if (size <= 0) {
+		usb_err(intf, "invalid size (%d)", size);
+		return -ENODEV;
+	}
+
+	if (size > MYRIAD_BUF_LEN) {
+		usb_dbg(intf, "size reduced %d -> %d\n", size, MYRIAD_BUF_LEN);
+		size = MYRIAD_BUF_LEN;
+	}
+
+	buf = kmalloc(size, GFP_KERNEL);
+	if (!buf)
+		return -ENOMEM;
+

drivers/usb/misc/myriad-ma24xx-vsc.c line 121

Quoted 10 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +10 -0
+	switch (MYRIAD_CHIP_VERSION(id)) {
+	case MA2450FW:
+		fw_family_name = MA2450_FIRMWARE;
+		break;
+	case MA2480FW:
+		fw_family_name = MA2480_FIRMWARE;
+		break;
+	default:
+		usb_err(intf, "unsupported myriad ma24xx firmware family\n");
+		return -ENODEV;

Memory leak

Regards
Oliver