myriad-ma24xx-vsc: Helper fw driver for AI inference accelerator
cover myriad-ma24xx-vsc: Helper fw driver for AI inference accelerator
So this series is an effort to start a conversation in the community, with the intent to free the Intel(R) Myriad(TM) ma24xx stack, as found within:
* USB Intel Neural Compute Stick (ma2450)
* USB Intel Neural Compute Stick 2 (ma2485)
What this patch series is:
* RFC for a helper driver that loads a (currently) non-redistributable firmware binary blob for this hotswappable AI inference hardware.
What this patch series is NOT:
* Kernel driver that provides clean API access to this AI accelerator.
* Any usermode support.
Hardware:
=======================================================================
The Myriad ma24xx in the USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.
This VPU chip is positioned as a specialised low-power processor chip for computer vision inference applications. It contains a dedicated hardware accelerator for neural network deep-learning inferences.
It was originally designed by Movidius Ltd, who announced they were to be acquired by Intel in September 2016.
Boot process:
=======================================================================
The core chip is housed within a USB enclosure (there is a different variant embedded on a SoC over a local bus, but that's out of scope).
When physically connected to a host system, the accelerator card first needs firmware to be loaded for the device to be useable. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.
Current software stack:
=======================================================================
+ *
+ * The firmware is non-free and must be extracted by the user.
+ */
Intel packages and distributes a downstream vendor Intel(R) Distribution of OpenVINO(TM) toolkit [0] (under a limited proprietary license) for these devices which, amongst other binary packages, contains the Deep Learning Deployment Toolkit [1] (Apache License Version 2.0).
However, the non-redistributable firmware binary blob that must be loaded onto the Myriad ma24xx is not available under a suitable FOSS license.
There have been periods of time during which support for non-x86 host platforms (e.g. Raspberry Pi ARMv8 64-bit) were missing or delayed in the Intel-distributed downstream vendor fork OpenVINO.
Whilst Movidius was an independent company, a separate (also non-FOSS) "Myriad Development Kit" was made available for developers, although it is no longer actively supported nor new features developed for it.
Let's try to open more of this stack to FOSS mainline, on any host!
This patch series would provide mainline Linux kernel support to load the Myriad ma24xx firmware, were it to become freely licensed.
As the linux-accelerators list is forming as somewhat of a home for Linux AI accelerators (training and inference) subsystem, I have copied that list as a heads-up to the larger effort to integrate this hardware into a fully open stack.
May the RFC comments flood in!
[0] https://software.intel.com/en-us/openvino-toolkit
[1] https://github.com/opencv/dldt
Note:
Intel, the Intel logo, Movidius, Myriad, OpenVINO and the OpenVINO logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.
Rhys Kidd (1):
USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx
MAINTAINERS | 6 + drivers/usb/misc/Kconfig | 8 ++ drivers/usb/misc/Makefile | 1 + drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++ 4 files changed, 186 insertions(+) create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.c
Signature · 1 line
2.20.1
RFC PATCH 1/1 USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx
The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.
However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.
--- MAINTAINERS | 6 + drivers/usb/misc/Kconfig | 8 ++ drivers/usb/misc/Makefile | 1 + drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++ 4 files changed, 186 insertions(+) create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.c
MAINTAINERS +6 -0
@@ -16682,6 +16682,12 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git S: Maintained F: sound/usb/midi.* +USB MYRIAD MA24XX FIRMWARE DRIVER +M: Rhys Kidd <[email protected]> +L: [email protected] +S: Maintained +F: drivers/usb/misc/myriad-ma24xx-vsc.c + USB NETWORKING DRIVERS L: [email protected] S: Odd Fixes
drivers/usb/misc/Kconfig +8 -0
@@ -275,3 +275,11 @@ config USB_CHAOSKEY To compile this driver as a module, choose M here: the module will be called chaoskey. + +config USB_MYRIAD_MA24XX_VSC + tristate "USB Intel Myriad MA24xx firmware loading support" + select FW_LOADER + help + This driver loads firmware for Myriad ma24xx AI inference accelerators, as + found in the USB Intel Neural Compute Stick (ma2450) and Intel Neural + Compute Stick 2 (ma2485).
drivers/usb/misc/Makefile +1 -0
@@ -17,6 +17,7 @@ obj-$(CONFIG_USB_ISIGHTFW) += isight_firmware.o obj-$(CONFIG_USB_LCD) += usblcd.o obj-$(CONFIG_USB_LD) += ldusb.o obj-$(CONFIG_USB_LEGOTOWER) += legousbtower.o +obj-$(CONFIG_USB_MYRIAD_MA24XX_VSC) += myriad-ma24xx-vsc.o obj-$(CONFIG_USB_RIO500) += rio500.o obj-$(CONFIG_USB_TEST) += usbtest.o obj-$(CONFIG_USB_EHSET_TEST_FIXTURE) += ehset.o
drivers/usb/misc/myriad-ma24xx-vsc.c new file +171 -0
171 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [RFC PATCH 1/1] USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx
Quoted 9 lines · Rhys Kidd
The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.
However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.
Is there any particular reason you need to have a kernel driver load the firmware? Why can't a user program take care of it? The usb_modeswitch program does that sort of thing all the time.
Alan Stern
Re: [RFC PATCH 1/1] USB: myriad-ma24xx-vsc: Firmware loader driver for USB Myriad ma24xx
In reply to Rhys KiddAm Montag, den 16.09.2019, 21:03 +1000 schrieb Rhys Kidd:
Quoted 9 lines
The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural Compute Stick 2 provides an API to accelerate AI inference calculations on the dedicated SHAVE VLIW vector co-processors, which are orchestrated by one or more LEON SPARC v8 real time cores.
However, they need firmware to be loaded beforehand. An uninitialised Myriad ma24xx presents with a distinctive USB ID. After firmware loading, the device detaches from the USB bus and reattaches with a new device ID. It can then be claimed by the usermode driver.
Why are you using a kernel space driver for this?
Quoted 82 lines
Signed-off-by Rhys Kidd <[email protected]>--- MAINTAINERS | 6 + drivers/usb/misc/Kconfig | 8 ++ drivers/usb/misc/Makefile | 1 + drivers/usb/misc/myriad-ma24xx-vsc.c | 171 +++++++++++++++++++++++++++ 4 files changed, 186 insertions(+) create mode 100644 drivers/usb/misc/myriad-ma24xx-vsc.cMAINTAINERS +6
-0@@ -16682,6 +16682,12 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git S: Maintained F: sound/usb/midi.* +USB MYRIAD MA24XX FIRMWARE DRIVER +M: Rhys Kidd <[email protected]> +L: [email protected] +S: Maintained +F: drivers/usb/misc/myriad-ma24xx-vsc.c + USB NETWORKING DRIVERS L: [email protected] S: Odd Fixesdrivers/usb/misc/Kconfig +8
-0@@ -275,3 +275,11 @@ config USB_CHAOSKEY To compile this driver as a module, choose M here: the module will be called chaoskey. + +config USB_MYRIAD_MA24XX_VSC + tristate "USB Intel Myriad MA24xx firmware loading support" + select FW_LOADER + help + This driver loads firmware for Myriad ma24xx AI inference accelerators, as + found in the USB Intel Neural Compute Stick (ma2450) and Intel Neural + Compute Stick 2 (ma2485).drivers/usb/misc/Makefile +1
-0@@ -17,6 +17,7 @@ obj-$(CONFIG_USB_ISIGHTFW) += isight_firmware.o obj-$(CONFIG_USB_LCD) += usblcd.o obj-$(CONFIG_USB_LD) += ldusb.o obj-$(CONFIG_USB_LEGOTOWER) += legousbtower.o +obj-$(CONFIG_USB_MYRIAD_MA24XX_VSC) += myriad-ma24xx-vsc.o obj-$(CONFIG_USB_RIO500) += rio500.o obj-$(CONFIG_USB_TEST) += usbtest.o obj-$(CONFIG_USB_EHSET_TEST_FIXTURE) += ehset.odrivers/usb/misc/myriad-ma24xx-vsc.c new file +21
-0@@ -0,0 +1,171 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Driver for loading USB Myriad ma24xx firmware +* +* Copyright (C) 2018 Rhys Kidd <[email protected]> +* +* The Myriad ma24xx in USB Intel Neural Compute Stick and Intel Neural +* Compute Stick 2 provides an API to accelerate AI inference calculations +* on the dedicated SHAVE VLIW vector co-processors, which are orchestrated +* by one or more LEON SPARC v8 real time cores. +* +* However, they need firmware to be loaded beforehand. An uninitialised +* Myriad ma24xx presents with a distinctive USB ID. After firmware +* loading, the device detaches from the USB bus and reattaches with a new +* device ID. It can then be claimed by the usermode driver. +* +* The firmware is non-free and must be extracted by the user. +*/ + +/* Standard include files */ +#include <linux/errno.h>
drivers/usb/misc/myriad-ma24xx-vsc.c line 31
Quoted 10 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +10
-0+#include <linux/firmware.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/usb.h> + +#define usb_dbg(usb_if, format, arg...) \ + dev_dbg(&(usb_if)->dev, format, ## arg) + +#define usb_err(usb_if, format, arg...) \ + dev_err(&(usb_if)->dev, format, ## arg)
Why?
Quoted 79 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +79
-0+/* Version Information */ +#define DRIVER_AUTHOR "Rhys Kidd <[email protected]>" +#define DRIVER_DESC "Driver for loading USB Myriad ma24xx firmware" +#define DRIVER_SHORT "myriad_ma24xx_vsc" + +MODULE_AUTHOR (DRIVER_AUTHOR ); +MODULE_DESCRIPTION (DRIVER_DESC ); +MODULE_LICENSE ("GPL" ); + +#define FW_DIR "myriad/" +#define MA2450_FIRMWARE FW_DIR "mvncapi-2450.mvcmd" +#define MA2480_FIRMWARE FW_DIR "mvncapi-2480.mvcmd" + +MODULE_FIRMWARE (MA2450_FIRMWARE ); +MODULE_FIRMWARE (MA2480_FIRMWARE ); + +enum { +MA2450FW =0 , +MA2480FW +}; + +/* macros for struct usb_device_id */ +#define MYRIAD_CHIP_VERSION (x) \ + ((x)->driver_info & 0xf) + +#define MYRIAD_VID 0x03e7/* Movidius Ltd, an Intel company */ +#define MA2450_PID 0x2150/* ma2450 VSC loopback device, without fw */ +#define MA2485_PID 0x2485/* ma2485 VSC loopback device, without fw */ + +#define MYRIAD_BUF_LEN 512/* max size of USB_SPEED_HIGH packet */ +#define WRITE_TIMEOUT 2000/* milliseconds */ + +static const struct usb_device_id id_table[] = { + {USB_DEVICE (MYRIAD_VID ,MA2450_PID ), .driver_info =MA2450FW }, + {USB_DEVICE (MYRIAD_VID ,MA2485_PID ), .driver_info =MA2480FW }, + { }, +}; +MODULE_DEVICE_TABLE (usb, id_table); + +static int myriad_ma24xx_vsc_probe (struct usb_interface *intf, +const struct usb_device_id *id) +{ +struct usb_device *dev =interface_to_usbdev (intf); +struct usb_host_interface *altsetting = intf->cur_altsetting; +struct usb_endpoint_descriptor *epd; +int out_ep, res, size; +const struct firmware *firmware =NULL ; +const u8 *ptr; +int offset, ret =0 ; +char *buf; +char *fw_family_name; + +usb_dbg (intf,"probe %s-%s" , dev->product, dev->serial); + +/* Find the first bulk OUT endpoint and its packet size */ + res =usb_find_bulk_out_endpoint (altsetting, &epd); +if (res) { +usb_err (intf,"no OUT endpoint found" ); +return res; + } + + out_ep =usb_endpoint_num (epd); + size =usb_endpoint_maxp (epd); + +/* Validate endpoint and size */ +if (size <=0 ) { +usb_err (intf,"invalid size (%d)" , size); +return -ENODEV ; + } + +if (size >MYRIAD_BUF_LEN ) { +usb_dbg (intf,"size reduced %d -> %d\n" , size,MYRIAD_BUF_LEN ); + size =MYRIAD_BUF_LEN ; + } + + buf =kmalloc (size,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; +
drivers/usb/misc/myriad-ma24xx-vsc.c line 121
Quoted 10 lines
drivers/usb/misc/myriad-ma24xx-vsc.c new file +10
-0+switch (MYRIAD_CHIP_VERSION (id)) { +case MA2450FW : + fw_family_name =MA2450_FIRMWARE ; +break ; +case MA2480FW : + fw_family_name =MA2480_FIRMWARE ; +break ; +default : +usb_err (intf,"unsupported myriad ma24xx firmware family\n" ); +return -ENODEV ;
Memory leak
Regards
Oliver