Skip to contentSkip to conversation

Qualcomm Cloud AI 100 driver

RFC PATCH Superseded Jeffrey Hugo 6 participants 40 messages May 14, 2020 – May 20, 2020

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Qualcomm Cloud AI 100 driver

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Design:

+--------------------------------+
|       AI application           |
|       (userspace)              |
+-------------+------------------+
              |
              | Char dev interface
              |
              |
+-------------+------------------+
|       QAIC driver              |
|       (kernel space)           |
|                                |
+----+------------------+--------+
     |                  |
     |                  |
     |                  |
     |                  |
     |Control path      | Data path
     |(MHI bus)         |
     |                  |
     |                  |
     |                  |
     |                  |
+--------------------------------+
| +--------+      +------------+ |
| | MHI HW |      |DMA Bridge  | |
| +--------+      |(DMA engine)| |
|                 +------------+ |
|                                |
|                                |
|                                |
|  Qualcomm Cloud AI 100 device  |
|                                |
|                                |
+--------------------------------+

A Qualcomm Cloud AI 100 device (QAIC device from here on) is a PCIe hardware accelerator for AI inference workloads. Over the PCIe bus fabric, a QAIC device exposes two interfaces via PCI BARs - a MHI hardware region and a DMA engine hardware region.

Before workloads can be run, a QAIC device needs to be initialized. Similar to other Qualcomm products with incorperate MHI, device firmware needs to be loaded onto the device from the host. This occurs in two stages. First, a secondary bootloader (SBL) needs to be loaded onto the device. This occurs via the BHI protocol, and is handled by the MHI bus. Once the SBL loaded and running, it activates the Sahara protocol. The Sahara protocol is used with a userspace application to load and initialize the remaining firmware. The Sahara protocol and associated userspace application are outside the scope of this series as they have no direct interaction with the QAIC driver.

Once a QAIC device is fully initialized, workloads can be sent to the device and run. This involves a per-device instance char dev that the QAIC driver exposes to userspace. Running a workload involves two phases - configuring the device, and interacting with the workload.

To configure the device, commands are sent via a MHI channel. This is referred to as the control path. A command is a single message. A message contains one or more transactions. Transactions are operations that the device is requested to perform. Most commands are opaque to the kernel driver, however some are not. For example, if the user application wishes to DMA data to the device, it requires the assistance of the kernel driver to translate the data addresses to an address space that the device can understand. In this instance the transaction for DMAing the data is visible to the kernel driver, and the driver will do the required transformation when encoding the message.

To interact with the workload, the workload is assigned a DMA Bridge Channel (dbc). This is dedicated hardware within the DMA engine. Interacting with the workload consists of sending it input data, and receiving output data. The user application requests appropiate buffers from the kernel driver, prepares the buffers, and directs the kernel driver to queue them to the hardware.

The kernel driver is required to support multiple QAIC devices, and also N users per device.

Status:
This series introduces the driver for QAIC devices, and builds up the minimum functionality for running workloads. Several features which have been omitted or are still planned are indicated in the future work section.

Before exiting the RFC phase, and attempting full acceptance, we wish to complete two features which are currently under development as we expect there to be userspace interface changes as a result.

The first feature is a variable length control message between the kernel driver and the device. This allows us to support the total number of DMA transactions we require for certain platforms, while minimizing memory usage. The interface impact of this would be to allow us to drop the size of the manage buffer between userspace and the kernel driver from the current 16k, much of which is wasted.

The second feature is an optimization and extension of the data path interface. We plan to move the bulk of the data in the qaic_execute structure to the qaic_mem_req structure, which optimized our critical path processing. We also plan to extend the qaic_execute structure to allow for a batch submit of multiple buffers as an optimization and convenience for userspace.

Future work:
For simplicity, we have omitted work related to the following features, and intend to submit in future series:

-debugfs
-trace points
-hwmon (device telemetry)

We are also investigating what it might mean to support dma_bufs. We expect that such support would come as an extension of the interface.

Jeffrey Hugo (8):
qaic: Add skeleton driver
qaic: Add and init a basic mhi controller
qaic: Create char dev
qaic: Implement control path
qaic: Implement data path
qaic: Implement PCI link status error handlers
qaic: Implement MHI error status handler
MAINTAINERS: Add entry for QAIC driver

 MAINTAINERS                        |    7 +
 drivers/misc/Kconfig               |    1 +
 drivers/misc/Makefile              |    1 +
 drivers/misc/qaic/Kconfig          |   20 +
 drivers/misc/qaic/Makefile         |   12 +
 drivers/misc/qaic/mhi_controller.c |  538 +++++++++++++++++++
 drivers/misc/qaic/mhi_controller.h |   20 +
 drivers/misc/qaic/qaic.h           |  111 ++++
 drivers/misc/qaic/qaic_control.c   | 1015 ++++++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_data.c      |  952 +++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_drv.c       |  699 +++++++++++++++++++++++++
 include/uapi/misc/qaic.h           |  246 +++++++++
 12 files changed, 3622 insertions(+)
 create mode 100644 drivers/misc/qaic/Kconfig
 create mode 100644 drivers/misc/qaic/Makefile
 create mode 100644 drivers/misc/qaic/mhi_controller.c
 create mode 100644 drivers/misc/qaic/mhi_controller.h
 create mode 100644 drivers/misc/qaic/qaic.h
 create mode 100644 drivers/misc/qaic/qaic_control.c
 create mode 100644 drivers/misc/qaic/qaic_data.c
 create mode 100644 drivers/misc/qaic/qaic_drv.c
 create mode 100644 include/uapi/misc/qaic.h
Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#1 lore raw

RFC PATCH 1/8 qaic: Add skeleton driver

Add the Qualcomm Cloud AI 100 skeleton driver that does nothing but registers a pci driver, probes on the proper device, and supports removal of the module.

---
 drivers/misc/Kconfig         |  1 +
 drivers/misc/Makefile        |  1 +
 drivers/misc/qaic/Kconfig    | 20 ++++++++++++++
 drivers/misc/qaic/Makefile   | 10 +++++++
 drivers/misc/qaic/qaic_drv.c | 62 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 94 insertions(+)
 create mode 100644 drivers/misc/qaic/Kconfig
 create mode 100644 drivers/misc/qaic/Makefile
 create mode 100644 drivers/misc/qaic/qaic_drv.c
drivers/misc/Kconfig +1 -0
index 99e1514..ef91d062 100644
@@ -481,4 +481,5 @@ source "drivers/misc/ocxl/Kconfig"
 source "drivers/misc/cardreader/Kconfig"
 source "drivers/misc/habanalabs/Kconfig"
 source "drivers/misc/uacce/Kconfig"
+source "drivers/misc/qaic/Kconfig"
 endmenu
drivers/misc/Makefile +1 -0
index 9abf292..64f0770 100644
@@ -58,3 +58,4 @@ obj-$(CONFIG_PVPANIC)   	+= pvpanic.o
 obj-$(CONFIG_HABANA_AI)		+= habanalabs/
 obj-$(CONFIG_UACCE)		+= uacce/
 obj-$(CONFIG_XILINX_SDFEC)	+= xilinx_sdfec.o
+obj-$(CONFIG_QAIC)		+= qaic/
drivers/misc/qaic/Kconfig new file +20 -0
new file mode 100644
index 0000000..502d176
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Qualcomm Cloud AI 100 accelerators driver
+#
+
+config QAIC
+	tristate "Qualcomm Cloud AI 100 accelerators"
+	depends on PCI && HAS_IOMEM
+	depends on MHI_BUS
+	help
+	  Enables driver for Qualcomm's Cloud AI 100 accelerator PCIe cards
+	  that are designed to accelerate Deep Learning inference workloads.
+
+	  The driver manages the PCIe devices and provides an IOCTL interface
+	  for users to submit workloads to the devices.
+
+	  If unsure, say N.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called qaic.
drivers/misc/qaic/Makefile new file +10 -0
new file mode 100644
index 0000000..42149ac
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Makefile for Qualcomm Cloud AI 100 accelerators driver
+#
+
+
+obj-$(CONFIG_QAIC)	:= qaic.o
+
+qaic-y := \
+	qaic_drv.o
drivers/misc/qaic/qaic_drv.c new file +62 -0

62 lines not on this page · raw message

Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#2 lore raw

Re: [RFC PATCH 1/8] qaic: Add skeleton driver

Quoted 1 line · Jeffrey Hugo

+#define QAIC_NAME "Qualcomm Cloud AI 100"

<snip>

drivers/misc/qaic/qaic_drv.c line 33

Quoted 2 lines
drivers/misc/qaic/qaic_drv.c new file +2 -0
+static struct pci_driver qaic_pci_driver = {
+	.name = QAIC_NAME,

A question about the community's preference here.

Our driver name is very verbose due to the desire to have the proper "branding". However, I can see it being a bit obtuse, particularly in logs.

Would the community prefer something more succinct here, such as "qaic"?

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#3 lore raw

Re: [RFC PATCH 1/8] qaic: Add skeleton driver

Quoted 8 lines · Jeffrey Hugo
Quoted 1 line · Jeffrey Hugo

+#define QAIC_NAME "Qualcomm Cloud AI 100"

<snip>

drivers/misc/qaic/qaic_drv.c line 33

Quoted 2 lines
drivers/misc/qaic/qaic_drv.c new file +2 -0
+static struct pci_driver qaic_pci_driver = {
+	.name = QAIC_NAME,

A question about the community's preference here.

Quoted 4 lines

Our driver name is very verbose due to the desire to have the proper "branding". However, I can see it being a bit obtuse, particularly in logs.

Would the community prefer something more succinct here, such as "qaic"?

Make it the same a KBUILD_MODNAME and then no one can complain :)

thanks,

greg k-h

#4 lore raw

RFC PATCH 2/8 qaic: Add and init a basic mhi controller

In reply to Jeffrey Hugo

An mhi controller is the "driver" for a specific mhi device with the mhi bus framework. Add a basic controller for the AIC100 device, and supporting code to init the controller. This will enable the PCIE device, init the mhi hardware, bring it to ready state, and use BHI to load the SBL image.

---
 drivers/misc/qaic/Makefile         |   4 +-
 drivers/misc/qaic/mhi_controller.c | 495 +++++++++++++++++++++++++++++++++++++
 drivers/misc/qaic/mhi_controller.h |  14 ++
 drivers/misc/qaic/qaic.h           |  18 ++
 drivers/misc/qaic/qaic_drv.c       | 110 +++++++++
 5 files changed, 639 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/qaic/mhi_controller.c
 create mode 100644 drivers/misc/qaic/mhi_controller.h
 create mode 100644 drivers/misc/qaic/qaic.h
drivers/misc/qaic/Makefile +2 -2
index 42149ac..b5fd819 100644
@@ -3,8 +3,8 @@
 # Makefile for Qualcomm Cloud AI 100 accelerators driver
 #
 
-
 obj-$(CONFIG_QAIC)	:= qaic.o
 
 qaic-y := \
-	qaic_drv.o
+	qaic_drv.o \
+	mhi_controller.o
drivers/misc/qaic/mhi_controller.c new file +495 -0

495 lines not on this page · raw message

drivers/misc/qaic/mhi_controller.h new file +14 -0
new file mode 100644
index 0000000..c81725e
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2019-2020, The Linux Foundation. All rights reserved.
+ */
+
+#ifndef MHICONTROLLERQAIC_H_
+#define MHICONTROLLERQAIC_H_
+
+struct mhi_controller *qaic_mhi_register_controller(struct pci_dev *pci_dev,
+						    void *mhi_bar,
+						    int mhi_irq);
+
+void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up);
+#endif /* MHICONTROLLERQAIC_H_ */
drivers/misc/qaic/qaic.h new file +18 -0
new file mode 100644
index 0000000..379aa82
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2019-2020, The Linux Foundation. All rights reserved.
+ */
+
+#ifndef QAICINTERNAL_H_
+#define QAICINTERNAL_H_
+
+#include <linux/mhi.h>
+#include <linux/pci.h>
+
+struct qaic_device {
+	struct pci_dev		*pdev;
+	int			bars;
+	void __iomem		*bar_0;
+	struct mhi_controller	*mhi_cntl;
+};
+#endif /* QAICINTERNAL_H_ */
drivers/misc/qaic/qaic_drv.c +110 -0

146 lines not on this page · raw message

Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#5 lore raw

Re: [RFC PATCH 2/8] qaic: Add and init a basic mhi controller

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Jeffrey,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on char-misc/char-misc-testing]
[also build test WARNING on arm-soc/for-next linus/master v5.7-rc5 next-20200514] [cannot apply to linux/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Jeffrey-Hugo/Qualcomm-Cloud-AI-100-driver/20200514-221359
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc.git 84c1e51d7df85332705ffebc40ef135205492036
config: parisc-allyesconfig (attached as .config)
compiler: hppa-linux-gcc (GCC) 9.3.0

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day GCC_VERSION=9.3.0 make.cross ARCH=parisc

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>, old ones prefixed by <<):

In file included from include/linux/kernel.h:7,
from arch/parisc/include/asm/bug.h:5,
from include/linux/bug.h:5,
from include/linux/thread_info.h:12,
from include/asm-generic/current.h:5,
from ./arch/parisc/include/generated/asm/current.h:1,
from include/linux/sched.h:12,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/mhi.h:9,
from drivers/misc/qaic/mhi_controller.c:6:
drivers/misc/qaic/mhi_controller.c: In function 'qaic_mhi_register_controller':

>> include/linux/limits.h:22:18: warning: conversion from 'long long unsigned int' to 'dma_addr_t' {aka 'unsigned int'} changes value from '18446744073709551614' to '4294967294' [-Woverflow]
22 | #define U64_MAX  ((u64)~0ULL)
|                  ^
>> drivers/misc/qaic/mhi_controller.c:450:24: note: in expansion of macro 'U64_MAX'

450 | mhi_cntl->iova_stop = U64_MAX - 1;
| ^~~~~~~
drivers/misc/qaic/mhi_controller.c:455:10: error: 'struct mhi_controller' has no member named 'link_status' 455 | mhi_cntl->link_status = mhi_link_status;
| ^~

vim +/U64_MAX +450 drivers/misc/qaic/mhi_controller.c

428
429 struct mhi_controller *qaic_mhi_register_controller(struct pci_dev *pci_dev,
430 void *mhi_bar,
431 int mhi_irq)
432 {
433 struct mhi_controller *mhi_cntl;
434 int ret;
435
436 pci_dbg(pci_dev, "%s\n", __func__);
437
438 mhi_cntl = kzalloc(sizeof(*mhi_cntl), GFP_KERNEL);
439 if (!mhi_cntl)
440 return ERR_PTR(-ENOMEM);
441
442 mhi_cntl->cntrl_dev = &pci_dev->dev;
443
444 /*
445 * Covers the entire possible physical ram region. Remote side is
446 * going to calculate a size of this range, so subtract 1 to prevent
447 * rollover.
448 */
449 mhi_cntl->iova_start = 0;

 > 450		mhi_cntl->iova_stop = U64_MAX - 1;

451
452 mhi_cntl->status_cb = mhi_status_cb;
453 mhi_cntl->runtime_get = mhi_runtime_get;
454 mhi_cntl->runtime_put = mhi_runtime_put;
455 mhi_cntl->link_status = mhi_link_status;
456 mhi_cntl->regs = mhi_bar;
457 mhi_cntl->nr_irqs = 1;
458 mhi_cntl->irq = kmalloc(sizeof(*mhi_cntl->irq), GFP_KERNEL);
459
460 if (!mhi_cntl->irq)
461 return ERR_PTR(-ENOMEM);
462
463 mhi_cntl->irq[0] = mhi_irq;
464
465 mhi_cntl->fw_image = "qcom/aic100/sbl.bin";
466
467 /* use latest configured timeout */
468 aic100_config.timeout_ms = mhi_timeout;
469 ret = mhi_register_controller(mhi_cntl, &aic100_config);
470 if (ret) {
471 pci_err(pci_dev, "register_mhi_controller failed %d\n", ret);
472 kfree(mhi_cntl->irq);
473 kfree(mhi_cntl);
474 return ERR_PTR(ret);
475 }
476
477 ret = mhi_async_power_up(mhi_cntl);
478 if (ret) {
479 pci_err(pci_dev, "mhi_async_power_up failed %d\n", ret);
480 mhi_unregister_controller(mhi_cntl);
481 kfree(mhi_cntl->irq);
482 kfree(mhi_cntl);
483 return ERR_PTR(ret);
484 }
485
486 return mhi_cntl;
487 }
488

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#6 lore raw

RFC PATCH 3/8 qaic: Create char dev

In reply to Jeffrey Hugo

Now that we can fully boot the device, we should start making it usable. The primary interface to the device will be via a char dev. Add the necessary framework to detect when the device is fully booted, and create the char dev at that point. The device is only usable when it is fully booted. The char dev does nothing useful yet, but we can easily build on this to provide functionality.

---
 drivers/misc/qaic/qaic.h     |  19 +++
 drivers/misc/qaic/qaic_drv.c | 303 ++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 321 insertions(+), 1 deletion(-)
drivers/misc/qaic/qaic.h +19 -0
index 379aa82..58ca167 100644
@@ -6,13 +6,32 @@
 #ifndef QAICINTERNAL_H_
 #define QAICINTERNAL_H_
 
+#include <linux/cdev.h>
+#include <linux/kref.h>
 #include <linux/mhi.h>
+#include <linux/mutex.h>
 #include <linux/pci.h>
+#include <linux/srcu.h>
+
+struct qaic_user {
+	pid_t			handle;
+	struct qaic_device	*qdev;
+	struct list_head	node;
+	struct srcu_struct	qdev_lock;
+	struct kref		ref_count;
+};
 
 struct qaic_device {
 	struct pci_dev		*pdev;
 	int			bars;
 	void __iomem		*bar_0;
 	struct mhi_controller	*mhi_cntl;
+	struct mhi_device	*cntl_ch;
+	struct cdev		*cdev;
+	struct device		*dev;
+	bool			in_reset;
+	struct srcu_struct	dev_lock;
+	struct list_head	users;
+	struct mutex		users_mutex;
 };
 #endif /* QAICINTERNAL_H_ */
drivers/misc/qaic/qaic_drv.c +302 -1

365 lines not on this page · raw message

Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#7 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 14 lines · Jeffrey Hugo

/* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */

drivers/misc/qaic/qaic_drv.c +6 -0
+#include <linux/cdev.h>
+#include <linux/idr.h>
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/mhi.h>
 #include <linux/module.h>
 #include <linux/msi.h>
+#include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/pci_ids.h>

drivers/misc/qaic/qaic_drv.c line 22

Quoted 4 lines
drivers/misc/qaic/qaic_drv.c +1 -0
@@ -13,9 +19,242 @@
 #define PCI_DEV_AIC100			0xa100
 
 #define QAIC_NAME			"Qualcomm Cloud AI 100"
+#define QAIC_MAX_MINORS			256

Why have a max?

Why not just use a misc device so you make the logic a lot simple, no class or chardev logic to mess with at all.

thanks,

greg k-h

#8 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Wow, thank you for the near immediate response. I'm am quite impressed.

Quoted 17 lines · Greg KH
Quoted 16 lines · Jeffrey Hugo

/* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */

drivers/misc/qaic/qaic_drv.c +6 -0
+#include <linux/cdev.h>
+#include <linux/idr.h>
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/mhi.h>
  #include <linux/module.h>
  #include <linux/msi.h>
+#include <linux/mutex.h>
  #include <linux/pci.h>
  #include <linux/pci_ids.h>
@@ -13,9 +19,242 @@
  #define PCI_DEV_AIC100			0xa100
 
Quoted 7 lines

drivers/misc/qaic/qaic_drv.c line 22

Quoted 2 lines
drivers/misc/qaic/qaic_drv.c +1 -0
  #define QAIC_NAME			"Qualcomm Cloud AI 100"
+#define QAIC_MAX_MINORS			256

Why have a max?

Why not just use a misc device so you make the logic a lot simple, no class or chardev logic to mess with at all.

It was our understanding that the preference is not to add new misc devices. As I go and try to find a supporting reference for that, I cannot find one, so I'm not entirely sure where that idea came from.

In addition, we see that the Habana Labs driver also uses chardev, and has chosen the same max. We assumed that since their driver is already accepted, using the same mechanisms where applicable would be the preferred approach.

Specific to the max, 256 was chosen as being a factor larger than the largest system we have, therefore we figured it wouldn't be hit for a long while even as we try to have a look at what might happen down the road. Looking at the Habana code, it looks like they have the same value for much of the same reasons, although their usecases may vary from ours somewhat.

At this time, I don't think we have a strong requirement for a chardev, so we could investigate a switch over to a misc dev if you would prefer that over following the Habana Labs precedent. All I ask is a confirmation that is the approach you would like to see going forward after reviewing the above.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#9 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 22 lines · Jeffrey Hugo

Wow, thank you for the near immediate response. I'm am quite impressed.

Quoted 19 lines · Greg KH
Quoted 12 lines · Jeffrey Hugo

/* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */

drivers/misc/qaic/qaic_drv.c +6 -0
+#include <linux/cdev.h>
+#include <linux/idr.h>
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/mhi.h>
  #include <linux/module.h>
  #include <linux/msi.h>
+#include <linux/mutex.h>
  #include <linux/pci.h>
  #include <linux/pci_ids.h>

drivers/misc/qaic/qaic_drv.c line 21

Quoted 3 lines
drivers/misc/qaic/qaic_drv.c +1 -0
@@ -13,9 +19,242 @@
  #define PCI_DEV_AIC100			0xa100
  #define QAIC_NAME			"Qualcomm Cloud AI 100"
+#define QAIC_MAX_MINORS			256

Why have a max?

Quoted 4 lines
Quoted 2 lines

Why not just use a misc device so you make the logic a lot simple, no class or chardev logic to mess with at all.

It was our understanding that the preference is not to add new misc devices.

Huh, who said that? Not the char/misc maintainer (i.e. me) :)

Quoted 7 lines

As I go and try to find a supporting reference for that, I cannot find one, so I'm not entirely sure where that idea came from.

In addition, we see that the Habana Labs driver also uses chardev, and has chosen the same max. We assumed that since their driver is already accepted, using the same mechanisms where applicable would be the preferred approach.

They had good reasons why not to use a chardev and convinced me of it. If you can't come up with them, then stick with a misc for now please.

Quoted 5 lines

Specific to the max, 256 was chosen as being a factor larger than the largest system we have, therefore we figured it wouldn't be hit for a long while even as we try to have a look at what might happen down the road. Looking at the Habana code, it looks like they have the same value for much of the same reasons, although their usecases may vary from ours somewhat.

Max numbers for no good reason are not a good thing to have.

Quoted 5 lines

At this time, I don't think we have a strong requirement for a chardev, so we could investigate a switch over to a misc dev if you would prefer that over following the Habana Labs precedent. All I ask is a confirmation that is the approach you would like to see going forward after reviewing the above.

Please use misc.

That is, if you even need a character device node :)

thanks,

greg k-h

#10 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 33 lines · Greg KH
Quoted 22 lines · Jeffrey Hugo

Wow, thank you for the near immediate response. I'm am quite impressed.

Quoted 19 lines · Greg KH
Quoted 12 lines · Jeffrey Hugo
   /* Copyright (c) 2019-2020, The Linux Foundation. All rights reserved. */
drivers/misc/qaic/qaic_drv.c +6 -0
+#include <linux/cdev.h>
+#include <linux/idr.h>
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/mhi.h>
   #include <linux/module.h>
   #include <linux/msi.h>
+#include <linux/mutex.h>
   #include <linux/pci.h>
   #include <linux/pci_ids.h>

drivers/misc/qaic/qaic_drv.c line 21

Quoted 3 lines
drivers/misc/qaic/qaic_drv.c +1 -0
@@ -13,9 +19,242 @@
   #define PCI_DEV_AIC100			0xa100
   #define QAIC_NAME			"Qualcomm Cloud AI 100"
+#define QAIC_MAX_MINORS			256

Why have a max?

Quoted 4 lines
Quoted 2 lines

Why not just use a misc device so you make the logic a lot simple, no class or chardev logic to mess with at all.

It was our understanding that the preference is not to add new misc devices.

Huh, who said that? Not the char/misc maintainer (i.e. me) :)

Quoted 3 lines

As I go and try to find a supporting reference for that, I cannot find one, so I'm not entirely sure where that idea came from.

Quoted 7 lines
Quoted 4 lines

In addition, we see that the Habana Labs driver also uses chardev, and has chosen the same max. We assumed that since their driver is already accepted, using the same mechanisms where applicable would be the preferred approach.

They had good reasons why not to use a chardev and convinced me of it. If you can't come up with them, then stick with a misc for now please.

Interesting. I didn't see any discussion on this.

Quoted 8 lines
Quoted 5 lines

Specific to the max, 256 was chosen as being a factor larger than the largest system we have, therefore we figured it wouldn't be hit for a long while even as we try to have a look at what might happen down the road. Looking at the Habana code, it looks like they have the same value for much of the same reasons, although their usecases may vary from ours somewhat.

Max numbers for no good reason are not a good thing to have.

Quoted 7 lines
Quoted 5 lines

At this time, I don't think we have a strong requirement for a chardev, so we could investigate a switch over to a misc dev if you would prefer that over following the Habana Labs precedent. All I ask is a confirmation that is the approach you would like to see going forward after reviewing the above.

Please use misc.

Ok, will investigate.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#11 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 5 lines · Jeffrey Hugo
Quoted 1 line · Greg KH

Please use misc.

Ok, will investigate.

Misc looks like it'll work, and I'm expecting to have that in the next revision. However, a few of us looked at misc vs chardev, and didn't see much of a difference.

We were hoping you'd be kind enough to educate us on the considerations between the two, in-case we missed something that ends up being very relevant. We attempted to find relevant documentation within the kernel, and in Linux Device Drivers 3rd edition, but did not find any besides just reading the misc dev code. If we missed something, please point us to it.

By going with a misc device, we see two possible limitations -

1. You don't have your own major number, so userspace may have to do a bit more work to identify your device. However, given that the driver assigns the device name, one would think that the device name would be pretty unique. So, it seems like this would have a minor impact to udev rules.

2. There are a limited number of dynamic minor numbers for misc devs (64), so if you are expecting more devices than that, a misc dev is not appropiate. Also, these minors are shared with other misc dev users, so depending on the system configuration, you might have significantly less than 64 minors available for use.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#12 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 5 lines · Jeffrey Hugo

2. There are a limited number of dynamic minor numbers for misc devs (64), so if you are expecting more devices than that, a misc dev is not appropiate. Also, these minors are shared with other misc dev users, so depending on the system configuration, you might have significantly less than 64 minors available for use.

I'm pretty sure we can have more than 64 misc devices, that limitation should have been removed a while ago. Try it and see :)

greg k-h

#13 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 9 lines · Greg KH
Quoted 5 lines · Jeffrey Hugo

2. There are a limited number of dynamic minor numbers for misc devs (64), so if you are expecting more devices than that, a misc dev is not appropiate. Also, these minors are shared with other misc dev users, so depending on the system configuration, you might have significantly less than 64 minors available for use.

I'm pretty sure we can have more than 64 misc devices, that limitation should have been removed a while ago. Try it and see :)

In total, there can be more tha 64 misc devices. However my previous comment was specific to dynamic minors (ie devices which do not have an assigned minor). The limit on dynamic minors still apears to be 64. Looking at the code -

DYNAMIC_MINORS is still 64
https://elixir.bootlin.com/linux/v5.7-rc5/source/drivers/char/misc.c#L63

I see the same in -next

DYNAMIC_MINORS is used to size a bitmap - one bit for each dynamic minor misc device that exists at one particular point in time. After all 64 bits are consumed by misc_register() by clients requesting a dynamic minor, no more dynamic minor misc devices can be registered until some are unregistered.

What am I missing?

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#14 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 21 lines · Jeffrey Hugo
Quoted 9 lines · Greg KH
Quoted 5 lines · Jeffrey Hugo

2. There are a limited number of dynamic minor numbers for misc devs (64), so if you are expecting more devices than that, a misc dev is not appropiate. Also, these minors are shared with other misc dev users, so depending on the system configuration, you might have significantly less than 64 minors available for use.

I'm pretty sure we can have more than 64 misc devices, that limitation should have been removed a while ago. Try it and see :)

In total, there can be more tha 64 misc devices. However my previous comment was specific to dynamic minors (ie devices which do not have an assigned minor). The limit on dynamic minors still apears to be 64. Looking at the code -

DYNAMIC_MINORS is still 64
https://elixir.bootlin.com/linux/v5.7-rc5/source/drivers/char/misc.c#L63

I see the same in -next

Quoted 7 lines

DYNAMIC_MINORS is used to size a bitmap - one bit for each dynamic minor misc device that exists at one particular point in time. After all 64 bits are consumed by misc_register() by clients requesting a dynamic minor, no more dynamic minor misc devices can be registered until some are unregistered.

What am I missing?

Oops, nothing, my fault. We fixed up the allocation of more dynamic majors for chardev in 2017 and for some reason I thought we also increased the number of misc dynamic minors at the same time, but that was incorrect.

I'll gladly take patches that bump up the number of misc minors if needed.

But to get back to the main issue here, you are only going to have 1 or maybe 2 of these devices in a system at a time, right? So "burning" a whole major number for that feels like a waste.

thanks,

greg k-h

#15 lore raw

Re: [RFC PATCH 3/8] qaic: Create char dev

Quoted 28 lines · Greg KH
Quoted 27 lines · Jeffrey Hugo
Quoted 9 lines · Greg KH
Quoted 5 lines · Jeffrey Hugo

2. There are a limited number of dynamic minor numbers for misc devs (64), so if you are expecting more devices than that, a misc dev is not appropiate. Also, these minors are shared with other misc dev users, so depending on the system configuration, you might have significantly less than 64 minors available for use.

I'm pretty sure we can have more than 64 misc devices, that limitation should have been removed a while ago. Try it and see :)

In total, there can be more tha 64 misc devices. However my previous comment was specific to dynamic minors (ie devices which do not have an assigned minor). The limit on dynamic minors still apears to be 64. Looking at the code -

DYNAMIC_MINORS is still 64
https://elixir.bootlin.com/linux/v5.7-rc5/source/drivers/char/misc.c#L63

I see the same in -next

DYNAMIC_MINORS is used to size a bitmap - one bit for each dynamic minor misc device that exists at one particular point in time. After all 64 bits are consumed by misc_register() by clients requesting a dynamic minor, no more dynamic minor misc devices can be registered until some are unregistered.

Quoted 6 lines
Quoted 1 line

What am I missing?

Oops, nothing, my fault. We fixed up the allocation of more dynamic majors for chardev in 2017 and for some reason I thought we also increased the number of misc dynamic minors at the same time, but that was incorrect.

No problem.

Quoted 2 lines

I'll gladly take patches that bump up the number of misc minors if needed.

I don't think its needed at this time, but I will keep that in mind.

Quoted 3 lines

But to get back to the main issue here, you are only going to have 1 or maybe 2 of these devices in a system at a time, right? So "burning" a whole major number for that feels like a waste.

Depends on what the customer wants to do. We support a number of systems, but one in particular has the capability of 6-12 devices.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#16 lore raw

RFC PATCH 4/8 qaic: Implement control path

In reply to Jeffrey Hugo

Once a qaic device is fully booted, the control path is available for use. The control path allows the host to use a MHI channel to send requests to the device. The canonical usecase for the control path is allowing userspace to configure the device for a workload. This is accomplished via the manage ioctl.

---
 drivers/misc/qaic/Makefile       |    3 +-
 drivers/misc/qaic/qaic.h         |   61 +++
 drivers/misc/qaic/qaic_control.c | 1015 ++++++++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_drv.c     |  159 +++++-
 include/uapi/misc/qaic.h         |  117 +++++
 5 files changed, 1343 insertions(+), 12 deletions(-)
 create mode 100644 drivers/misc/qaic/qaic_control.c
 create mode 100644 include/uapi/misc/qaic.h
drivers/misc/qaic/Makefile +2 -1
index b5fd819..7a5513b 100644
@@ -7,4 +7,5 @@ obj-$(CONFIG_QAIC)	:= qaic.o
 
 qaic-y := \
 	qaic_drv.o \
-	mhi_controller.o
+	mhi_controller.o \
+	qaic_control.o
drivers/misc/qaic/qaic.h +61 -0

89 lines not on this page · raw message

drivers/misc/qaic/qaic_control.c new file +1015 -0

1,015 lines not on this page · raw message

drivers/misc/qaic/qaic_drv.c +148 -11

282 lines not on this page · raw message

include/uapi/misc/qaic.h new file +117 -0

117 lines not on this page · raw message

Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#17 lore raw

RFC PATCH 5/8 qaic: Implement data path

In reply to Jeffrey Hugo

Once a user has configured the device via the control path for their workload, the user may use the data path to execute the workload. Using the data path involves several steps.

First, the user must use the memory ioctl to allocate one or more buffers for use by the workload. These buffers hold input data, and results. The memory ioctl can also be used to free buffers once no longer needed.

Next, the user must mmap() the buffers, to gain access to them.

To submit buffers to the device, the user uses the execute ioctl.

Finally, the user may use the wait for execute ioctl to determine when the device has completed its handling of submitted buffers (ie consumed input data, or produced output).

---
 drivers/misc/qaic/Makefile    |   3 +-
 drivers/misc/qaic/qaic.h      |  11 +
 drivers/misc/qaic/qaic_data.c | 952 ++++++++++++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_drv.c  | 116 +++--
 include/uapi/misc/qaic.h      | 129 ++++++
 5 files changed, 1175 insertions(+), 36 deletions(-)
 create mode 100644 drivers/misc/qaic/qaic_data.c
drivers/misc/qaic/Makefile +2 -1
index 7a5513b..d17ee0a 100644
@@ -8,4 +8,5 @@ obj-$(CONFIG_QAIC)	:= qaic.o
 qaic-y := \
 	qaic_drv.o \
 	mhi_controller.o \
-	qaic_control.o
+	qaic_control.o \
+	qaic_data.o
drivers/misc/qaic/qaic.h +11 -0
index c35a4e1..5f774d4 100644
@@ -8,6 +8,7 @@
 
 #include <linux/cdev.h>
 #include <linux/idr.h>
+#include <linux/interrupt.h>
 #include <linux/kref.h>
 #include <linux/mhi.h>
 #include <linux/mutex.h>
@@ -80,6 +81,14 @@ int get_cntl_version(struct qaic_device *qdev, struct qaic_user *usr,
 		     u16 *major, u16 *minor);
 int qaic_manage_ioctl(struct qaic_device *qdev, struct qaic_user *usr,
 		      unsigned long arg);
+int qaic_mem_ioctl(struct qaic_device *qdev, struct qaic_user *usr,
+		   unsigned long arg);
+int qaic_execute_ioctl(struct qaic_device *qdev, struct qaic_user *usr,
+		       unsigned long arg);
+int qaic_wait_exec_ioctl(struct qaic_device *qdev, struct qaic_user *usr,
+			 unsigned long arg);
+int qaic_data_mmap(struct qaic_device *qdev, struct qaic_user *usr,
+		   struct vm_area_struct *vma);
 
 void qaic_mhi_ul_xfer_cb(struct mhi_device *mhi_dev,
 			 struct mhi_result *mhi_result);
@@ -91,7 +100,9 @@ int qaic_control_open(struct qaic_device *qdev);
 void qaic_control_close(struct qaic_device *qdev);
 void qaic_release_usr(struct qaic_device *qdev, struct qaic_user *usr);
 
+irqreturn_t dbc_irq_handler(int irq, void *data);
 int disable_dbc(struct qaic_device *qdev, u32 dbc_id, struct qaic_user *usr);
+void wakeup_dbc(struct qaic_device *qdev, u32 dbc_id);
 void release_dbc(struct qaic_device *qdev, u32 dbc_id);
 
 void wake_all_cntl(struct qaic_device *qdev);
drivers/misc/qaic/qaic_data.c new file +952 -0

952 lines not on this page · raw message

drivers/misc/qaic/qaic_drv.c +81 -35

188 lines not on this page · raw message

include/uapi/misc/qaic.h +129 -0

146 lines not on this page · raw message

Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#18 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

thanks,

greg k-h

#19 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 6 lines · Greg KH

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

Huh. We had thought the botching ioctls document advised having a version, but as I double check that document, it infact does not.

Will remove.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#20 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 8 lines · Jeffrey Hugo
Quoted 6 lines · Greg KH

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

Quoted 4 lines

Huh. We had thought the botching ioctls document advised having a version, but as I double check that document, it infact does not.

Will remove.

Thanks, you can also remove the "reserved" variables as well as those will not be needed either.

greg k-h

#21 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 9 lines · Greg KH
Quoted 8 lines · Jeffrey Hugo
Quoted 6 lines · Greg KH

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

Quoted 7 lines
Quoted 4 lines

Huh. We had thought the botching ioctls document advised having a version, but as I double check that document, it infact does not.

Will remove.

Thanks, you can also remove the "reserved" variables as well as those will not be needed either.

Are you sure?

Documentation/process/botching-up-ioctls.rst
Starting at Line 38:

"Pad the entire struct to a multiple of 64-bits if the structure contains 64-bit types - the structure size will otherwise differ on 32-bit versus 64-bit. Having a different structure size hurts when passing arrays of structures to the kernel, or if the kernel checks the structure size, which e.g. the drm core does."

The "reserved" variables seem to be in line with that.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#22 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 23 lines · Jeffrey Hugo
Quoted 9 lines · Greg KH
Quoted 8 lines · Jeffrey Hugo
Quoted 6 lines · Greg KH

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

Quoted 7 lines
Quoted 4 lines

Huh. We had thought the botching ioctls document advised having a version, but as I double check that document, it infact does not.

Will remove.

Thanks, you can also remove the "reserved" variables as well as those will not be needed either.

Are you sure?

Documentation/process/botching-up-ioctls.rst
Starting at Line 38:

Quoted 7 lines

"Pad the entire struct to a multiple of 64-bits if the structure contains 64-bit types - the structure size will otherwise differ on 32-bit versus 64-bit. Having a different structure size hurts when passing arrays of structures to the kernel, or if the kernel checks the structure size, which e.g. the drm core does."

The "reserved" variables seem to be in line with that.

Padding is fine to use, but don't use that as a "I'm reserving this to use it for later" type of thing which is how I read the structure definitions. I might be totally wrong, but you should be explicit here.

thanks,

greg k-h

#23 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 30 lines · Greg KH
Quoted 29 lines · Jeffrey Hugo
Quoted 9 lines · Greg KH
Quoted 8 lines · Jeffrey Hugo
Quoted 6 lines · Greg KH

include/uapi/misc/qaic.h struct qaic_manage_msg { · line 124

Quoted 2 lines · Jeffrey Hugo
include/uapi/misc/qaic.h +2 -0
+struct qaic_execute {
+	__u16		ver;    /* struct version, must be 1 */

No need for structures to be versioned. If you change something, then add a new ioctl if you really needed it.

Quoted 7 lines
Quoted 4 lines

Huh. We had thought the botching ioctls document advised having a version, but as I double check that document, it infact does not.

Will remove.

Thanks, you can also remove the "reserved" variables as well as those will not be needed either.

Are you sure?

Documentation/process/botching-up-ioctls.rst
Starting at Line 38:

"Pad the entire struct to a multiple of 64-bits if the structure contains 64-bit types - the structure size will otherwise differ on 32-bit versus 64-bit. Having a different structure size hurts when passing arrays of structures to the kernel, or if the kernel checks the structure size, which e.g. the drm core does."

Quoted 5 lines
Quoted 1 line

The "reserved" variables seem to be in line with that.

Padding is fine to use, but don't use that as a "I'm reserving this to use it for later" type of thing which is how I read the structure definitions. I might be totally wrong, but you should be explicit here.

Ok, I think I see your point. I'll change them to be more explicit as padding.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#24 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

In reply to Jeffrey Hugo
Quoted 2 lines · Jeffrey Hugo

+struct dbc_req { /* everything must be little endian encoded */

Instead of the comment, I suppose you want to use __le16 and __le32 types and let sparse check that you got it right.

Quoted 8 lines
drivers/misc/qaic/qaic_data.c new file +8 -0
+       u16     req_id;
+       u8      seq_id;
+       u8      cmd;
+       u32     resv;
+       u64     src_addr;
+       u64     dest_addr;
+       u32     len;
+       u32     resv2;

drivers/misc/qaic/qaic_data.c line 64

Quoted 10 lines
drivers/misc/qaic/qaic_data.c new file +10 -0
+       u64     db_addr; /* doorbell address */
+       u8      db_len; /* not a raw value, special encoding */
+       u8      resv3;
+       u16     resv4;
+       u32     db_data;
+       u32     sem_cmd0;
+       u32     sem_cmd1;
+       u32     sem_cmd2;
+       u32     sem_cmd3;
+} __packed;

All members are naturally aligned, so better drop the __packed annotation get better code, unless the structure itself is at an unaligned offset in memory.

drivers/misc/qaic/qaic_data.c line 69

Quoted 4 lines
drivers/misc/qaic/qaic_data.c new file +4 -0
+struct dbc_rsp { /* everything must be little endian encoded */
+       u16     req_id;
+       u16     status;
+} __packed;

Same here.

drivers/misc/qaic/qaic_data.c line 622

Quoted 4 lines
drivers/misc/qaic/qaic_data.c new file +4 -0
+       init_completion(&mem->xfer_done);
+       list_add_tail(&mem->list, &dbc->xfer_list);
+       tail = (tail + mem->nents) % dbc->nelem;
+       __raw_writel(cpu_to_le32(tail), dbc->dbc_base + REQTP_OFF);

What is this __raw accessor for? This generally results in non-portable code that should be replaced with writel() or something specific to the bus on the architecture you deal with.

drivers/misc/qaic/qaic_data.c line 703

Quoted 5 lines
drivers/misc/qaic/qaic_data.c new file +5 -0
+       spin_lock_irqsave(&qdev->dbc[exec->dbc_id].xfer_lock, flags);
+       req_id = qdev->dbc[exec->dbc_id].next_req_id++;
+       queued = mem->queued;
+       mem->queued = true;
+       spin_unlock_irqrestore(&qdev->dbc[exec->dbc_id].xfer_lock, flags);

No need for 'irqsave' locks when you know that interrupts are enabled.

drivers/misc/qaic/qaic_data.c line 770

Quoted 2 lines
drivers/misc/qaic/qaic_data.c new file +2 -0
+       head = le32_to_cpu(__raw_readl(dbc->dbc_base + RSPHP_OFF));
+       tail = le32_to_cpu(__raw_readl(dbc->dbc_base + RSPTP_OFF));

More __raw accessors to replace.

drivers/misc/qaic/qaic_drv.c static long qaic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) · line 185

Quoted 5 lines
drivers/misc/qaic/qaic_drv.c +5 -0
+       case QAIC_IOCTL_MEM_NR:
+               if (_IOC_DIR(cmd) != (_IOC_READ | _IOC_WRITE) ||
+                   _IOC_SIZE(cmd) != sizeof(struct qaic_mem_req)) {
+                       ret = -EINVAL;
+                       break;

This looks like a very verbose way to check 'cmd' against a known constant. Why not use 'switch (cmd)' like all other drivers?

Arnd

#25 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Thanks for the review.

Quoted 6 lines · Arnd Bergmann
Quoted 2 lines · Jeffrey Hugo

+struct dbc_req { /* everything must be little endian encoded */

Instead of the comment, I suppose you want to use __le16 and __le32 types and let sparse check that you got it right.

Ah yes, I was curious if those should be applied here. Their use seems inconsistent. I will do that.

Quoted 20 lines
Quoted 9 lines
drivers/misc/qaic/qaic_data.c new file +8 -0
+       u16     req_id;
+       u8      seq_id;
+       u8      cmd;
+       u32     resv;
+       u64     src_addr;
+       u64     dest_addr;
+       u32     len;
+       u32     resv2;

drivers/misc/qaic/qaic_data.c line 64

Quoted 10 lines
drivers/misc/qaic/qaic_data.c new file +10 -0
+       u64     db_addr; /* doorbell address */
+       u8      db_len; /* not a raw value, special encoding */
+       u8      resv3;
+       u16     resv4;
+       u32     db_data;
+       u32     sem_cmd0;
+       u32     sem_cmd1;
+       u32     sem_cmd2;
+       u32     sem_cmd3;
+} __packed;
Quoted 3 lines

All members are naturally aligned, so better drop the __packed annotation get better code, unless the structure itself is at an unaligned offset in memory.

I'm going to have to disagree. While most "sane" compilers would not add extra padding, I've debugged enough issues in the past when sending/receiving data with foreign environments to never trust anything that isn't "packed".

Unless I missed something in the C spec that requires naturally aligned structures to have an identical layout in memory, I'll take safety and functional correctness over performance.

Quoted 8 lines

drivers/misc/qaic/qaic_data.c line 69

Quoted 5 lines
drivers/misc/qaic/qaic_data.c new file +4 -0
+struct dbc_rsp { /* everything must be little endian encoded */
+       u16     req_id;
+       u16     status;
+} __packed;

Same here.

Quoted 8 lines

drivers/misc/qaic/qaic_data.c line 622

Quoted 4 lines
drivers/misc/qaic/qaic_data.c new file +4 -0
+       init_completion(&mem->xfer_done);
+       list_add_tail(&mem->list, &dbc->xfer_list);
+       tail = (tail + mem->nents) % dbc->nelem;
+       __raw_writel(cpu_to_le32(tail), dbc->dbc_base + REQTP_OFF);

What is this __raw accessor for? This generally results in non-portable code that should be replaced with writel() or something specific to the bus on the architecture you deal with.

The barrier(s) that comes with writel are unnecessary in this case. Since this is part of our critical path, we are sensitive to its performance.

What are the portability issues around the __raw variant?

Quoted 8 lines

drivers/misc/qaic/qaic_data.c line 703

Quoted 6 lines
drivers/misc/qaic/qaic_data.c new file +5 -0
+       spin_lock_irqsave(&qdev->dbc[exec->dbc_id].xfer_lock, flags);
+       req_id = qdev->dbc[exec->dbc_id].next_req_id++;
+       queued = mem->queued;
+       mem->queued = true;
+       spin_unlock_irqrestore(&qdev->dbc[exec->dbc_id].xfer_lock, flags);

No need for 'irqsave' locks when you know that interrupts are enabled.

Fair enough.

Quoted 5 lines

drivers/misc/qaic/qaic_data.c line 770

Quoted 3 lines
drivers/misc/qaic/qaic_data.c new file +2 -0
+       head = le32_to_cpu(__raw_readl(dbc->dbc_base + RSPHP_OFF));
+       tail = le32_to_cpu(__raw_readl(dbc->dbc_base + RSPTP_OFF));

More __raw accessors to replace.

Same answer as before.

Quoted 9 lines

drivers/misc/qaic/qaic_drv.c static long qaic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) · line 185

Quoted 6 lines
drivers/misc/qaic/qaic_drv.c +5 -0
+       case QAIC_IOCTL_MEM_NR:
+               if (_IOC_DIR(cmd) != (_IOC_READ | _IOC_WRITE) ||
+                   _IOC_SIZE(cmd) != sizeof(struct qaic_mem_req)) {
+                       ret = -EINVAL;
+                       break;

This looks like a very verbose way to check 'cmd' against a known constant. Why not use 'switch (cmd)' like all other drivers?

Huh. That actually does sound more elegant. Will do.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
maintainer
#26 lore raw

Re: [RFC PATCH 5/8] qaic: Implement data path

Quoted 23 lines · Jeffrey Hugo
Quoted 19 lines
Quoted 8 lines
drivers/misc/qaic/qaic_data.c new file +8 -0
+       u16     req_id;
+       u8      seq_id;
+       u8      cmd;
+       u32     resv;
+       u64     src_addr;
+       u64     dest_addr;
+       u32     len;
+       u32     resv2;

drivers/misc/qaic/qaic_data.c line 64

Quoted 10 lines
drivers/misc/qaic/qaic_data.c new file +10 -0
+       u64     db_addr; /* doorbell address */
+       u8      db_len; /* not a raw value, special encoding */
+       u8      resv3;
+       u16     resv4;
+       u32     db_data;
+       u32     sem_cmd0;
+       u32     sem_cmd1;
+       u32     sem_cmd2;
+       u32     sem_cmd3;
+} __packed;
Quoted 3 lines

All members are naturally aligned, so better drop the __packed annotation get better code, unless the structure itself is at an unaligned offset in memory.

Quoted 8 lines

I'm going to have to disagree. While most "sane" compilers would not add extra padding, I've debugged enough issues in the past when sending/receiving data with foreign environments to never trust anything that isn't "packed".

Unless I missed something in the C spec that requires naturally aligned structures to have an identical layout in memory, I'll take safety and functional correctness over performance.

The C standard does not mandate the layout and individual ABIs can be different, but Linux does only runs on ABIs that have the correct layout for the structure above

The problem is that the compiler will split up word accesses into bytes on some architectures such as older ARM, to avoid unaligned loads and stores. It should be fine if you add both __packed and __aligned(8) to make the structure aligned again.

Quoted 9 lines
Quoted 8 lines

drivers/misc/qaic/qaic_data.c line 622

Quoted 4 lines
drivers/misc/qaic/qaic_data.c new file +4 -0
+       init_completion(&mem->xfer_done);
+       list_add_tail(&mem->list, &dbc->xfer_list);
+       tail = (tail + mem->nents) % dbc->nelem;
+       __raw_writel(cpu_to_le32(tail), dbc->dbc_base + REQTP_OFF);

What is this __raw accessor for? This generally results in non-portable code that should be replaced with writel() or something specific to the bus on the architecture you deal with.

Quoted 5 lines

The barrier(s) that comes with writel are unnecessary in this case. Since this is part of our critical path, we are sensitive to its performance.

What are the portability issues around the __raw variant?

The access may not be atomic on all architectures but get either split up or combined with adjacent accesses.

There is no guarantee about endianess: while most architectures treat __raw access as a simple load/store, some might have an implied byteswap.

__raw accesses might be completely reordered around other mmio accesses or spinlocks.

If you want to avoid the barrier, there is the writel_relaxed()/readl_relaxed() that skips most barriers, so they can be reordered against MSI interrupts and DMA accesses on architectures that allow (e.g. ARM or PowerPC) but not against each other.

If you do use them, I'd still expect a comment that explains why this instance is performance critical and how it is synchronized against DMA or MSI if necessary.

Arnd

#27 lore raw

RFC PATCH 6/8 qaic: Implement PCI link status error handlers

In reply to Jeffrey Hugo

Whenever the link goes down, the device loses all state. Therefore, we need to be aware of when the link goes down and comes back up so that our state is kept in sync with the device. Implement the PCI error handlers to be notified of links tate changes.

---
 drivers/misc/qaic/mhi_controller.c | 16 ++++++++++++++++
 drivers/misc/qaic/mhi_controller.h |  4 ++++
 drivers/misc/qaic/qaic_drv.c       | 29 +++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+)
drivers/misc/qaic/mhi_controller.c +16 -0
index ba4808c..18fc830 100644
@@ -493,3 +493,19 @@ void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up)
 	kfree(mhi_cntl->irq);
 	kfree(mhi_cntl);
 }
+
+void qaic_mhi_link_down(struct mhi_controller *mhi_cntl)
+{
+	mhi_power_down(mhi_cntl, false);
+}
+
+void qaic_mhi_link_up(struct mhi_controller *mhi_cntl)
+{
+	struct pci_dev *pci_dev = container_of(mhi_cntl->cntrl_dev,
+					       struct pci_dev, dev);
+	int ret;
+
+	ret = mhi_async_power_up(mhi_cntl);
+	if (ret)
+		pci_err(pci_dev, "mhi_async_power_up failed when link came back %d\n", ret);
+}
drivers/misc/qaic/mhi_controller.h +4 -0
index c81725e..5eca586 100644
@@ -11,4 +11,8 @@ struct mhi_controller *qaic_mhi_register_controller(struct pci_dev *pci_dev,
 						    int mhi_irq);
 
 void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up);
+
+void qaic_mhi_link_down(struct mhi_controller *mhi_cntl);
+void qaic_mhi_link_up(struct mhi_controller *mhi_cntl);
+
 #endif /* MHICONTROLLERQAIC_H_ */
drivers/misc/qaic/qaic_drv.c +29 -0
index 800769d..1c8eefd 100644
@@ -558,6 +558,28 @@ static void qaic_pci_remove(struct pci_dev *pdev)
 	kfree(qdev);
 }
 
+static pci_ers_result_t qaic_pci_error_detected(struct pci_dev *pdev,
+						enum pci_channel_state error)
+{
+	return PCI_ERS_RESULT_NEED_RESET;
+}
+
+static void qaic_pci_reset_prepare(struct pci_dev *pdev)
+{
+	struct qaic_device *qdev = pci_get_drvdata(pdev);
+
+	qaic_dev_reset_clean_local_state(qdev);
+	qaic_mhi_link_down(qdev->mhi_cntl);
+}
+
+static void qaic_pci_reset_done(struct pci_dev *pdev)
+{
+	struct qaic_device *qdev = pci_get_drvdata(pdev);
+
+	qdev->in_reset = false;
+	qaic_mhi_link_up(qdev->mhi_cntl);
+}
+
 static const struct mhi_device_id qaic_mhi_match_table[] = {
 	{ .chan = "QAIC_CONTROL", },
 	{},
@@ -581,11 +603,18 @@ static const struct pci_device_id ids[] = {
 };
 MODULE_DEVICE_TABLE(pci, ids);
 
+static const struct pci_error_handlers qaic_pci_err_handler = {
+	.error_detected = qaic_pci_error_detected,
+	.reset_prepare = qaic_pci_reset_prepare,
+	.reset_done = qaic_pci_reset_done,
+};
+
 static struct pci_driver qaic_pci_driver = {
 	.name = QAIC_NAME,
 	.id_table = ids,
 	.probe = qaic_pci_probe,
 	.remove = qaic_pci_remove,
+	.err_handler = &qaic_pci_err_handler,
 };
 
 static int __init qaic_init(void)
Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#28 lore raw

RFC PATCH 7/8 qaic: Implement MHI error status handler

In reply to Jeffrey Hugo

If the MHI bus has detected some kind of error with the device, it will invoke the registered status_cb(). We are intrested in both the fatal error and sys error events.

For a fatal error, the device has essentially gone away, and needs to be reinited from scratch. We lose all state. The MHI bus expects the controller to fully handle this scenario

For a sys error, the device can recover by going through the MHI reset flow, but we lose all state. The MHI bus is going to handle the MHI reset flow, but we need to still clean up our local state.

---
 drivers/misc/qaic/mhi_controller.c | 27 +++++++++++++++++++++++++++
 drivers/misc/qaic/mhi_controller.h |  2 ++
 drivers/misc/qaic/qaic.h           |  2 ++
 drivers/misc/qaic/qaic_drv.c       | 16 +++++++++++++++-
 4 files changed, 46 insertions(+), 1 deletion(-)
drivers/misc/qaic/mhi_controller.c +27 -0
index 18fc830..c1498d4 100644
@@ -7,6 +7,8 @@
 #include <linux/moduleparam.h>
 #include <linux/pci.h>
 
+#include "qaic.h"
+
 static unsigned int mhi_timeout = 20000; /* 20 sec default */
 module_param(mhi_timeout, uint, 0600);
 
@@ -424,6 +426,15 @@ static void mhi_runtime_put(struct mhi_controller *mhi_cntl)
 static void mhi_status_cb(struct mhi_controller *mhi_cntl,
 			  enum mhi_callback reason)
 {
+	struct qaic_device *qdev = (struct qaic_device *)pci_get_drvdata(
+					to_pci_dev(mhi_cntl->cntrl_dev));
+
+	/* this event occurs in atomic context */
+	if (reason == MHI_CB_FATAL_ERROR && !qdev->in_reset)
+		schedule_work(&qdev->reset_mhi_work);
+	/* this event occurs in non-atomic context */
+	if (reason == MHI_CB_SYS_ERROR && !qdev->in_reset)
+		qaic_dev_reset_clean_local_state(qdev);
 }
 
 struct mhi_controller *qaic_mhi_register_controller(struct pci_dev *pci_dev,
@@ -509,3 +520,19 @@ void qaic_mhi_link_up(struct mhi_controller *mhi_cntl)
 	if (ret)
 		pci_err(pci_dev, "mhi_async_power_up failed when link came back %d\n", ret);
 }
+
+void qaic_mhi_start_reset(struct mhi_controller *mhi_cntl)
+{
+	mhi_power_down(mhi_cntl, true);
+}
+
+void qaic_mhi_reset_done(struct mhi_controller *mhi_cntl)
+{
+	struct pci_dev *pci_dev = container_of(mhi_cntl->cntrl_dev,
+					       struct pci_dev, dev);
+	int ret;
+
+	ret = mhi_async_power_up(mhi_cntl);
+	if (ret)
+		pci_err(pci_dev, "mhi_async_power_up failed after reset %d\n", ret);
+}
drivers/misc/qaic/mhi_controller.h +2 -0
index 5eca586..1081f0a 100644
@@ -14,5 +14,7 @@ void qaic_mhi_free_controller(struct mhi_controller *mhi_cntl, bool link_up);
 
 void qaic_mhi_link_down(struct mhi_controller *mhi_cntl);
 void qaic_mhi_link_up(struct mhi_controller *mhi_cntl);
+void qaic_mhi_start_reset(struct mhi_controller *mhi_cntl);
+void qaic_mhi_reset_done(struct mhi_controller *mhi_cntl);
 
 #endif /* MHICONTROLLERQAIC_H_ */
drivers/misc/qaic/qaic.h +2 -0
index 5f774d4..bbb7512 100644
@@ -68,6 +68,7 @@ struct qaic_device {
 	struct cdev		*cdev;
 	struct device		*dev;
 	struct dma_bridge_chan	dbc[QAIC_NUM_DBC];
+	struct work_struct	reset_mhi_work;
 	struct workqueue_struct	*cntl_wq;
 	bool			in_reset;
 	struct srcu_struct	dev_lock;
@@ -106,4 +107,5 @@ void wakeup_dbc(struct qaic_device *qdev, u32 dbc_id);
 void release_dbc(struct qaic_device *qdev, u32 dbc_id);
 
 void wake_all_cntl(struct qaic_device *qdev);
+void qaic_dev_reset_clean_local_state(struct qaic_device *qdev);
 #endif /* QAICINTERNAL_H_ */
drivers/misc/qaic/qaic_drv.c +15 -1
index 1c8eefd..010cf88 100644
@@ -386,6 +386,18 @@ void qaic_dev_reset_clean_local_state(struct qaic_device *qdev)
 		release_dbc(qdev, i);
 }
 
+static void reset_mhi_work_func(struct work_struct *work)
+{
+	struct qaic_device *qdev;
+
+	qdev = container_of(work, struct qaic_device, reset_mhi_work);
+
+	qaic_dev_reset_clean_local_state(qdev);
+	qaic_mhi_start_reset(qdev->mhi_cntl);
+	qdev->in_reset = false;
+	qaic_mhi_reset_done(qdev->mhi_cntl);
+}
+
 static int qaic_pci_probe(struct pci_dev *pdev,
 			  const struct pci_device_id *id)
 {
@@ -411,6 +423,7 @@ static int qaic_pci_probe(struct pci_dev *pdev,
 	qdev->pdev = pdev;
 	mutex_init(&qdev->cntl_mutex);
 	INIT_LIST_HEAD(&qdev->cntl_xfer_list);
+	INIT_WORK(&qdev->reset_mhi_work, reset_mhi_work_func);
 	init_srcu_struct(&qdev->dev_lock);
 	INIT_LIST_HEAD(&qdev->users);
 	mutex_init(&qdev->users_mutex);
@@ -541,6 +554,7 @@ static void qaic_pci_remove(struct pci_dev *pdev)
 		return;
 
 	qaic_dev_reset_clean_local_state(qdev);
+	cancel_work_sync(&qdev->reset_mhi_work);
 	qaic_mhi_free_controller(qdev->mhi_cntl, link_up);
 	for (i = 0; i < QAIC_NUM_DBC; ++i) {
 		devm_free_irq(&pdev->dev, pci_irq_vector(pdev, i + 1),
@@ -682,4 +696,4 @@ module_exit(qaic_exit);
 MODULE_AUTHOR("Qualcomm Cloud AI 100 Accelerator Kernel Driver Team");
 MODULE_DESCRIPTION("Qualcomm Cloud 100 AI Accelerators Driver");
 MODULE_LICENSE("GPL v2");
-MODULE_VERSION("0.0.0"); /* MAJOR.MINOR.PATCH */
+MODULE_VERSION("3.0.1"); /* MAJOR.MINOR.PATCH */
Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#29 lore raw

RFC PATCH 8/8 MAINTAINERS: Add entry for QAIC driver

In reply to Jeffrey Hugo

Add MAINTAINERS entry for the Qualcomm Cloud AI 100 driver.

---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
index b816a45..7822d07 100644
@@ -13962,6 +13962,13 @@ F:	Documentation/devicetree/bindings/media/qcom,camss.txt
 F:	Documentation/media/v4l-drivers/qcom_camss.rst
 F:	drivers/media/platform/qcom/camss/
 
+QUALCOMM CLOUD AI 100 (QAIC) DRIVER
+M:	Jeffrey Hugo <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/misc/qaic/
+F:	include/uapi/misc/qaic.h
+
 QUALCOMM CORE POWER REDUCTION (CPR) AVS DRIVER
 M:	Niklas Cassel <[email protected]>
 L:	[email protected]
Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#30 lore raw
In reply to Jeffrey Hugo
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

Although this isn't a graphics driver, and Greg will likely merge anything to the kernel you throw at him, I do wonder how to validate the uapi from a security perspective. It's always interesting when someone wraps a DMA engine with user ioctls, and without enough information to decide if the DMA engine is secure against userspace misprogramming it.

Also if we don't understand the programming API on board the device, we can't tell if the "core" on the device are able to reprogram the device engines either.

Figuring this out is difficult at the best of times, it helps if there is access to the complete device documentation or user space side drivers in order to faciliate this.

The other area I mention is testing the uAPI, how do you envisage regression testing and long term sustainability of the uAPI?

Thanks,
Dave.

#31 lore raw
Quoted 24 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 6 lines

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

We have daily internal testing, although I don't expect you to take my word for that.

I would like to get one of these devices into the hands of Linaro, so that it can be put into KernelCI. Similar to other Qualcomm products. I'm trying to convince the powers that be to make this happen.

Regarding what the community could do on its own, everything but the Linux driver is considered proprietary - that includes the on device firmware and the entire userspace stack. This is a decision above my pay grade.

I've asked for authorization to develop and publish a simple userspace application that might enable the community to do such testing, but obtaining that authorization has been slow.

Quoted 6 lines

Although this isn't a graphics driver, and Greg will likely merge anything to the kernel you throw at him, I do wonder how to validate the uapi from a security perspective. It's always interesting when someone wraps a DMA engine with user ioctls, and without enough information to decide if the DMA engine is secure against userspace misprogramming it.

I'm curious, what information might you be looking for? Are you concerned about the device attacking the host, or the host attacking the device?

Quoted 3 lines

Also if we don't understand the programming API on board the device, we can't tell if the "core" on the device are able to reprogram the device engines either.

So, you are looking for details about the messaging protocol which are considered opaque to the kernel driver? Or something else?

Quoted 3 lines

Figuring this out is difficult at the best of times, it helps if there is access to the complete device documentation or user space side drivers in order to faciliate this.

Regarding access to documentation, sadly that isn't going to happen now, or in the near future. Again, above my pay grade. The only public "documentation" is what you can see from my emails.

I understand your position, and if I can "bound" the information you are looking for, I can see what I can do about getting you what you want. No promises, but I will try.

Quoted 2 lines

The other area I mention is testing the uAPI, how do you envisage regression testing and long term sustainability of the uAPI?

Can you clarify what you mean by "uAPI"? Are you referring to the interface between the device and the kernel driver?

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#32 lore raw
Quoted 35 lines · Jeffrey Hugo
Quoted 24 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 6 lines

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

We have daily internal testing, although I don't expect you to take my word for that.

Quoted 7 lines

I would like to get one of these devices into the hands of Linaro, so that it can be put into KernelCI. Similar to other Qualcomm products. I'm trying to convince the powers that be to make this happen.

Regarding what the community could do on its own, everything but the Linux driver is considered proprietary - that includes the on device firmware and the entire userspace stack. This is a decision above my pay grade.

Ok, that's a decision you are going to have to push upward on, as we really can't take this without a working, open, userspace.

Especially given the copyright owner of this code, that would be just crazy and foolish to not have open userspace code as well. Firmware would also be wonderful as well, go poke your lawyers about derivative work issues and the like for fun conversations :)

So without that changed, I'm not going to take this, and push to object that anyone else take this.

I'm not going to be able to review any of this code anymore until that changes, sorry.

thanks,

greg k-h

#33 lore raw
Quoted 40 lines · Greg Kroah-Hartman
Quoted 39 lines · Jeffrey Hugo
Quoted 24 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 6 lines

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

We have daily internal testing, although I don't expect you to take my word for that.

I would like to get one of these devices into the hands of Linaro, so that it can be put into KernelCI. Similar to other Qualcomm products. I'm trying to convince the powers that be to make this happen.

Quoted 6 lines
Quoted 3 lines

Regarding what the community could do on its own, everything but the Linux driver is considered proprietary - that includes the on device firmware and the entire userspace stack. This is a decision above my pay grade.

Ok, that's a decision you are going to have to push upward on, as we really can't take this without a working, open, userspace.

Fair enough. I hope that your position may have made things easier for me.

I hope this doesn't widen the rift as it were, but what is the "bar" for this userspace?

Is a simple test application that adds two numbers on the hardware acceptable?

What is the bar "working"? I intend to satisfy this request in good faith, but I wonder, if no one has the hardware besides our customers, and possibly KernelCI, can you really say that I've provided a working userspace?

Quoted 4 lines

Especially given the copyright owner of this code, that would be just crazy and foolish to not have open userspace code as well. Firmware would also be wonderful as well, go poke your lawyers about derivative work issues and the like for fun conversations :)

Those are the kind of conversations I try to avoid :)

Quoted 10 lines

So without that changed, I'm not going to take this, and push to object that anyone else take this.

I'm not going to be able to review any of this code anymore until that changes, sorry.

thanks,

greg k-h

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#34 lore raw
Quoted 48 lines · Jeffrey Hugo
Quoted 40 lines · Greg Kroah-Hartman
Quoted 39 lines · Jeffrey Hugo
Quoted 24 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 6 lines

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

We have daily internal testing, although I don't expect you to take my word for that.

I would like to get one of these devices into the hands of Linaro, so that it can be put into KernelCI. Similar to other Qualcomm products. I'm trying to convince the powers that be to make this happen.

Quoted 6 lines
Quoted 3 lines

Regarding what the community could do on its own, everything but the Linux driver is considered proprietary - that includes the on device firmware and the entire userspace stack. This is a decision above my pay grade.

Ok, that's a decision you are going to have to push upward on, as we really can't take this without a working, open, userspace.

Quoted 7 lines

Fair enough. I hope that your position may have made things easier for me.

I hope this doesn't widen the rift as it were, but what is the "bar" for this userspace?

Is a simple test application that adds two numbers on the hardware acceptable?

Make it the real library that you use for your applications that anyone can then also use as well if they have the hardware. Why would you want something "crippled"?

Quoted 3 lines

What is the bar "working"? I intend to satisfy this request in good faith, but I wonder, if no one has the hardware besides our customers, and possibly KernelCI, can you really say that I've provided a working userspace?

How do you know who your customers really are, or who they sell the chips to? I could end up with one of these... :)

Quoted 6 lines
Quoted 4 lines

Especially given the copyright owner of this code, that would be just crazy and foolish to not have open userspace code as well. Firmware would also be wonderful as well, go poke your lawyers about derivative work issues and the like for fun conversations :)

Those are the kind of conversations I try to avoid :)

Sounds like you are going to now have to have them, have fun!

greg k-h

#35 lore raw
Quoted 54 lines · Greg Kroah-Hartman
Quoted 53 lines · Jeffrey Hugo
Quoted 40 lines · Greg Kroah-Hartman
Quoted 39 lines · Jeffrey Hugo
Quoted 24 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 6 lines

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

We have daily internal testing, although I don't expect you to take my word for that.

I would like to get one of these devices into the hands of Linaro, so that it can be put into KernelCI. Similar to other Qualcomm products. I'm trying to convince the powers that be to make this happen.

Quoted 6 lines
Quoted 3 lines

Regarding what the community could do on its own, everything but the Linux driver is considered proprietary - that includes the on device firmware and the entire userspace stack. This is a decision above my pay grade.

Ok, that's a decision you are going to have to push upward on, as we really can't take this without a working, open, userspace.

Fair enough. I hope that your position may have made things easier for me.

I hope this doesn't widen the rift as it were, but what is the "bar" for this userspace?

Quoted 6 lines
Quoted 2 lines

Is a simple test application that adds two numbers on the hardware acceptable?

Make it the real library that you use for your applications that anyone can then also use as well if they have the hardware. Why would you want something "crippled"?

It makes it easier to dance around real or perceived IP issues, and thus I can likely more successfully "push upward" as you put it.

Quoted 7 lines
Quoted 4 lines

What is the bar "working"? I intend to satisfy this request in good faith, but I wonder, if no one has the hardware besides our customers, and possibly KernelCI, can you really say that I've provided a working userspace?

How do you know who your customers really are, or who they sell the chips to? I could end up with one of these... :)

At this time, I don't think that is going to happen, but I would like to see it regardless.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Especially given the copyright owner of this code, that would be just crazy and foolish to not have open userspace code as well. Firmware would also be wonderful as well, go poke your lawyers about derivative work issues and the like for fun conversations :)

Those are the kind of conversations I try to avoid :)

Sounds like you are going to now have to have them, have fun!

Honestly, I fail to see where you think there is a derivative work, so, I'm not really sure what discussions I need to revisit with our lawyers.

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#36 lore raw
Quoted 6 lines · Jeffrey Hugo
Quoted 5 lines · Greg Kroah-Hartman
Quoted 5 lines
Quoted 4 lines

Especially given the copyright owner of this code, that would be just crazy and foolish to not have open userspace code as well. Firmware would also be wonderful as well, go poke your lawyers about derivative work issues and the like for fun conversations :)

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Those are the kind of conversations I try to avoid :)

Sounds like you are going to now have to have them, have fun!

Honestly, I fail to see where you think there is a derivative work, so, I'm not really sure what discussions I need to revisit with our lawyers.

Given that we are not lawyers, why don't we leave those types of discussions up to the lawyers, and not depend on people like me and you for that? :)

If your lawyers think that the code division is fine as-is, that's great, I'd be glad to review it if they add their signed-off-by: on it verifying that the api divide is approved by them.

thanks!

greg k-h

#37 lore raw
In reply to Dave Airlie
Quoted 31 lines · Dave Airlie
Quoted 16 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 5 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Hi Jeffery,

Just wondering what the userspace/testing plans for this driver.

This introduces a new user facing API for a device without pointers to users or tests for that API.

Quoted 6 lines

Although this isn't a graphics driver, and Greg will likely merge anything to the kernel you throw at him, I do wonder how to validate the uapi from a security perspective. It's always interesting when someone wraps a DMA engine with user ioctls, and without enough information to decide if the DMA engine is secure against userspace misprogramming it.

Hey, I'll not merge just anything!

Oh, well, maybe, if it's in staging :)

Quoted 10 lines

Also if we don't understand the programming API on board the device, we can't tell if the "core" on the device are able to reprogram the device engines either.

Figuring this out is difficult at the best of times, it helps if there is access to the complete device documentation or user space side drivers in order to faciliate this.

The other area I mention is testing the uAPI, how do you envisage regression testing and long term sustainability of the uAPI?

I agree with this request, we should have some code that we can run in order to test that things work properly.

thanks,

greg k-h

#38 lore raw
In reply to Jeffrey Hugo

Hi Jeff,

Quoted 15 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

Quoted 7 lines

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Design:

Can you add documentation in next revision with all this information (or more)? In restructured text ofc. Eventhough it is an RFC series, adding documentation doesn't hurt and it will help reviewers to understand the hardware better.

Thanks,
Mani

Quoted 145 lines
+--------------------------------+
|       AI application           |
|       (userspace)              |
+-------------+------------------+
              |
              | Char dev interface
              |
              |
+-------------+------------------+
|       QAIC driver              |
|       (kernel space)           |
|                                |
+----+------------------+--------+
     |                  |
     |                  |
     |                  |
     |                  |
     |Control path      | Data path
     |(MHI bus)         |
     |                  |
     |                  |
     |                  |
     |                  |
+--------------------------------+
| +--------+      +------------+ |
| | MHI HW |      |DMA Bridge  | |
| +--------+      |(DMA engine)| |
|                 +------------+ |
|                                |
|                                |
|                                |
|  Qualcomm Cloud AI 100 device  |
|                                |
|                                |
+--------------------------------+

A Qualcomm Cloud AI 100 device (QAIC device from here on) is a PCIe hardware accelerator for AI inference workloads. Over the PCIe bus fabric, a QAIC device exposes two interfaces via PCI BARs - a MHI hardware region and a DMA engine hardware region.

Before workloads can be run, a QAIC device needs to be initialized. Similar to other Qualcomm products with incorperate MHI, device firmware needs to be loaded onto the device from the host. This occurs in two stages. First, a secondary bootloader (SBL) needs to be loaded onto the device. This occurs via the BHI protocol, and is handled by the MHI bus. Once the SBL loaded and running, it activates the Sahara protocol. The Sahara protocol is used with a userspace application to load and initialize the remaining firmware. The Sahara protocol and associated userspace application are outside the scope of this series as they have no direct interaction with the QAIC driver.

Once a QAIC device is fully initialized, workloads can be sent to the device and run. This involves a per-device instance char dev that the QAIC driver exposes to userspace. Running a workload involves two phases - configuring the device, and interacting with the workload.

To configure the device, commands are sent via a MHI channel. This is referred to as the control path. A command is a single message. A message contains one or more transactions. Transactions are operations that the device is requested to perform. Most commands are opaque to the kernel driver, however some are not. For example, if the user application wishes to DMA data to the device, it requires the assistance of the kernel driver to translate the data addresses to an address space that the device can understand. In this instance the transaction for DMAing the data is visible to the kernel driver, and the driver will do the required transformation when encoding the message.

To interact with the workload, the workload is assigned a DMA Bridge Channel (dbc). This is dedicated hardware within the DMA engine. Interacting with the workload consists of sending it input data, and receiving output data. The user application requests appropiate buffers from the kernel driver, prepares the buffers, and directs the kernel driver to queue them to the hardware.

The kernel driver is required to support multiple QAIC devices, and also N users per device.

Status:
This series introduces the driver for QAIC devices, and builds up the minimum functionality for running workloads. Several features which have been omitted or are still planned are indicated in the future work section.

Before exiting the RFC phase, and attempting full acceptance, we wish to complete two features which are currently under development as we expect there to be userspace interface changes as a result.

The first feature is a variable length control message between the kernel driver and the device. This allows us to support the total number of DMA transactions we require for certain platforms, while minimizing memory usage. The interface impact of this would be to allow us to drop the size of the manage buffer between userspace and the kernel driver from the current 16k, much of which is wasted.

The second feature is an optimization and extension of the data path interface. We plan to move the bulk of the data in the qaic_execute structure to the qaic_mem_req structure, which optimized our critical path processing. We also plan to extend the qaic_execute structure to allow for a batch submit of multiple buffers as an optimization and convenience for userspace.

Future work:
For simplicity, we have omitted work related to the following features, and intend to submit in future series:

(quoted diff) +0 -3
-debugfs
-trace points
-hwmon (device telemetry)

We are also investigating what it might mean to support dma_bufs. We expect that such support would come as an extension of the interface.

Jeffrey Hugo (8):
qaic: Add skeleton driver
qaic: Add and init a basic mhi controller
qaic: Create char dev
qaic: Implement control path
qaic: Implement data path
qaic: Implement PCI link status error handlers
qaic: Implement MHI error status handler
MAINTAINERS: Add entry for QAIC driver

 MAINTAINERS                        |    7 +
 drivers/misc/Kconfig               |    1 +
 drivers/misc/Makefile              |    1 +
 drivers/misc/qaic/Kconfig          |   20 +
 drivers/misc/qaic/Makefile         |   12 +
 drivers/misc/qaic/mhi_controller.c |  538 +++++++++++++++++++
 drivers/misc/qaic/mhi_controller.h |   20 +
 drivers/misc/qaic/qaic.h           |  111 ++++
 drivers/misc/qaic/qaic_control.c   | 1015 ++++++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_data.c      |  952 +++++++++++++++++++++++++++++++++
 drivers/misc/qaic/qaic_drv.c       |  699 +++++++++++++++++++++++++
 include/uapi/misc/qaic.h           |  246 +++++++++
 12 files changed, 3622 insertions(+)
 create mode 100644 drivers/misc/qaic/Kconfig
 create mode 100644 drivers/misc/qaic/Makefile
 create mode 100644 drivers/misc/qaic/mhi_controller.c
 create mode 100644 drivers/misc/qaic/mhi_controller.h
 create mode 100644 drivers/misc/qaic/qaic.h
 create mode 100644 drivers/misc/qaic/qaic_control.c
 create mode 100644 drivers/misc/qaic/qaic_data.c
 create mode 100644 drivers/misc/qaic/qaic_drv.c
 create mode 100644 include/uapi/misc/qaic.h
Signature · 2 lines
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
#39 lore raw
Quoted 24 lines · Manivannan Sadhasivam

Hi Jeff,

Quoted 21 lines · Jeffrey Hugo

Introduction:
Qualcomm Cloud AI 100 is a PCIe adapter card which contains a dedicated SoC ASIC for the purpose of efficently running Deep Learning inference workloads in a data center environment.

The offical press release can be found at -
https://www.qualcomm.com/news/releases/2019/04/09/qualcomm-brings-power-efficient-artificial-intelligence-inference

The offical product website is -
https://www.qualcomm.com/products/datacenter-artificial-intelligence

At the time of the offical press release, numerious technology news sites also covered the product. Doing a search of your favorite site is likely to find their coverage of it.

It is our goal to have the kernel driver for the product fully upstream. The purpose of this RFC is to start that process. We are still doing development (see below), and thus not quite looking to gain acceptance quite yet, but now that we have a working driver we beleive we are at the stage where meaningful conversation with the community can occur.

Quoted 5 lines
Quoted 1 line

Design:

Can you add documentation in next revision with all this information (or more)? In restructured text ofc. Eventhough it is an RFC series, adding documentation doesn't hurt and it will help reviewers to understand the hardware better.

Sorry, saw this hit my inbox as I was sending out the next rev. There will be another rev.

Sure. I'm open to doing that. Hmm, Documentation/misc-devices seem good?

Do you have specific additional information you think would be good?

Signature · 3 lines
Jeffrey Hugo
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.