Skip to contentSkip to conversation

media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

PATCH Superseded Wentong Wu 2 participants 5 messages Feb 10, 2023 – Feb 11, 2023

Subsystems (2)

v1 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Wentong Wu

author
#0 lore raw

cover media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing when
IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera sensor
data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications from
CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 341 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1089 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/3 media: pci: intel: ivsc: Add CSI submodule

CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.

The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To route camera sensor data to host, the caller specifies link frequency and number of data lanes. This information is sent to firmware by sending MEI command.

Two APIs are exported: csi_set_link_owner is used to switch ownership of CSI-2 link, and csi_set_link_cfg is to configure CSI-2 link when routing camera sensor data to host.

CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via the registered callback, so that user can be notified.

---
 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   5 +
 drivers/media/pci/intel/ivsc/mei_csi.c | 341 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 +++++
 include/linux/ivsc.h                   |  19 ++
 7 files changed, 440 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
drivers/media/pci/Kconfig +1 -0
index 1224d908713a..16661ab0b816 100644
@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON
 	  when developing new drivers.
 
 source "drivers/media/pci/intel/ipu3/Kconfig"
+source "drivers/media/pci/intel/ivsc/Kconfig"
 
 endif #MEDIA_PCI_SUPPORT
 endif #PCI
drivers/media/pci/intel/Makefile +2 -0
index 0b4236c4db49..d27ca636c860 100644
@@ -4,3 +4,5 @@
 #
 
 obj-y	+= ipu3/
+
+obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
new file mode 100644
index 000000000000..9535ac10f4f7
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+config INTEL_VSC
+	tristate "Intel Visual Sensing Controller"
+	depends on INTEL_MEI
+	help
+	  This adds support for Intel Visual Sensing Controller (IVSC).
+
+	  Enables the IVSC firmware services required for controlling
+	  camera sensor ownership and CSI-2 link through Image Processing
+	  Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +5 -0
new file mode 100644
index 000000000000..1825aad45cff
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+obj-$(CONFIG_INTEL_VSC) += mei_csi.o
drivers/media/pci/intel/ivsc/mei_csi.c new file +341 -0

341 lines not on this page · raw message

drivers/media/pci/intel/ivsc/mei_csi.h new file +60 -0

60 lines not on this page · raw message

include/linux/ivsc.h new file +19 -0
new file mode 100644
index 000000000000..6572ca4f340c
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#ifndef _LINUX_IVSC_H_
+#define _LINUX_IVSC_H_
+
+#include <linux/types.h>
+
+/* IVSC privacy status definition */
+enum ivsc_privacy_status {
+	IVSC_PRIVACY_OFF,
+	IVSC_PRIVACY_ON,
+	IVSC_PRIVACY_MAX,
+};
+
+#endif
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v1 1/3] media: pci: intel: ivsc: Add CSI submodule

Thank you for the patch! Yet something to improve:

Hi Wentong,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on media-tree/master]
[also build test ERROR on sailus-media-tree/streams linus/master v6.2-rc7 next-20230210] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230210-090202
base:   git://linuxtv.org/media_tree.git master
patch link:    https://lore.kernel.org/r/20230210010221.2466486-2-wentong.wu%40intel.com
patch subject: [PATCH v1 1/3] media: pci: intel: ivsc: Add CSI submodule
config: i386-allyesconfig (https://download.01.org/0day-ci/archive/20230211/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0

reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/bb45f3cb357165838a3c90d4bddfad40ad2d8ce7
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230210-090202 git checkout bb45f3cb357165838a3c90d4bddfad40ad2d8ce7
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=i386 olddefconfig
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

ld: drivers/media/pci/intel/ivsc/mei_csi.o: in function `csi_set_link_cfg':

>> mei_csi.c:(.text+0x1a6): undefined reference to `__udivdi3'
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
author
#3 lore raw

PATCH 2/3 media: pci: intel: ivsc: Add ACE submodule

In reply to Wentong Wu

ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.

The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.

The only exported API ace_set_camera_owner is to switch ownership of camera sensor.

---
 drivers/media/pci/intel/ivsc/Makefile  |   1 +
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 3 files changed, 509 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
drivers/media/pci/intel/ivsc/Makefile +1 -0
index 1825aad45cff..de0a425c22c2 100644
@@ -3,3 +3,4 @@
 # Copyright (C) 2023, Intel Corporation. All rights reserved.
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
+obj-$(CONFIG_INTEL_VSC) += mei_ace.o
drivers/media/pci/intel/ivsc/mei_ace.c new file +472 -0

472 lines not on this page · raw message

drivers/media/pci/intel/ivsc/mei_ace.h new file +36 -0
new file mode 100644
index 000000000000..0302bc899521
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller ACE external interface
+ */
+
+#ifndef _MEI_ACE_H_
+#define _MEI_ACE_H_
+
+#include <linux/types.h>
+
+/* identify camera sensor ownership */
+enum ace_camera_owner {
+	ACE_CAMERA_IVSC,
+	ACE_CAMERA_HOST,
+};
+
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief set camera sensor ownership
+ *
+ * @param owner Camera sensor ownership being set
+ *
+ * @return 0 on success, negative on failure
+ */
+int ace_set_camera_owner(enum ace_camera_owner owner);
+
+#else
+static inline int ace_set_camera_owner(enum ace_camera_owner owner)
+{
+	return 0;
+}
+
+#endif
+
+#endif
Signature · 1 line
2.25.1
author
#4 lore raw

PATCH 3/3 media: pci: intel: ivsc: Add acquire/release API for ivsc

In reply to Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0

84 lines not on this page · raw message

include/linux/ivsc.h +55 -0

59 lines not on this page · raw message

Signature · 1 line
2.25.1