Skip to contentSkip to conversation

media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

PATCH Superseded Wentong Wu 1 participant 4 messages May 25, 2023

Subsystems (3)

v7 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Wentong Wu

author
#0 lore raw

v7 cover media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using the interfaces exported via v4l2 sub-device.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via v4l2 control API.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- ivsc_csi: MEI client driver to send commands and receive notifications
from CSI module.
- ivsc_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

---
v7:
- add more comments for IVSC ACPI dev

v6:
- add V4L2_SUBDEV_FL_HAS_EVENTS flag for csi subdev
- add the LINK FREQUENCY volatile control
- query sensor link frequency with v4l2_get_link_freq api
- add error handling for creating device link
- fix coding style issues

v5:
- probe mei_csi only after software node has been setup

v4:
- call v4l2_ctrl_handler_free() if setting up the handler failed
- set V4L2_CTRL_FLAG_READ_ONLY for privacy_ctrl
- add dev_warn if failed to switch CSI-2 link to IVSC
- use v4l2_fwnode_endpoint_alloc_parse to get num_data_lanes
- add document about how sensor connected to IVSC is powered
- move lock to mei_ace_send
- check return value for device_link_add

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
ACPI: delay enumeration of devices with a _DEP pointing to IVSC device

 drivers/acpi/scan.c                    |   2 +
 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   9 +
 drivers/media/pci/intel/ivsc/mei_ace.c | 591 ++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.c | 809 +++++++++++++++++++++++++++++++++
 7 files changed, 1426 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v7 1/3 media: pci: intel: ivsc: Add CSI submodule

CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.

The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To route camera sensor data to host, the information of link frequency and number of data lanes is sent to firmware by sending MEI command when starting stream.

CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, user is notified via v4l2 control callback.

---
 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   6 +
 drivers/media/pci/intel/ivsc/mei_csi.c | 809 +++++++++++++++++++++++++++++++++
 5 files changed, 830 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
drivers/media/pci/Kconfig +1 -0
index dff0b45..c157f7c 100644
@@ -72,6 +72,7 @@ config VIDEO_PCI_SKELETON
 	  when developing new drivers.
 
 source "drivers/media/pci/intel/ipu3/Kconfig"
+source "drivers/media/pci/intel/ivsc/Kconfig"
 
 endif #MEDIA_PCI_SUPPORT
 endif #PCI
drivers/media/pci/intel/Makefile +2 -0
index 0b4236c..d27ca63 100644
@@ -4,3 +4,5 @@
 #
 
 obj-y	+= ipu3/
+
+obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
new file mode 100644
index 0000000..9535ac1
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+config INTEL_VSC
+	tristate "Intel Visual Sensing Controller"
+	depends on INTEL_MEI
+	help
+	  This adds support for Intel Visual Sensing Controller (IVSC).
+
+	  Enables the IVSC firmware services required for controlling
+	  camera sensor ownership and CSI-2 link through Image Processing
+	  Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +6 -0
new file mode 100644
index 0000000..cbd194a
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o
+ivsc-csi-y += mei_csi.o
drivers/media/pci/intel/ivsc/mei_csi.c new file +809 -0

809 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#2 lore raw

PATCH v7 2/3 media: pci: intel: ivsc: Add ACE submodule

In reply to Wentong Wu

ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.

The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.

Device link(device_link_add) is used to set the right camera sensor ownership before accessing the sensor via I2C. With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE, the supplier device will be PM runtime resumed before the consumer(camera sensor). So use runtime PM callbacks to transfer the ownership between host and IVSC.

---
 drivers/media/pci/intel/ivsc/Makefile  |   3 +
 drivers/media/pci/intel/ivsc/mei_ace.c | 591 +++++++++++++++++++++++++++++++++
 2 files changed, 594 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
drivers/media/pci/intel/ivsc/Makefile +3 -0
index cbd194a..00fad29 100644
@@ -4,3 +4,6 @@
 
 obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o
 ivsc-csi-y += mei_csi.o
+
+obj-$(CONFIG_INTEL_VSC) += ivsc-ace.o
+ivsc-ace-y += mei_ace.o
drivers/media/pci/intel/ivsc/mei_ace.c new file +591 -0

591 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#3 lore raw

PATCH v7 3/3 ACPI: delay enumeration of devices with a _DEP pointing to IVSC device

In reply to Wentong Wu

Inside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.

This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a _DEP dependency on the matching mei bus ACPI device, so adding IVSC mei bus ACPI device to acpi_honor_dep_ids allows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a _DEP dependency on an IVSC mei bus ACPI device.

On TGL platform, the HID of IVSC mei bus ACPI device is INTC1059, and on ADL platform, the HID is INTC1095. So add both of them to acpi_honor_dep_ids.

---
 drivers/acpi/scan.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/acpi/scan.c +2 -0
index 2743444..04560e8 100644
@@ -796,6 +796,8 @@ static const char * const acpi_ignore_dep_ids[] = {
 /* List of HIDs for which we honor deps of matching ACPI devs, when checking _DEP lists. */
 static const char * const acpi_honor_dep_ids[] = {
 	"INT3472", /* Camera sensor PMIC / clk and regulator info */
+	"INTC1059", /* IVSC (TGL) driver must be loaded to allow i2c access to camera sensors */
+	"INTC1095", /* IVSC (ADL) driver must be loaded to allow i2c access to camera sensors */
 	NULL
 };
 
Signature · 1 line
2.7.4