Skip to contentSkip to conversation

media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

PATCH Superseded Wentong Wu 6 participants 54 messages Feb 13, 2023 – Mar 19, 2023 4 messages not in the archive

Subsystems (2)

v2 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Wentong Wu

author
#0 lore raw

v2 cover media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/3 media: pci: intel: ivsc: Add CSI submodule

CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.

The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To route camera sensor data to host, the caller specifies link frequency and number of data lanes. This information is sent to firmware by sending MEI command.

Two APIs are exported: csi_set_link_owner is used to switch ownership of CSI-2 link, and csi_set_link_cfg is to configure CSI-2 link when routing camera sensor data to host.

CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via the registered callback, so that user can be notified.

---
 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   5 +
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 +++++
 include/linux/ivsc.h                   |  19 ++
 7 files changed, 441 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
drivers/media/pci/Kconfig +1 -0
index 1224d908713a..16661ab0b816 100644
@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON
 	  when developing new drivers.
 
 source "drivers/media/pci/intel/ipu3/Kconfig"
+source "drivers/media/pci/intel/ivsc/Kconfig"
 
 endif #MEDIA_PCI_SUPPORT
 endif #PCI
drivers/media/pci/intel/Makefile +2 -0
index 0b4236c4db49..d27ca636c860 100644
@@ -4,3 +4,5 @@
 #
 
 obj-y	+= ipu3/
+
+obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
new file mode 100644
index 000000000000..9535ac10f4f7
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+config INTEL_VSC
+	tristate "Intel Visual Sensing Controller"
+	depends on INTEL_MEI
+	help
+	  This adds support for Intel Visual Sensing Controller (IVSC).
+
+	  Enables the IVSC firmware services required for controlling
+	  camera sensor ownership and CSI-2 link through Image Processing
+	  Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +5 -0
new file mode 100644
index 000000000000..1825aad45cff
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+obj-$(CONFIG_INTEL_VSC) += mei_csi.o
drivers/media/pci/intel/ivsc/mei_csi.c new file +342 -0

342 lines not on this page · raw message

drivers/media/pci/intel/ivsc/mei_csi.h new file +60 -0

60 lines not on this page · raw message

include/linux/ivsc.h new file +19 -0
new file mode 100644
index 000000000000..6572ca4f340c
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#ifndef _LINUX_IVSC_H_
+#define _LINUX_IVSC_H_
+
+#include <linux/types.h>
+
+/* IVSC privacy status definition */
+enum ivsc_privacy_status {
+	IVSC_PRIVACY_OFF,
+	IVSC_PRIVACY_ON,
+	IVSC_PRIVACY_MAX,
+};
+
+#endif
Signature · 1 line
2.25.1
maintainer
#2 lore raw

Re: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule

Hi Wentong,

Cc'ing Hans as well.

Quoted 22 lines · Wentong Wu

CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.

The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To route camera sensor data to host, the caller specifies link frequency and number of data lanes. This information is sent to firmware by sending MEI command.

Two APIs are exported: csi_set_link_owner is used to switch ownership of CSI-2 link, and csi_set_link_cfg is to configure CSI-2 link when routing camera sensor data to host.

CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via the registered callback, so that user can be notified.

Quoted 8 lines
---
 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   5 +
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 +++++
 include/linux/ivsc.h                   |  19 ++

Please add a MAINTAINERS entry for this (ivsc dir).

Quoted 64 lines
 7 files changed, 441 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
drivers/media/pci/Kconfig +1 -0
index 1224d908713a..16661ab0b816 100644
@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON
 	  when developing new drivers.
 
 source "drivers/media/pci/intel/ipu3/Kconfig"
+source "drivers/media/pci/intel/ivsc/Kconfig"
 
 endif #MEDIA_PCI_SUPPORT
 endif #PCI
drivers/media/pci/intel/Makefile +2 -0
index 0b4236c4db49..d27ca636c860 100644
@@ -4,3 +4,5 @@
 #
 
 obj-y	+= ipu3/
+
+obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
new file mode 100644
index 000000000000..9535ac10f4f7
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+config INTEL_VSC
+	tristate "Intel Visual Sensing Controller"
+	depends on INTEL_MEI
+	help
+	  This adds support for Intel Visual Sensing Controller (IVSC).
+
+	  Enables the IVSC firmware services required for controlling
+	  camera sensor ownership and CSI-2 link through Image Processing
+	  Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +5 -0
new file mode 100644
index 000000000000..1825aad45cff
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+obj-$(CONFIG_INTEL_VSC) += mei_csi.o
drivers/media/pci/intel/ivsc/mei_csi.c new file +0 -0
new file mode 100644
index 000000000000..309e52387502

drivers/media/pci/intel/ivsc/mei_csi.c line 3

Quoted 3 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +3 -0
@@ -0,0 +1,342 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.

Do you still need the "all rights reserved" part?

drivers/media/pci/intel/ivsc/mei_csi.c line 7

Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +4 -0
+ * Intel Visual Sensing Controller CSI Linux driver
+ */
+
+/*

You can merge the two comments.

Quoted 137 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +137 -0
+ * To set ownership of CSI-2 link and to configure CSI-2 link, there
+ * are specific commands, which are sent via MEI protocol. The send
+ * command function uses "completion" as a synchronization mechanism.
+ * The response for command is received via a mei callback which wakes
+ * up the caller. There can be only one outstanding command at a time.
+ */
+
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/kernel.h>
+#include <linux/math64.h>
+#include <linux/mei_cl_bus.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/units.h>
+#include <linux/uuid.h>
+#include <linux/workqueue.h>
+
+#include "mei_csi.h"
+
+#define MEI_CSI_DRIVER_NAME "ivsc_csi"
+
+/* the 5s used here is based on experiment */
+#define CSI_CMD_TIMEOUT (5 * HZ)
+/* to setup CSI-2 link an extra delay needed and determined experimentally */
+#define CSI_FW_READY_DELAY_MS 100
+/* link frequency unit is 100kHz */
+#define CSI_LINK_FREQ(x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ)))
+
+/*
+ * identify the command id supported by firmware
+ * IPC, as well as the privacy notification id
+ * used when processing privacy event.
+ */
+enum csi_cmd_id {
+	/* used to set csi ownership */
+	CSI_SET_OWNER = 0,
+
+	/* used to configure CSI-2 link */
+	CSI_SET_CONF = 2,
+
+	/* privacy notification id used when privacy state changes */
+	CSI_PRIVACY_NOTIF = 6,
+};
+
+/* configuration of the CSI-2 link between host and IVSC */
+struct csi_link_cfg {
+	/* number of data lanes used on the CSI-2 link */
+	u32 nr_of_lanes;
+
+	/* frequency of the CSI-2 link */
+	u32 link_freq;
+
+	/* for future use */
+	u32 rsvd[2];
+} __packed;
+
+/* CSI command structure */
+struct csi_cmd {
+	u32 cmd_id;
+	union _cmd_param {
+		u32 param;
+		struct csi_link_cfg conf;
+	} param;
+} __packed;
+
+/* CSI notification structure */
+struct csi_notif {
+	u32 cmd_id;
+	int status;
+	union _resp_cont {
+		u32 cont;
+		struct csi_link_cfg conf;
+	} cont;
+} __packed;
+
+struct mei_csi {
+	struct mei_cl_device *cldev;
+
+	/* command response */
+	struct csi_notif cmd_response;
+	/* used to wait for command response from firmware */
+	struct completion cmd_completion;
+
+	/* work element used to handle firmware event */
+	struct work_struct event_work;
+
+	/* privacy status */
+	enum ivsc_privacy_status status;
+	/* privacy callback */
+	void (*callback)(void *, enum ivsc_privacy_status);
+	/* privacy callback runtime context */
+	void *context;
+};
+
+/* only one for now */
+static struct mei_csi *csi;
+/* lock used to prevent multiple call to csi */
+static DEFINE_MUTEX(csi_mutex);
+
+/* send a command to firmware and mutex must be held by caller */
+static int mei_csi_send(u8 *buf, size_t len)
+{
+	struct csi_cmd *cmd = (struct csi_cmd *)buf;
+	int ret;
+
+	reinit_completion(&csi->cmd_completion);
+
+	ret = mei_cldev_send(csi->cldev, buf, len);
+	if (ret < 0)
+		goto out;
+
+	ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
+						   CSI_CMD_TIMEOUT);
+	if (ret < 0) {
+		goto out;
+	} else if (!ret) {
+		ret = -ETIMEDOUT;
+		goto out;
+	}
+
+	/* command response status */
+	ret = csi->cmd_response.status;
+	if (ret) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (csi->cmd_response.cmd_id != cmd->cmd_id)
+		ret = -EINVAL;
+
+out:
+	return ret;
+}
+
+int csi_set_link_owner(enum csi_link_owner owner,

drivers/media/pci/intel/ivsc/mei_csi.c line 154

Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10 -0
+		       void (*callback)(void *, enum ivsc_privacy_status),
+		       void *context)
+{
+	struct csi_cmd cmd = { 0 };
+	size_t cmd_size;
+	int ret;
+
+	cmd.cmd_id = CSI_SET_OWNER;
+	cmd.param.param = owner;
+	cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.param);

You could initialise these in variable declaration.

Quoted 98 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +98 -0
+
+	mutex_lock(&csi_mutex);
+	if (unlikely(!csi)) {
+		mutex_unlock(&csi_mutex);
+		return -EAGAIN;
+	}
+
+	ret = mei_csi_send((u8 *)&cmd, cmd_size);
+
+	/*
+	 * cancel possible event work and wait for it to finish
+	 * to make sure no work running for the ongoing changes
+	 * to callback and context.
+	 */
+	cancel_work_sync(&csi->event_work);
+	csi->callback = callback;
+	csi->context = context;
+	mutex_unlock(&csi_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_NS_GPL(csi_set_link_owner, IVSC);
+
+int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
+{
+	struct csi_cmd cmd = { 0 };
+	size_t cmd_size;
+	int ret;
+
+	cmd.cmd_id = CSI_SET_CONF;
+	cmd.param.conf.nr_of_lanes = nr_of_lanes;
+	cmd.param.conf.link_freq = CSI_LINK_FREQ(link_freq);
+	cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.conf);
+
+	mutex_lock(&csi_mutex);
+	if (unlikely(!csi)) {
+		mutex_unlock(&csi_mutex);
+		return -EAGAIN;
+	}
+
+	ret = mei_csi_send((u8 *)&cmd, cmd_size);
+	/*
+	 * wait configuration ready if download success. placing
+	 * delay under mutex is to make sure current command flow
+	 * completed before starting a possible new one.
+	 */
+	if (!ret)
+		msleep(CSI_FW_READY_DELAY_MS);
+	mutex_unlock(&csi_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_NS_GPL(csi_set_link_cfg, IVSC);
+
+/* event handling routine */
+static void mei_csi_event_work(struct work_struct *work)
+{
+	if (csi->callback)
+		csi->callback(csi->context, READ_ONCE(csi->status));
+}
+
+/* callback for receive */
+static void mei_csi_rx(struct mei_cl_device *cldev)
+{
+	struct csi_notif notif = { 0 };
+	int ret;
+
+	ret = mei_cldev_recv(cldev, (u8 *)&notif, sizeof(notif));
+	if (ret < 0) {
+		dev_err(&cldev->dev, "recv error: %d\n", ret);
+		return;
+	}
+
+	switch (notif.cmd_id) {
+	case CSI_PRIVACY_NOTIF:
+		if (notif.cont.cont < IVSC_PRIVACY_MAX) {
+			WRITE_ONCE(csi->status, notif.cont.cont);
+
+			schedule_work(&csi->event_work);
+		}
+		break;
+	case CSI_SET_OWNER:
+	case CSI_SET_CONF:
+		memcpy(&csi->cmd_response, &notif, ret);
+
+		complete(&csi->cmd_completion);
+		break;
+	default:
+		break;
+	}
+}
+
+static int mei_csi_probe(struct mei_cl_device *cldev,
+			 const struct mei_cl_device_id *id)
+{
+	int ret;
+
+	mutex_lock(&csi_mutex);

drivers/media/pci/intel/ivsc/mei_csi.c line 262

Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10 -0
+	/*
+	 * only instance is possible in the current hardware,
+	 * but adding protection for future hardware.
+	 */
+	if (csi) {
+		ret = -EBUSY;
+		goto err_unlock;
+	}
+
+	csi = kzalloc(sizeof(*csi), GFP_KERNEL);

How do you know the MEI CSI and the MEI ACE devices are related to a given camera sensor? I'd expect this information to come from system firmware but I don't see this being done in this patchset.

Ideally there should be also a way to obtain the appropriate MEI CSI and MEI ACE devices without assuming there's a single one in the system.

Quoted 171 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +80 -0
+	if (!csi) {
+		ret = -ENOMEM;
+		goto err_unlock;
+	}
+
+	csi->cldev = cldev;
+	init_completion(&csi->cmd_completion);
+	INIT_WORK(&csi->event_work, mei_csi_event_work);
+
+	mei_cldev_set_drvdata(cldev, csi);
+
+	ret = mei_cldev_enable(cldev);
+	if (ret < 0) {
+		dev_err(&cldev->dev, "mei_cldev_enable failed: %d\n", ret);
+		goto err_free;
+	}
+
+	ret = mei_cldev_register_rx_cb(cldev, mei_csi_rx);
+	if (ret) {
+		dev_err(&cldev->dev, "event cb registration failed: %d\n", ret);
+		goto err_disable;
+	}
+	mutex_unlock(&csi_mutex);
+
+	return 0;
+
+err_disable:
+	mei_cldev_disable(cldev);
+
+err_free:
+	kfree(csi);
+	/* disable csi */
+	csi = NULL;
+
+err_unlock:
+	mutex_unlock(&csi_mutex);
+
+	return ret;
+}
+
+static void mei_csi_remove(struct mei_cl_device *cldev)
+{
+	mutex_lock(&csi_mutex);
+	/* disable mei csi client device */
+	mei_cldev_disable(cldev);
+
+	/* cancel event work and wait for it to finish */
+	cancel_work_sync(&csi->event_work);
+
+	kfree(csi);
+	/* disable csi */
+	csi = NULL;
+	mutex_unlock(&csi_mutex);
+}
+
+#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \
+			     0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA)
+
+static const struct mei_cl_device_id mei_csi_tbl[] = {
+	{ MEI_CSI_DRIVER_NAME, MEI_CSI_UUID, MEI_CL_VERSION_ANY },
+
+	/* required last entry */
+	{ }
+};
+MODULE_DEVICE_TABLE(mei, mei_csi_tbl);
+
+static struct mei_cl_driver mei_csi_driver = {
+	.id_table = mei_csi_tbl,
+	.name = MEI_CSI_DRIVER_NAME,
+
+	.probe = mei_csi_probe,
+	.remove = mei_csi_remove,
+};
+
+module_mei_cl_driver(mei_csi_driver);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_DESCRIPTION("Device driver for IVSC CSI");
+MODULE_LICENSE("GPL");
drivers/media/pci/intel/ivsc/mei_csi.h new file +60 -0
new file mode 100644
index 000000000000..0cc51e6a5250
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller CSI external interface
+ */
+
+#ifndef _MEI_CSI_H_
+#define _MEI_CSI_H_
+
+#include <linux/ivsc.h>
+
+/* CSI-2 link ownership definition */
+enum csi_link_owner {
+	CSI_LINK_IVSC,
+	CSI_LINK_HOST,
+};
+
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief set CSI-2 link ownership
+ *
+ * @param owner The csi ownership being set
+ * @param callback The pointer of privacy callback being set
+ * @param context Privacy callback runtime context
+ *
+ * @return 0 on success, negative on failure
+ */
+int csi_set_link_owner(enum csi_link_owner owner,
+		       void (*callback)(void *, enum ivsc_privacy_status),
+		       void *context);
+
+/*
+ * @brief configure CSI-2 link between host and IVSC
+ * with provided parameters
+ *
+ * @param nr_of_lanes The number of data lanes being used
+ * on the CSI-2 link
+ * @param link_freq The frequency being set on the CSI-2 link
+ *
+ * @return 0 on success, negative on failure
+ */
+int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq);
+
+#else
+static inline
+int csi_set_link_owner(enum csi_link_owner owner,
+		       void (*callback)(void *, enum ivsc_privacy_status),
+		       void *context)
+{
+	return 0;
+}
+
+static inline int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
+{
+	return 0;
+}
+
+#endif
+
+#endif
include/linux/ivsc.h new file +19 -0
new file mode 100644
index 000000000000..6572ca4f340c
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#ifndef _LINUX_IVSC_H_
+#define _LINUX_IVSC_H_
+
+#include <linux/types.h>
+
+/* IVSC privacy status definition */
+enum ivsc_privacy_status {
+	IVSC_PRIVACY_OFF,
+	IVSC_PRIVACY_ON,
+	IVSC_PRIVACY_MAX,
+};
+
+#endif
Signature · 3 lines
Kind regards,

Sakari Ailus
author
#3 lore raw

PATCH v2 2/3 media: pci: intel: ivsc: Add ACE submodule

In reply to Wentong Wu

ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.

The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.

To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.

The only exported API ace_set_camera_owner is to switch ownership of camera sensor.

---
 drivers/media/pci/intel/ivsc/Makefile  |   1 +
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 3 files changed, 509 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
drivers/media/pci/intel/ivsc/Makefile +1 -0
index 1825aad45cff..de0a425c22c2 100644
@@ -3,3 +3,4 @@
 # Copyright (C) 2023, Intel Corporation. All rights reserved.
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
+obj-$(CONFIG_INTEL_VSC) += mei_ace.o
drivers/media/pci/intel/ivsc/mei_ace.c new file +472 -0

472 lines not on this page · raw message

drivers/media/pci/intel/ivsc/mei_ace.h new file +36 -0
new file mode 100644
index 000000000000..0302bc899521
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller ACE external interface
+ */
+
+#ifndef _MEI_ACE_H_
+#define _MEI_ACE_H_
+
+#include <linux/types.h>
+
+/* identify camera sensor ownership */
+enum ace_camera_owner {
+	ACE_CAMERA_IVSC,
+	ACE_CAMERA_HOST,
+};
+
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief set camera sensor ownership
+ *
+ * @param owner Camera sensor ownership being set
+ *
+ * @return 0 on success, negative on failure
+ */
+int ace_set_camera_owner(enum ace_camera_owner owner);
+
+#else
+static inline int ace_set_camera_owner(enum ace_camera_owner owner)
+{
+	return 0;
+}
+
+#endif
+
+#endif
Signature · 1 line
2.25.1
author
#4 lore raw

PATCH v2 3/3 media: pci: intel: ivsc: Add acquire/release API for ivsc

In reply to Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0

84 lines not on this page · raw message

include/linux/ivsc.h +55 -0

59 lines not on this page · raw message

Signature · 1 line
2.25.1
maintainer
#5 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Wentong,

Thanks for the patchset.

Quoted 9 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 172 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0
new file mode 100644
index 000000000000..12996b587639
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+
+#include "mei_ace.h"
+#include "mei_csi.h"
+
+/* lock for ivsc APIs */
+static DEFINE_MUTEX(ivsc_mutex);
+
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;
+
+	mutex_unlock(&ivsc_mutex);
+
+	return 0;
+
+release_csi:
+	csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+
+release_camera:
+	ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
+
+int ivsc_release_camera(void)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch CSI-2 link to IVSC */
+	ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+	if (ret)
+		goto error;
+
+	/* switch camera sensor ownership to IVSC */
+	ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_release_camera);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_SOFTDEP("pre: mei_csi mei_ace");
+MODULE_DESCRIPTION("IVSC interface");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS(IVSC);
include/linux/ivsc.h +55 -0
index 6572ca4f340c..bc9006cd6efc 100644
@@ -16,4 +16,59 @@ enum ivsc_privacy_status {
 	IVSC_PRIVACY_MAX,
 };
 
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief Acquire camera sensor ownership to host and setup
+ * the CSI-2 link between host and IVSC
+ *
+ * IVSC provides a privacy mode. When the privacy mode is turned
+ * on, camera sensor can't be used. This means that both IVSC and
+ * host Image Processing Unit(IPU) can't get image data. And when
+ * this mode is turned on, host Image Processing Unit(IPU) driver
+ * is informed via the registered callback, so that user can be
+ * notified.
+ *
+ * @param nr_of_lanes Number of data lanes used on the CSI-2 link
+ * @param link_freq Frequency of the CSI-2 link
+ * @param callback The pointer of privacy callback function
+ * @param context Privacy callback function runtime context
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context);
+
+/*
+ * @brief Release camera sensor ownership and stop the CSI-2
+ * link between host and IVSC
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_release_camera(void);
+
+#else
+static inline
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	return 0;
+}
+
+static inline int ivsc_release_camera(void)
+{
+	return 0;
+}
+
+#endif
+
 #endif
Signature · 3 lines
Kind regards,

Sakari Ailus
#6 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Sakari,

Quoted 20 lines · Sakari Ailus

Hi Wentong,

Thanks for the patchset.

Quoted 9 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 4 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Regards,

Hans

Quoted 190 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 172 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0
new file mode 100644
index 000000000000..12996b587639
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+
+#include "mei_ace.h"
+#include "mei_csi.h"
+
+/* lock for ivsc APIs */
+static DEFINE_MUTEX(ivsc_mutex);
+
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;
+
+	mutex_unlock(&ivsc_mutex);
+
+	return 0;
+
+release_csi:
+	csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+
+release_camera:
+	ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
+
+int ivsc_release_camera(void)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch CSI-2 link to IVSC */
+	ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+	if (ret)
+		goto error;
+
+	/* switch camera sensor ownership to IVSC */
+	ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_release_camera);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_SOFTDEP("pre: mei_csi mei_ace");
+MODULE_DESCRIPTION("IVSC interface");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS(IVSC);
include/linux/ivsc.h +55 -0
index 6572ca4f340c..bc9006cd6efc 100644
@@ -16,4 +16,59 @@ enum ivsc_privacy_status {
 	IVSC_PRIVACY_MAX,
 };
 
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief Acquire camera sensor ownership to host and setup
+ * the CSI-2 link between host and IVSC
+ *
+ * IVSC provides a privacy mode. When the privacy mode is turned
+ * on, camera sensor can't be used. This means that both IVSC and
+ * host Image Processing Unit(IPU) can't get image data. And when
+ * this mode is turned on, host Image Processing Unit(IPU) driver
+ * is informed via the registered callback, so that user can be
+ * notified.
+ *
+ * @param nr_of_lanes Number of data lanes used on the CSI-2 link
+ * @param link_freq Frequency of the CSI-2 link
+ * @param callback The pointer of privacy callback function
+ * @param context Privacy callback function runtime context
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context);
+
+/*
+ * @brief Release camera sensor ownership and stop the CSI-2
+ * link between host and IVSC
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_release_camera(void);
+
+#else
+static inline
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	return 0;
+}
+
+static inline int ivsc_release_camera(void)
+{
+	return 0;
+}
+
+#endif
+
 #endif
#7 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Quoted 23 lines · Hans de Goede

Hi Sakari,

Quoted 20 lines · Sakari Ailus

Hi Wentong,

Thanks for the patchset.

Quoted 14 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 8 lines
Quoted 4 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Agreed.

Quoted 188 lines
Quoted 188 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 172 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0
new file mode 100644
index 000000000000..12996b587639
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+
+#include "mei_ace.h"
+#include "mei_csi.h"
+
+/* lock for ivsc APIs */
+static DEFINE_MUTEX(ivsc_mutex);
+
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;
+
+	mutex_unlock(&ivsc_mutex);
+
+	return 0;
+
+release_csi:
+	csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+
+release_camera:
+	ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
+
+int ivsc_release_camera(void)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch CSI-2 link to IVSC */
+	ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+	if (ret)
+		goto error;
+
+	/* switch camera sensor ownership to IVSC */
+	ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_release_camera);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_SOFTDEP("pre: mei_csi mei_ace");
+MODULE_DESCRIPTION("IVSC interface");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS(IVSC);
include/linux/ivsc.h +55 -0
index 6572ca4f340c..bc9006cd6efc 100644
@@ -16,4 +16,59 @@ enum ivsc_privacy_status {
 	IVSC_PRIVACY_MAX,
 };
 
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief Acquire camera sensor ownership to host and setup
+ * the CSI-2 link between host and IVSC
+ *
+ * IVSC provides a privacy mode. When the privacy mode is turned
+ * on, camera sensor can't be used. This means that both IVSC and
+ * host Image Processing Unit(IPU) can't get image data. And when
+ * this mode is turned on, host Image Processing Unit(IPU) driver
+ * is informed via the registered callback, so that user can be
+ * notified.
+ *
+ * @param nr_of_lanes Number of data lanes used on the CSI-2 link
+ * @param link_freq Frequency of the CSI-2 link
+ * @param callback The pointer of privacy callback function
+ * @param context Privacy callback function runtime context
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context);
+
+/*
+ * @brief Release camera sensor ownership and stop the CSI-2
+ * link between host and IVSC
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_release_camera(void);
+
+#else
+static inline
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	return 0;
+}
+
+static inline int ivsc_release_camera(void)
+{
+	return 0;
+}
+
+#endif
+
 #endif
Signature · 3 lines
Regards,

Laurent Pinchart
#8 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi,

Thanks for your review

Quoted 32 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Wednesday, February 15, 2023 5:46 PM

Quoted 27 lines · Hans de Goede

Hi Sakari,

Quoted 24 lines · Sakari Ailus

Hi Wentong,

Thanks for the patchset.

Quoted 9 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image

Quoted 7 lines

pipeline.

Quoted 4 lines

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

Quoted 192 lines
Quoted 9 lines
Quoted 9 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

Quoted 3 lines
Quoted 3 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the

moment.

Quoted 103 lines
Quoted 103 lines

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 98 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)  create mode 100644

drivers/media/pci/intel/ivsc/ivsc.c

drivers/media/pci/intel/ivsc/Makefile +1 -0
index de0a425c22c2..b8b6fc1083be 100644
@@ -4,3 +4,4 @@
 
 obj-$(CONFIG_INTEL_VSC) += mei_csi.o
 obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +72 -0
new file mode 100644
index 000000000000..12996b587639
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface  */
+
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+
+#include "mei_ace.h"
+#include "mei_csi.h"
+
+/* lock for ivsc APIs */
+static DEFINE_MUTEX(ivsc_mutex);
+
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;
+
+	mutex_unlock(&ivsc_mutex);
+
+	return 0;
+
+release_csi:
+	csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+
+release_camera:
+	ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
+
+int ivsc_release_camera(void)
+{
+	int ret;
+
+	mutex_lock(&ivsc_mutex);
+
+	/* switch CSI-2 link to IVSC */
+	ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+	if (ret)
+		goto error;
+
+	/* switch camera sensor ownership to IVSC */
+	ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+	mutex_unlock(&ivsc_mutex);
Quoted 8 lines
Quoted 8 lines

drivers/media/pci/intel/ivsc/ivsc.c line 81

Quoted 8 lines
drivers/media/pci/intel/ivsc/ivsc.c new file +8 -0
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_release_camera);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_SOFTDEP("pre: mei_csi mei_ace");

MODULE_DESCRIPTION("IVSC

Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
drivers/media/pci/intel/ivsc/ivsc.c new file +1 -0
+interface"); MODULE_LICENSE("GPL"); MODULE_IMPORT_NS(IVSC);
include/linux/ivsc.h index renamed from include/linux/ivsc.h +0 -0

6572ca4f340c..bc9006cd6efc 100644

include/linux/ivsc.h +45 -0
@@ -16,4 +16,59 @@ enum ivsc_privacy_status {
 	IVSC_PRIVACY_MAX,
 };
 
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief Acquire camera sensor ownership to host and setup
+ * the CSI-2 link between host and IVSC
+ *
+ * IVSC provides a privacy mode. When the privacy mode is turned
+ * on, camera sensor can't be used. This means that both IVSC and
+ * host Image Processing Unit(IPU) can't get image data. And when
+ * this mode is turned on, host Image Processing Unit(IPU) driver
+ * is informed via the registered callback, so that user can be
+ * notified.
+ *
+ * @param nr_of_lanes Number of data lanes used on the CSI-2 link
+ * @param link_freq Frequency of the CSI-2 link
+ * @param callback The pointer of privacy callback function
+ * @param context Privacy callback function runtime context
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors  */ int
+ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context);
+
+/*
+ * @brief Release camera sensor ownership and stop the CSI-2
+ * link between host and IVSC
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors  */ int
+ivsc_release_camera(void);
+
+#else
+static inline
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+			void (*callback)(void *, enum ivsc_privacy_status),
+			void *context)
+{
+	return 0;
+}
Quoted 8 lines
Quoted 8 lines

include/linux/ivsc.h enum ivsc_privacy_status { · line 74

Quoted 8 lines
include/linux/ivsc.h +7 -0
+
+static inline int ivsc_release_camera(void) {
+	return 0;
+}
+
+#endif
+
 #endif

--
Regards,

Laurent Pinchart

#9 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Wentong,

Quoted 29 lines · Wentong
Quoted 28 lines · Laurent Pinchart
Quoted 26 lines · Hans de Goede
Quoted 16 lines · Sakari Ailus
Quoted 9 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 7 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84 +++++++++++++++++++++++++++
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)  create mode 100644

drivers/media/pci/intel/ivsc/ivsc.c

[snip]

Signature · 3 lines
Regards,

Laurent Pinchart
#10 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Quoted 63 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

Quoted 6 lines

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?

Thanks,
Wentong

Quoted 33 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation

gets called before streaming is started.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 4 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84

+++++++++++++++++++++++++++

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)  create mode 100644

drivers/media/pci/intel/ivsc/ivsc.c

[snip]

--
Regards,

Laurent Pinchart

maintainer
#11 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Wentong,

Quoted 66 lines · Wentong
Quoted 66 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

Quoted 8 lines
Quoted 3 lines

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?

Please wrap the lines at 74 characters or so when replying.

Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?

Signature · 3 lines
Regards,

Sakari Ailus
#12 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Quoted 81 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PM

Hi Wentong,

Quoted 10 lines · Wentong
Quoted 10 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 1 line · Wentong

On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Hans de Goede
Quoted 7 lines · Sakari Ailus
Quoted 6 lines · Wentong Wu
Quoted 5 lines

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to

ISH.

Quoted 53 lines
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines

On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware
architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Quoted 6 lines
Quoted 4 lines

Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?

Please wrap the lines at 74 characters or so when replying.

Thanks, I will.

Quoted 3 lines

Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?

Yes

Quoted 5 lines

--
Regards,

Sakari Ailus

#13 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi,

Quoted 88 lines · Wentong
Quoted 81 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PM

Hi Wentong,

Quoted 10 lines · Wentong
Quoted 10 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 1 line · Wentong

On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Hans de Goede
Quoted 7 lines · Sakari Ailus
Quoted 6 lines · Wentong Wu
Quoted 5 lines

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to

ISH.

Quoted 53 lines
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines

On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware
architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Quoted 6 lines
Quoted 4 lines

Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?

Please wrap the lines at 74 characters or so when replying.

Quoted 7 lines

Thanks, I will.

Quoted 3 lines

Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?

Yes

Hmm, that is going to be a problem since we really want to have independent driver for the 2 which are not aware of each other.

I think that maybe we can model this part of the ivsc functionality as an i2c-mux. But then we will need to somehow change the parent of the i2c device for the sensor to the output of this mux ...

I guess this means adding some code (some hack likely) to drivers/platform/x86/intel/int3472/discrete.c which' proe function is already guaranteed to run before the sensor's i2c-client gets instantiated, because of the ACPI _DEP on the INT3472 ACPI device in the DSDT.

This is going to be a tricky problem to tackle.

Regards,

Hans

#14 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Quoted 98 lines · Hans de Goede
Quoted 81 lines · Wentong
Quoted 81 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PM

Hi Wentong,

Quoted 73 lines · Wentong
Quoted 62 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI
CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware
architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

Quoted 6 lines

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?

Quoted 8 lines
Quoted 1 line

Please wrap the lines at 74 characters or so when replying.

Thanks, I will.

Quoted 2 lines

Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?

Yes

Hmm, that is going to be a problem since we really want to have independent driver for the 2 which are not aware of each other.

I think that maybe we can model this part of the ivsc functionality as an i2c-mux. But then we will need to somehow change the parent of the i2c device for the sensor to the output of this mux ...

Quoted 7 lines

I guess this means adding some code (some hack likely) to drivers/platform/x86/intel/int3472/discrete.c which' proe function is already guaranteed to run before the sensor's i2c-client gets instantiated, because of the ACPI _DEP on the INT3472 ACPI device in the DSDT.

This is going to be a tricky problem to tackle.

From a framework point of view I would also recommend i2c-mux, but when it comes to the ACPI integration... I can feel your pain and I'm sorry about that :-(

Signature · 3 lines
Regards,

Laurent Pinchart
#15 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

In reply to Laurent Pinchart

Hi Laurent,

Quoted 61 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PM

Hi Wentong,

Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

Quoted 5 lines

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.

Quoted 6 lines
Quoted 1 line

Agreed.

Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?

This is a tentative proposal, as I'm not very familiar with the hardware architecture:

- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.

- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.

- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.

Quoted 5 lines

- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.

- The driver should be validated with v4l2-compliance, part of v4l-utils.

Thanks, one more question, because ivsc just routes the received data from sensor and no format convert, how we handle the get_fmt and set_fmt callback for ivsc?

Thanks
Wentong

Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation

gets called before streaming is started.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The privacy status can be a V4L2 control.

Also cc Hans.

Quoted 4 lines
---
 drivers/media/pci/intel/ivsc/Makefile |  1 +
 drivers/media/pci/intel/ivsc/ivsc.c   | 84

+++++++++++++++++++++++++++

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
 include/linux/ivsc.h                  | 55 ++++++++++++++++++
 3 files changed, 140 insertions(+)  create mode 100644

drivers/media/pci/intel/ivsc/ivsc.c

[snip]

--
Regards,

Laurent Pinchart

#16 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

In reply to Sakari Ailus

Hi Sakari,

few questions as switching to v4l2 sub-dev framework.

Quoted 35 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AM

Hi Wentong,

Thanks for the patchset.

Quoted 8 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

Quoted 5 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

This should be a control or event? If control, how user-space handle privacy stuff?

For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?

Thanks,
Wentong

Quoted 4 lines

Also cc Hans.

Quoted 1 line
maintainer
#17 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Wentong,

Quoted 39 lines · Wentong

Hi Sakari,

few questions as switching to v4l2 sub-dev framework.

Quoted 35 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AM

Hi Wentong,

Thanks for the patchset.

Quoted 8 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.

Quoted 5 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.

I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.

Quoted 8 lines
Quoted 5 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.

The privacy status can be a V4L2 control.

This should be a control or event? If control, how user-space handle privacy stuff?

Changing control events generates events for the user space.

<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>

Quoted 5 lines

For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?

You should obtain these using the V4L2 fwnode interface. Please see e.g. drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers a V4L2 async sub-device and a V4L2 async notifier.

Signature · 3 lines
Kind regards,

Sakari Ailus
#18 lore raw

RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Quoted 57 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, February 28, 2023 4:24 PM

Hi Wentong,

Quoted 20 lines · Wentong

Hi Sakari,

few questions as switching to v4l2 sub-dev framework.

Quoted 16 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AM

Hi Wentong,

Thanks for the patchset.

Quoted 7 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are

defined to sync the ownership.

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image

pipeline.

Quoted 8 lines
Quoted 8 lines

I suppose the intended use cases assume a single instance of IVSC (as well as
MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream()

operation gets called before streaming is started.

Quoted 4 lines
Quoted 4 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the

moment.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The privacy status can be a V4L2 control.

This should be a control or event? If control, how user-space handle privacy stuff?

Changing control events generates events for the user space.

<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>

Ok, V4L2_EVENT_CTRL event which reports 'struct v4l2_event_ctrl' data to user space

Quoted 11 lines
Quoted 7 lines

For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and
privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?

You should obtain these using the V4L2 fwnode interface. Please see e.g. drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers a V4L2 async sub-device and a V4L2 async notifier.

Ok, something like v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_LINK_FREQ); is to get sensor's link frequency, and the code like 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config); num_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes;' is to get sensor's lane number.

BR,
Wentong

Quoted 5 lines

--
Kind regards,

Sakari Ailus

maintainer
#19 lore raw

Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc

Hi Wentong,

Quoted 80 lines · Wentong
Quoted 57 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, February 28, 2023 4:24 PM

Hi Wentong,

Quoted 20 lines · Wentong

Hi Sakari,

few questions as switching to v4l2 sub-dev framework.

Quoted 16 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AM

Hi Wentong,

Thanks for the patchset.

Quoted 7 lines · Wentong Wu

IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are

defined to sync the ownership.

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.

The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image

pipeline.

Quoted 8 lines
Quoted 8 lines

I suppose the intended use cases assume a single instance of IVSC (as well as
MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream()

operation gets called before streaming is started.

Quoted 4 lines
Quoted 4 lines

The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the

moment.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The privacy status can be a V4L2 control.

This should be a control or event? If control, how user-space handle privacy stuff?

Changing control events generates events for the user space.

<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>

Ok, V4L2_EVENT_CTRL event which reports 'struct v4l2_event_ctrl' data to user space

Quoted 11 lines
Quoted 7 lines

For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and
privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?

You should obtain these using the V4L2 fwnode interface. Please see e.g. drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers a V4L2 async sub-device and a V4L2 async notifier.

Quoted 5 lines

Ok, something like v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_LINK_FREQ); is to get sensor's link frequency, and the code like 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config); num_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes;' is to get sensor's lane number.

Please use the V4L2 fwnode interface instead.

The privacy control should only be needed in the user space.

Signature · 3 lines
Regards,

Sakari Ailus
#20 lore raw
In reply to Wentong Wu
Quoted 1 line · Wentong Wu

v2 without change log, why?

Quoted 169 lines

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
companion chip designed to provide secure and low power vision capability
to IA platforms. IVSC is available in existing commercial platforms from
multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link
and processes the image data with the embedded AI engine. The detected
events are sent over I2C to ISH (Intel Sensor Hub) for additional data
fusion from multiple sensors. The fusion results are used to implement
advanced use cases like:
 - Face detection to unlock screen
 - Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to
configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and
IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel
MEI (The Intel Management Engine) commands. These two hardware modules
have two different MEI UUIDs to enumerate. These hardware modules are:
 - ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's
ownership. This hardware module is used to set ownership of camera sensor.
 - CSI (Camera Serial Interface): This module is used to route camera sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't
get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules
inside IVSC.
 - mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
 - mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
---------------------------------------------------------------------
--------

Quoted 21 lines

Host
Processor
  |

       |
      -----------------       -----------------       ------------- --     |

      |               |       |               |       |            
| I2C |
      |      IPU      |       |      ISH      |       |camera
driver|--|  |
      |               |       |               |       |            
|  |  |
      -----------------       -----------------       -------------
--  |  |
              |                       |                     
|         |  |
              |                       |               -------------
--  |  |
              |                       |               |            
|  |  |
Quoted 8 lines
              |                       |               | IVSC driver
|  |  |
              |                       |               |            
|  |  |
              |                       |               -------------
--  |  |
              |                       |                     
|         |  |

----------------|-----------------------|----------------------|----- ----|---

                | CSI                   | I2C                 
|SPI      |
                |                       |                     
|         |
----------------|-----------------------|----------------     
|         |
> IVSC          |                                       |     
> |         |
>               |                                       |     
> |         |
>       -----------------       -----------------       |     
> |         |
>       |               |       |               |       |     
> |         |
>       |      CSI      |       |      ACE      |       |------
> |         |
>       |               |       |               |      
> |                |
Quoted 4 lines

      -----------------       -----------------

|                |
              |                       | I2C          
|                |

----------------|-----------------------|---------------- |
                | CSI

|                                |
                |                      
|                                |
            --------------------------------
                             |
            |                              |
I2C                         |
            |         camera sensor        |-------------------------
----|
            |                              |

            --------------------------------

Wentong Wu (3):
  media: pci: intel: ivsc: Add CSI submodule
  media: pci: intel: ivsc: Add ACE submodule
  media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 + drivers/media/pci/intel/Makefile       |   2 + drivers/media/pci/intel/ivsc/Kconfig   |  12 + drivers/media/pci/intel/ivsc/Makefile  |   7 + drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472
+++++++++++++++++++++++++

 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)

 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h

#21 lore raw

Hi Srinivas,

Thanks for your review

Quoted 8 lines

-----Original Message-----
From: Pandruvada, Srinivas <[email protected]>
Sent: Monday, February 13, 2023 11:21 AM

Quoted 1 line · Wentong Wu

v2 without change log, why?

This addressed 'mei_csi.c:(.text+0x1a6): undefined reference to __udivdi3' issue on i386 arch. I will add change log on following version.

Quoted 153 lines
Quoted 153 lines

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
 - Face detection to unlock screen
 - Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules
are:
 - ACE (Algorithm Context Engine): This module is for algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
 - CSI (Camera Serial Interface): This module is used to route camera sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
 - mei_csi: MEI client driver to send commands and receive notifications from CSI module.
 - mei_ace: MEI client driver to send commands and get status from ACE module.
Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
---------------------------------------------------------------------
--------

Quoted 21 lines

Host
Processor
  |

       |
      -----------------       -----------------       ------------- --     |

      |               |       |               |       |
| I2C |
      |      IPU      |       |      ISH      |       |camera
driver|--|  |
      |               |       |               |       |
|  |  |
      -----------------       -----------------       -------------
--  |  |
              |                       |
|         |  |
              |                       |               -------------
--  |  |
              |                       |               |
|  |  |
Quoted 8 lines
              |                       |               | IVSC driver
|  |  |
              |                       |               |
|  |  |
              |                       |               -------------
--  |  |
              |                       |
|         |  |

----------------|-----------------------|----------------------|----- ----|---

                | CSI                   | I2C
|SPI      |
                |                       |
|         |
----------------|-----------------------|----------------
|         |
> IVSC          |                                       |
> |         |
>               |                                       |
> |         |
>       -----------------       -----------------       |
> |         |
>       |               |       |               |       |
> |         |
>       |      CSI      |       |      ACE      |       |------
> |         |
>       |               |       |               |
> |                |
Quoted 4 lines

      -----------------       -----------------

|                |
              |                       | I2C
|                |

----------------|-----------------------|---------------- |
                | CSI

|                                |
                |
|                                |
            --------------------------------
                             |
            |                              | I2C
|
            |         camera sensor        |-------------------------
----|
            |                              |

            --------------------------------

Wentong Wu (3):
  media: pci: intel: ivsc: Add CSI submodule
  media: pci: intel: ivsc: Add ACE submodule
  media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 + drivers/media/pci/intel/Makefile       |   2 + drivers/media/pci/intel/ivsc/Kconfig   |  12 + drivers/media/pci/intel/ivsc/Makefile  |   7 + drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472
+++++++++++++++++++++++++

 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)

 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h

A message that is not in the archive (sent off-list or not yet received): [email protected]

#23 lore raw

RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule

Hi Hillf Danton,

Thanks for your review

Quoted 5 lines

-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Monday, February 13, 2023 11:42 AM

On Mon, 13 Feb 2023 10:23:45 +0800 Wentong Wu <[email protected]>

Quoted 9 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+/* send a command to firmware and mutex must be held by caller */
+static int mei_csi_send(u8 *buf, size_t len) {
+	struct csi_cmd *cmd = (struct csi_cmd *)buf;
+	int ret;
+
+	reinit_completion(&csi->cmd_completion);

Could you specify why reinit is needed here?

This allows new command(e.g. 'get status' command to check current firmware status) to be downloaded to firmware for debugging purpose in case no response for current command though I have never saw this happen.

Quoted 1 line

What is hurt without it?

No hurt for current implementation. I can remove it.

Quoted 2 lines

Same question for the reinit in 2/3.

same ack for 2/3

Quoted 14 lines
Quoted 14 lines
(quoted diff) +13 -0
+
+	ret = mei_cldev_send(csi->cldev, buf, len);
+	if (ret < 0)
+		goto out;
+
+	ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
+						   CSI_CMD_TIMEOUT);
+	if (ret < 0) {
+		goto out;
+	} else if (!ret) {
+		ret = -ETIMEDOUT;
+		goto out;
+	}

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule

Quoted 13 lines

-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Tuesday, February 14, 2023 4:36 PM

On Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>

Quoted 6 lines

This allows new command(e.g. 'get status' command to check current firmware=
status) to be downloaded to firmware for debugging purpose in case no
resp= onse for current command though I have never saw this happen. =20

Quoted 4 lines

Thanks for your specification.

Another question, is it likely for the wakeup to come after reinit because of for instance timeout on the waiter side?

Thanks, first I want to clarify why we have timeout here, in case firmware stuck somehow or IVSC device is removed suddenly, calling thread can't be wakeup if wait forever here.

And firmware gets high priority to handle command and response, also there is timeout mechanism inside firmware, I think there will be some problem with firmware if driver gets command response after 5s which will be captured by driver code(if (csi->cmd_response.cmd_id != cmd->cmd_id)) but I have never saw this happen.

BR,
Wentong

A message that is not in the archive (sent off-list or not yet received): [email protected]

#27 lore raw

RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule

Quoted 39 lines

-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Wednesday, February 15, 2023 7:54 AM

On Tue, 14 Feb 2023 12:40:12 +0000 Wentong Wu <[email protected]>

Quoted 10 lines
Quoted 10 lines

-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Tuesday, February 14, 2023 4:36 PM
=20
On Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>

Quoted 5 lines

This allows new command(e.g. 'get status' command to check current firmware=3D
status) to be downloaded to firmware for debugging purpose in
case no resp=3D onse for current command though I have never saw this

happen.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

=3D20

=20
Thanks for your specification.
=20
Another question, is it likely for the wakeup to come after reinit becaus=

Quoted 6 lines

e of for

Quoted 1 line

instance timeout on the waiter side?

Thanks, first I want to clarify why we have timeout here, in case firmware = stuck somehow or IVSC device is removed suddenly, calling thread can't be w= akeup if wait forever here.

Fair enough on the waiter side.

Quoted 6 lines

And firmware gets high priority to handle command and response, also there = is timeout mechanism inside firmware, I think there will be some problem wi= th firmware if driver gets command response after 5s which will be captured= by driver code(if (csi->cmd_response.cmd_id !=3D cmd->cmd_id)) but I have = never saw this happen.

Quoted 3 lines

The problem with firmware could be emulated by directly skipping the wait for completion and returning ETIMEDOUT, in order to see what will happen if wakeup comes later than timeout.

Thanks, but what do you mean "emulated"?

A message that is not in the archive (sent off-list or not yet received): [email protected]

#29 lore raw

RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule

Quoted 6 lines

-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Friday, February 17, 2023 2:28 PM

Quoted 6 lines

On Fri, 17 Feb 2023 05:52:38 +0000 Wentong Wu <[email protected]>

Quoted 3 lines

Thanks, but what do you mean "emulated"?=20

Construct a scenario that ensures wakeup comes late because of no wait for completion. And we can see what will happen with the current test cases.

Thanks, I will remove reinit_completion

Quoted 37 lines

Review

Quoted 11 lines

static int mei_csi_send(u8 *buf, size_t len)

(quoted diff) +9 -0
+{
+	struct csi_cmd *cmd = (struct csi_cmd *)buf;
+	int ret;
+
+	reinit_completion(&csi->cmd_completion);
+
+	ret = mei_cldev_send(csi->cldev, buf, len);
+	if (ret < 0)
+		goto out;
	return  -ETIMEDOUT;
Quoted 23 lines
(quoted diff) +23 -0
+
+	ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
+						   CSI_CMD_TIMEOUT);
+	if (ret < 0) {
+		goto out;
+	} else if (!ret) {
+		ret = -ETIMEDOUT;
+		goto out;
+	}
+
+	/* command response status */
+	ret = csi->cmd_response.status;
+	if (ret) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (csi->cmd_response.cmd_id != cmd->cmd_id)
+		ret = -EINVAL;
+
+out:
+	return ret;
+}
#30 lore raw
In reply to Wentong Wu

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 8 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 17 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 4 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 9 lines

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.

Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?

Quoted 61 lines

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
Signature · 3 lines
Regards,

Laurent Pinchart
#31 lore raw

Hi, Wentong,

Quoted 37 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 8 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 17 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 8 lines
Quoted 4 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.

Quoted 79 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 9 lines

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.

Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?

Quoted 61 lines

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
Signature · 2 lines
Best regards,
Bingbu Cao
maintainer
#32 lore raw

Hi Bingbu, Wentong,

Quoted 50 lines · Bingbu Cao

Hi, Wentong,

Quoted 37 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 8 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 17 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 8 lines
Quoted 4 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

Quoted 5 lines

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.

How does the privacy mode work, does it just pass zeroes (or other dummy data) towards the host or nothing?

A V4L2 control can be used for the purpose of passing the information to the user space at least.

Signature · 3 lines
Regards,

Sakari Ailus
#33 lore raw

Hi, Sakari,

Quoted 51 lines · Sakari Ailus

Hi Bingbu, Wentong,

Quoted 48 lines · Bingbu Cao

Hi, Wentong,

Quoted 44 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 8 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 17 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 4 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.

Wentong, is IVSC trying to cut off the stream and then notify user and IPU?

Quoted 14 lines
Quoted 7 lines
Quoted 1 line

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.

How does the privacy mode work, does it just pass zeroes (or other dummy data) towards the host or nothing?

A V4L2 control can be used for the purpose of passing the information to the user space at least.

Signature · 2 lines
Best regards,
Bingbu Cao
#34 lore raw
Quoted 64 lines

-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AM

Hi, Sakari,

Quoted 39 lines · Sakari Ailus

Hi Bingbu, Wentong,

Quoted 36 lines · Bingbu Cao

Hi, Wentong,

Quoted 32 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 9 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 11 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate.

These hardware modules are:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

Quoted 6 lines

As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.

Wentong, is IVSC trying to cut off the stream and then notify user and IPU?

yes

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?

IVSC has already defined privacy callback for host IPU/camera driver.

Quoted 7 lines
Quoted 2 lines

I'd have concern if IVSC driver

Quoted 1 line

need private callback to request back-end(e.g. ISP driver) to handle stream

cutting.

Quoted 4 lines

How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?

No data on CSI transmitter side

Quoted 3 lines
Quoted 3 lines

A V4L2 control can be used for the purpose of passing the information to the user space at least.

I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.

BR,
Wentong

Quoted 5 lines
Quoted 1 line

--
Best regards,
Bingbu Cao

maintainer
#35 lore raw

Hi Wentong,

Quoted 64 lines · Wentong
Quoted 64 lines

-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AM

Hi, Sakari,

Quoted 39 lines · Sakari Ailus

Hi Bingbu, Wentong,

Quoted 36 lines · Bingbu Cao

Hi, Wentong,

Quoted 32 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 9 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 11 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate.

These hardware modules are:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?

Quoted 8 lines
Quoted 6 lines

As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.

Wentong, is IVSC trying to cut off the stream and then notify user and IPU?

yes

Does the CSI-2 transmitter on IVCS go to some LP mode during this time, or does the receiver need to initialise the bus again when the stream resuems?

Quoted 12 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?

IVSC has already defined privacy callback for host IPU/camera driver.

Quoted 1 line
Quoted 1 line

I'd have concern if IVSC driver

Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line

need private callback to request back-end(e.g. ISP driver) to handle stream

cutting.

Quoted 4 lines

How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?

No data on CSI transmitter side

Can it stop in the middle of the frame? Or is it guaranteed to produce full frames (assuming the sensor does)?

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

A V4L2 control can be used for the purpose of passing the information to the user space at least.

I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.

Signature · 3 lines
Kind regards,

Sakari Ailus
#36 lore raw

Hi Sakari,

Quoted 83 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Friday, February 17, 2023 6:50 PM

Hi Wentong,

Quoted 39 lines · Wentong
Quoted 39 lines

-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AM

Hi, Sakari,

Quoted 30 lines · Sakari Ailus

Hi Bingbu, Wentong,

Quoted 27 lines · Bingbu Cao

Hi, Wentong,

Quoted 23 lines · Laurent Pinchart

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 11 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a
CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor
Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up system

Do you have plan to support these features in the ivsc driver in the future ?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces

provided by this IVSC driver.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to

enumerate.

Quoted 8 lines
Quoted 8 lines

These hardware modules are:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy

settings ?

Quoted 5 lines
Quoted 5 lines

As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final

Quoted 7 lines

solution on Linux so far I think.

Quoted 4 lines
Quoted 2 lines

Wentong, is IVSC trying to cut off the stream and then notify user and IPU?

yes

Does the CSI-2 transmitter on IVCS go to some LP mode during this time,

Yes, The low power mode is following the D-Phy spec.

Quoted 2 lines

or does
the receiver need to initialise the bus again when the stream resuems?

No, it's the same link.

Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?

IVSC has already defined privacy callback for host IPU/camera driver.

Quoted 5 lines
Quoted 3 lines

I'd have concern if IVSC driver

Quoted 2 lines

need private callback to request back-end(e.g. ISP driver) to handle stream

cutting.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?

No data on CSI transmitter side

Can it stop in the middle of the frame?

No,

Or is it guaranteed to produce full frames

Quoted 1 line

(assuming the sensor does)?

Yes, full frame will be guaranteed as camera sensor does.

Quoted 13 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

A V4L2 control can be used for the purpose of passing the information to the user space at least.

I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.

--
Kind regards,

Sakari Ailus

#37 lore raw
In reply to Laurent Pinchart
Quoted 23 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Wednesday, February 15, 2023 5:43 PM

Hello Wentong,

Quoted 5 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

Quoted 9 lines

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 1 line

Do you have plan to support these features in the ivsc driver in the future ?

Not sure, but the first step is to upstream this driver.

Quoted 20 lines
Quoted 8 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Quoted 4 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

modules are:

Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of

camera sensor.

Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 7 lines
Quoted 4 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?

No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.

Quoted 86 lines
Quoted 5 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 10 lines

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive
notifications from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.

Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?

Quoted 54 lines

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
----------------------------------------------------------------------
-------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472
+++++++++++++++++++++++++  drivers/media/pci/intel/ivsc/mei_ace.h |
36 ++  drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)
Quoted 8 lines
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h

--
Regards,

Laurent Pinchart

#38 lore raw

Hello Wentong,

Quoted 17 lines · Wentong
Quoted 16 lines · Laurent Pinchart
Quoted 14 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 3 lines
Quoted 1 line

Do you have plan to support these features in the ivsc driver in the future ?

Not sure, but the first step is to upstream this driver.

Sure, no problem.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

Quoted 5 lines
Quoted 2 lines

How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?

No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.

But how can I be sure that the IVSC will not use the camera behind my back, if it's all controlled through a closed-source firmware ?

Quoted 76 lines
Quoted 76 lines
Quoted 11 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.

Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?

Quoted 61 lines

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++
 drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
 drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)
 create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h
Signature · 3 lines
Regards,

Laurent Pinchart
#39 lore raw
Quoted 57 lines

-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:12 PM

Hello Wentong,

Quoted 17 lines · Wentong
Quoted 16 lines · Laurent Pinchart
Quoted 14 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 3 lines
Quoted 1 line

Do you have plan to support these features in the ivsc driver in the future ?

Not sure, but the first step is to upstream this driver.

Sure, no problem.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by

this IVSC driver.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?

No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.

But how can I be sure that the IVSC will not use the camera behind my back, if it's all controlled through a closed-source firmware ?

Actually I don't know how to answer your question, but this is guaranteed though we have no plan to open-source the firmware.

Quoted 86 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications

from CSI module.

Quoted 1 line
Quoted 1 line
Quoted 1 line

- mei_ace: MEI client driver to send commands and get status from ACE

module.

Quoted 61 lines
Quoted 61 lines
Quoted 3 lines

Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.

Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?

Quoted 54 lines

Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
------------------------------------------------------------------
-----------

| Host Processor                                                            |
|                                                                           |
|       -----------------       -----------------       ---------------     |
|       |               |       |               |       |             | I2C |
|       |      IPU      |       |      ISH      |       |camera driver|--|  |
|       |               |       |               |       |             |  |  |
|       -----------------       -----------------       ---------------  |  |
|               |                       |                      |         |  |
|               |                       |               ---------------  |  |
|               |                       |               |             |  |  |
|               |                       |               | IVSC driver |  |  |
|               |                       |               |             |  |  |
|               |                       |               ---------------  |  |
|               |                       |                      |         |  |
----------------|-----------------------|----------------------|---------|---
                | CSI                   | I2C                  |SPI      |
                |                       |                      |         |
----------------|-----------------------|----------------      |         |
| IVSC          |                                       |      |         |
|               |                                       |      |         |
|       -----------------       -----------------       |      |         |
|       |               |       |               |       |      |         |
|       |      CSI      |       |      ACE      |       |------|         |
|       |               |       |               |       |                |
|       -----------------       -----------------       |                |
|               |                       | I2C           |                |
----------------|-----------------------|----------------                |
                | CSI                   |                                |
                |                       |                                |
            --------------------------------                             |
            |                              | I2C                         |
            |         camera sensor        |-----------------------------|
            |                              |
            --------------------------------

Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc

 drivers/media/pci/Kconfig              |   1 +
 drivers/media/pci/intel/Makefile       |   2 +
 drivers/media/pci/intel/ivsc/Kconfig   |  12 +
 drivers/media/pci/intel/ivsc/Makefile  |   7 +
 drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
 drivers/media/pci/intel/ivsc/mei_ace.c | 472
+++++++++++++++++++++++++  drivers/media/pci/intel/ivsc/mei_ace.h
|> 36 ++  drivers/media/pci/intel/ivsc/mei_csi.c | 342
++++++++++++++++++  drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
 include/linux/ivsc.h                   |  74 ++++
 10 files changed, 1090 insertions(+)  create mode 100644
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/media/pci/intel/ivsc/Kconfig

 create mode 100644 drivers/media/pci/intel/ivsc/Makefile
 create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
 create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
 create mode 100644 include/linux/ivsc.h

--
Regards,

Laurent Pinchart

maintainer
#40 lore raw
In reply to Wentong Wu

Hi Wentong,

Quoted 31 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 8 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the following:

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE, the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).

- Device links need to be set up via IPU bridge which now is called CIO2 bridge (cio2-bridge.c).

Any questions, comments?

Signature · 3 lines
Kind regards,

Sakari Ailus
#41 lore raw

Hi,

Quoted 49 lines · Sakari Ailus

Hi Wentong,

Quoted 31 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

Quoted 8 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the following:

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

Quoted 8 lines

- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE, the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.

And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c-transfer.

Regards,

Hans

Quoted 9 lines

- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).

- Device links need to be set up via IPU bridge which now is called CIO2 bridge (cio2-bridge.c).

Any questions, comments?

#42 lore raw
Quoted 81 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 29 lines · Sakari Ailus

Hi Wentong,

Quoted 26 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

modules are:

Quoted 3 lines
Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 16 lines
Quoted 11 lines

- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the
following:

Quoted 7 lines

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 4 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.

Quoted 8 lines

And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.

Thanks
Wentong

Quoted 16 lines

Regards,

Hans

Quoted 9 lines

- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).

- Device links need to be set up via IPU bridge which now is called CIO2 bridge (cio2-bridge.c).

Any questions, comments?

maintainer
#43 lore raw

Hi Wentong,

Quoted 85 lines · Wentong
Quoted 85 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 29 lines · Sakari Ailus

Hi Wentong,

Quoted 26 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

modules are:

Quoted 3 lines
Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 16 lines
Quoted 11 lines

- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the
following:

Quoted 7 lines

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 4 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.

And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.

Quoted 7 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.

What do you mean? The supplier devices are resumed before the driver's probe is called.

Signature · 3 lines
Regards,

Sakari Ailus
#44 lore raw
Quoted 108 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 27 lines · Wentong
Quoted 27 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 18 lines · Sakari Ailus

Hi Wentong,

Quoted 15 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

Quoted 8 lines
Quoted 8 lines

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

the

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 4 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

Quoted 8 lines
Quoted 8 lines

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 5 lines
Quoted 5 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 6 lines
Quoted 5 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Quoted 5 lines
Quoted 2 lines

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.

What do you mean? The supplier devices are resumed before the driver's probe is called.

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Quoted 5 lines

--
Regards,

Sakari Ailus

#45 lore raw

Hi,

Quoted 108 lines · Wentong
Quoted 108 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 27 lines · Wentong
Quoted 27 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 18 lines · Sakari Ailus

Hi Wentong,

Quoted 15 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

Quoted 8 lines
Quoted 8 lines

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

the

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 4 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

Quoted 8 lines
Quoted 8 lines

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 5 lines
Quoted 5 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 6 lines
Quoted 5 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.

What do you mean? The supplier devices are resumed before the driver's probe is called.

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#46 lore raw
Quoted 128 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 71 lines · Wentong
Quoted 71 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 27 lines · Wentong
Quoted 27 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 18 lines · Sakari Ailus

Hi Wentong,

Quoted 15 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

Quoted 8 lines
Quoted 8 lines

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

Quoted 8 lines
Quoted 8 lines

the

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 3 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32 flags)

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct mei_device).

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

BR,
Wentong

Quoted 25 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#47 lore raw

Hi,

Quoted 131 lines · Wentong
Quoted 131 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 71 lines · Wentong
Quoted 71 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 27 lines · Wentong
Quoted 27 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 18 lines · Sakari Ailus

Hi Wentong,

Quoted 15 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

Quoted 8 lines
Quoted 8 lines

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

Quoted 8 lines
Quoted 8 lines

the

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 3 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Quoted 11 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32 flags)

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct mei_device).

Quoted 3 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace to show up).

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting registered. The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with atomisp2 there is going to be a need to add a single line to probe() like this:

	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :

https://github.com/jwrdegoede/linux-sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 25 lines
Quoted 24 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#48 lore raw

Hi

Quoted 150 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 90 lines · Wentong
Quoted 90 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 71 lines · Wentong
Quoted 71 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 27 lines · Wentong
Quoted 27 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 18 lines · Sakari Ailus

Hi Wentong,

Quoted 15 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up system

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

Quoted 8 lines
Quoted 8 lines

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 2 lines
Quoted 2 lines

- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

the

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 2 lines

the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks to
transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.

Quoted 6 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 91 lines
Quoted 14 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct mei_device).

Quoted 3 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace to show up).

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting registered. The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 7 lines
Quoted 7 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#49 lore raw

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 6 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power. And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC
2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead

But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 282 lines
Quoted 282 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 49 lines · Wentong

Hi

Quoted 47 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 40 lines · Wentong
Quoted 40 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 31 lines · Wentong
Quoted 31 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 22 lines · Wentong
Quoted 22 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 13 lines · Sakari Ailus

Hi Wentong,

Quoted 10 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple

sensors.

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up system

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the
CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided

by this IVSC driver.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

modules are:

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership

of camera sensor.

Quoted 3 lines
Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be

notified.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the
following:

- The IVSC sub-device implements a control for privacy

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

(V4L2_CID_PRIVACY

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that

purpose

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 1 line

the supplier devices will be PM runtime resumed before the

consumer

Quoted 52 lines
Quoted 52 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on host
side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks to
transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.

Quoted 29 lines
Quoted 2 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 22 lines
Quoted 14 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct mei_device).

Quoted 3 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 3 lines
Quoted 3 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 67 lines
Quoted 66 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 7 lines
Quoted 7 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#50 lore raw
Quoted 19 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 5 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 7 lines
Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead

Below is your dmesg log, the required SPI controller for IVSC isn't here.

[   35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[   35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[   35.538621] ljca 2-6:1.0: LJCA USB device init success
[   35.538776] usbcore: registered new interface driver ljca

Also I checked your SSDT, there is no IVSC device and the sensor device depends on INT3472 instead of IVSC device as on my setup.

Quoted 4 lines

But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.

Quoted 8 lines

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor start probe?

BR,
Wentong

Quoted 309 lines

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 83 lines
Quoted 83 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 50 lines · Wentong

Hi

Quoted 48 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 41 lines · Wentong
Quoted 41 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 32 lines · Wentong
Quoted 32 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 23 lines · Wentong
Quoted 23 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 14 lines · Sakari Ailus

Hi Wentong,

Quoted 11 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiple

sensors.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up system

Since the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces provided

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

by this IVSC driver.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The Intel

Management Engine) commands.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

These two hardware modules have two different MEI UUIDs to
enumerate. These hardware

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownership

of camera sensor.

Quoted 3 lines
Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can be

notified.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the
following:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

sensor via I²C. The IVSC ownership control needs to be in the

right

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

setting for this to work, and device links can be used for
that

purpose

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 1 line

the supplier devices will be PM runtime resumed before the

consumer

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on
host side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks
to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSC

enabled.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't

work.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 6 lines
Quoted 6 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct

mei_device).

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 4 lines
Quoted 4 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low

or active high.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted

upstream) :

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 8 lines
Quoted 8 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#51 lore raw

Hi,

Quoted 29 lines · Wentong
Quoted 19 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 5 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 7 lines
Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead

Below is your dmesg log, the required SPI controller for IVSC isn't here.

Quoted 7 lines
[   35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[   35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[   35.538621] ljca 2-6:1.0: LJCA USB device init success
[   35.538776] usbcore: registered new interface driver ljca

Also I checked your SSDT, there is no IVSC device and the sensor device depends on INT3472 instead of IVSC device as on my setup.

Ack.

Quoted 12 lines
Quoted 11 lines

But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Quoted 2 lines

Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor start probe?

Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?

If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().

Regards,

Hans

Quoted 309 lines
Quoted 308 lines

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 83 lines
Quoted 83 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 50 lines · Wentong

Hi

Quoted 48 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 41 lines · Wentong
Quoted 41 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 32 lines · Wentong
Quoted 32 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 23 lines · Wentong
Quoted 23 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 14 lines · Sakari Ailus

Hi Wentong,

On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiple

sensors.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up system

Since the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces provided

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

by this IVSC driver.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The Intel

Management Engine) commands.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

These two hardware modules have two different MEI UUIDs to
enumerate. These hardware

modules are:

Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownership

of camera sensor.

Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can be

notified.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can
suggest the
following:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC
before accessing

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

sensor via I²C. The IVSC ownership control needs to be in the

right

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

setting for this to work, and device links can be used for
that

purpose

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 1 line

the supplier devices will be PM runtime resumed before the

consumer

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on
host side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks
to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSC

enabled.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't

work.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 6 lines
Quoted 6 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct

mei_device).

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 4 lines
Quoted 4 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low

or active high.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted

upstream) :

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 8 lines
Quoted 8 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#52 lore raw
Quoted 64 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PM

Hi,

Quoted 30 lines · Wentong
Quoted 20 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 6 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 7 lines
Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead

Below is your dmesg log, the required SPI controller for IVSC isn't here.

Quoted 8 lines
[   35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[   35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[   35.538621] ljca 2-6:1.0: LJCA USB device init success
[   35.538776] usbcore: registered new interface driver ljca

Also I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.

Ack.

Quoted 1 line
Quoted 1 line

But you might very well be right, that would also explain the "intel vsc not

ready"

Quoted 5 lines
Quoted 5 lines

messages in dmesg.

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor

drivers already try to use.

Quoted 6 lines
Quoted 5 lines

The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Quoted 5 lines
Quoted 2 lines

Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor

start probe?

Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?

Yes,

Quoted 3 lines

If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().

But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is hard to achieve this.
+ /* switch camera sensor ownership to host */
+ ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+ if (ret)
+ goto error;
+
+ /* switch CSI-2 link to host */
+ ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+ if (ret)
+ goto release_camera;
+
+ /* configure CSI-2 link */
+ ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+ if (ret)
+ goto release_csi;

BR,
Wentong

Quoted 327 lines

Regards,

Hans

Quoted 72 lines
Quoted 72 lines

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 58 lines
Quoted 58 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 39 lines · Wentong

Hi

Quoted 37 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 30 lines · Wentong
Quoted 30 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 21 lines · Wentong
Quoted 21 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 12 lines · Wentong
Quoted 12 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

On 3/1/23 11:34, Sakari Ailus wrote:
Hi Wentong,

On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure
and low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiple

sensors.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The fusion results are used to implement advanced use cases

like:

Quoted 169 lines
Quoted 169 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up system

Since the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces provided

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

by this IVSC driver.

Quoted 4 lines
Quoted 4 lines

Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The Intel

Management Engine) commands.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines

These two hardware modules have two different MEI UUIDs
to enumerate. These hardware
modules are:
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also
ACE module controls camera sensor's ownership. This
hardware module is used to set ownership
of camera sensor.
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for
IPU driver and

application.

Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can be

notified.

Quoted 10 lines
Quoted 10 lines

In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and
then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I
can suggest the
following:

- The IVSC sub-device implements a control for privacy

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

(V4L2_CID_PRIVACY

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC
before accessing

the

Quoted 2 lines
Quoted 2 lines

sensor via I²C. The IVSC ownership control needs to be
in the

right

Quoted 50 lines
Quoted 50 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

setting for this to work, and device links can be used
for that

purpose

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

(see device_link_add()). With DL_FLAG_PM_RUNTIME and
DL_FLAG_RPM_ACTIVE,
the supplier devices will be PM runtime resumed before
the

consumer

Quoted 42 lines
Quoted 42 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on
host side and

has

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks
to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines

ownership.

Interesting proposal to use device-links + runtime-pm for
this instead of modelling this as an i2c-mux. FWIW I'm fine
with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and
the problem is that we are not really muxing but want to
turn on/off control and AFAIK the i2c-mux framework simply
leaves the mux muxed to the last used i2c-chain, so control
will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there
already is some release
callback) then the downside becomes that the i2c-mux core
code operates at the i2c transfer level. So each i2c
read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a
much better fit because then we request control once on
probe / stream-on and release it once we are fully done,
rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 6 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSC

enabled.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't

work.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 6 lines
Quoted 6 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct

mei_device).

Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 4 lines
Quoted 4 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 4 lines
Quoted 4 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low

or active high.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like

this:

Quoted 40 lines
Quoted 40 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted

upstream) :

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 15 lines
Quoted 15 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 4 lines
Quoted 4 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the

sensor.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Regards,

Hans

maintainer
#53 lore raw

Hi Wentong,

Quoted 70 lines · Wentong
Quoted 64 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PM

Hi,

Quoted 30 lines · Wentong
Quoted 20 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 6 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 7 lines
Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead

Below is your dmesg log, the required SPI controller for IVSC isn't here.

Quoted 8 lines
[   35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[   35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[   35.538621] ljca 2-6:1.0: LJCA USB device init success
[   35.538776] usbcore: registered new interface driver ljca

Also I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.

Ack.

Quoted 1 line
Quoted 1 line

But you might very well be right, that would also explain the "intel vsc not

ready"

Quoted 5 lines
Quoted 5 lines

messages in dmesg.

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor

drivers already try to use.

Quoted 6 lines
Quoted 5 lines

The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Quoted 5 lines
Quoted 2 lines

Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor

start probe?

Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?

Quoted 8 lines

Yes,

Quoted 3 lines

If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().

But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is hard to achieve this.

How so?

I don't insist on the runtime PM based solution but I'd rather not have changes to virtually all sensor drivers --- this is an external chip to them.

Quoted 14 lines
(quoted diff) +14 -0
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;
Signature · 3 lines
Kind regards,

Sakari Ailus
#54 lore raw
Quoted 88 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Friday, March 17, 2023 4:59 PM

Hi Wentong,

Quoted 24 lines · Wentong
Quoted 24 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PM

Hi,

Quoted 15 lines · Wentong
Quoted 15 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large
attachments>

Hi Wentong,

Quoted 2 lines · Wentong

Hi Hans,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 1 line

And I will double confirm with people who know dell product well

tomorrow.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead

Below is your dmesg log, the required SPI controller for IVSC isn't here.

[   35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20

ack:0

Quoted 36 lines
Quoted 26 lines
Quoted 7 lines
[   35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[   35.538621] ljca 2-6:1.0: LJCA USB device init success
[   35.538776] usbcore: registered new interface driver ljca

Also I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.

Ack.

Quoted 2 lines
Quoted 2 lines

But you might very well be right, that would also explain the "intel vsc not

ready"

Quoted 6 lines
Quoted 6 lines

messages in dmesg.

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor

drivers already try to use.

Quoted 6 lines
Quoted 5 lines

The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Quoted 6 lines
Quoted 3 lines

Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor

start probe?

Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?

Yes,

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().

But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is

hard to achieve this.

How so?

But we have to find a way to download commands to firmware following below sequence before writing camera sensor's registers to start camera sensor's steam.

BR,
Wentong

Quoted 23 lines

I don't insist on the runtime PM based solution but I'd rather not have changes to virtually all sensor drivers --- this is an external chip to them.

Quoted 4 lines
(quoted diff) +4 -0
+	/* switch camera sensor ownership to host */
+	ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+	if (ret)
+		goto error;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	/* switch CSI-2 link to host */
+	ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+	if (ret)
+		goto release_camera;
+
+	/* configure CSI-2 link */
+	ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+	if (ret)
+		goto release_csi;

--
Kind regards,

Sakari Ailus

#55 lore raw
In reply to Hans de Goede
Quoted 30 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 5 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead

But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.

Quoted 8 lines

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Thanks,

So the drivers of sensors connected to IVSC have to add power up/down code.

BR,
Wentong

Quoted 309 lines

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 83 lines
Quoted 83 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 50 lines · Wentong

Hi

Quoted 48 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 41 lines · Wentong
Quoted 41 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 32 lines · Wentong
Quoted 32 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 23 lines · Wentong
Quoted 23 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 14 lines · Sakari Ailus

Hi Wentong,

Quoted 11 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiple

sensors.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up system

Since the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces provided

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

by this IVSC driver.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The Intel

Management Engine) commands.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

These two hardware modules have two different MEI UUIDs to
enumerate. These hardware

modules are:

Quoted 4 lines
Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownership

of camera sensor.

Quoted 3 lines
Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can be

notified.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the
following:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC before accessing

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

sensor via I²C. The IVSC ownership control needs to be in the

right

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

setting for this to work, and device links can be used for
that

purpose

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 1 line

the supplier devices will be PM runtime resumed before the

consumer

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on
host side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks
to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSC

enabled.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't

work.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 6 lines
Quoted 6 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct

mei_device).

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 4 lines
Quoted 4 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low

or active high.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted

upstream) :

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 8 lines
Quoted 8 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#56 lore raw

Hi,

Quoted 39 lines · Wentong
Quoted 30 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM

<re-added the previous Cc list, which I dropped because of the large attachments>

Hi Wentong,

Quoted 5 lines · Wentong

Hi Hans,

Thanks

And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the

platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.

Quoted 1 line

And I will double confirm with people who know dell product well tomorrow.

Ah, I was under the impression that there was an IVSC there because:

1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead

But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.

Quoted 8 lines

If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.

Quoted 3 lines

Thanks,

So the drivers of sensors connected to IVSC have to add power up/down code.

If we go the model it as a powerdown GPIO route, yes. Note most sensor drivers already have support for this since this is used in the INT3472 case already.

Regards,

Hans

Quoted 313 lines

BR,
Wentong

Quoted 309 lines

Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?

This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.

Regards,

Hans

Quoted 83 lines
Quoted 83 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

Hi Wentong,

Attached are the requested dmesg + acpidump for the Dell Latitude 9420.

Regards,

Hans

Quoted 50 lines · Wentong

Hi

Quoted 48 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PM

Hi,

Quoted 41 lines · Wentong
Quoted 41 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PM

Hi,

Quoted 32 lines · Wentong
Quoted 32 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PM

Hi Wentong,

Quoted 23 lines · Wentong
Quoted 23 lines

-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PM

Hi,

Quoted 14 lines · Sakari Ailus

Hi Wentong,

On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiple

sensors.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up system

Since the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces provided

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

by this IVSC driver.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The Intel

Management Engine) commands.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

These two hardware modules have two different MEI UUIDs to
enumerate. These hardware

modules are:

Quoted 4 lines

- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownership

of camera sensor.

Quoted 3 lines

- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver and

application.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can be

notified.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can
suggest the
following:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- The IVSC sub-device implements a control for privacy

(V4L2_CID_PRIVACY

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

is a good fit).

- Camera sensor access needs to be requested from IVSC
before accessing

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

sensor via I²C. The IVSC ownership control needs to be in the

right

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

setting for this to work, and device links can be used for
that

purpose

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(see device_link_add()). With DL_FLAG_PM_RUNTIME and

DL_FLAG_RPM_ACTIVE,

Quoted 1 line

the supplier devices will be PM runtime resumed before the

consumer

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

(camera sensor). As these devices are purely virtual on
host side and

has

Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

no power state as such, you can use runtime PM callbacks
to transfer

the

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

ownership.

Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.

I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c

transfers are done.

Quoted 6 lines
Quoted 6 lines

And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +

disavle control.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.

Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling

way.

Quoted 4 lines
Quoted 3 lines

What do you mean? The supplier devices are resumed before the driver's probe is called.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.

Then maybe we need to do the device_link_add somewhere else.

sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSC

enabled.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't

work.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?

Could you please share your kernel log and DSDT? Thanks

BR,
Wentong

Quoted 6 lines
Quoted 6 lines

struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32
flags)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct

mei_device).

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.

Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace

to show up).

Quoted 4 lines
Quoted 4 lines

But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting

registered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.

As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low

or active high.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().

Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line to probe() like this:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);

To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted

upstream) :

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines

https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d

To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.

Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.

Regards,

Hans

Quoted 10 lines
Quoted 10 lines

The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.

This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.

So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.

Quoted 8 lines
Quoted 8 lines

So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.

Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls

acpi_dev_clear_dependencies().

Quoted 9 lines
Quoted 8 lines

The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.

Regards,

Hans

#57 lore raw
In reply to Sakari Ailus
Quoted 46 lines

-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 1, 2023 6:34 PM

Hi Wentong,

Quoted 22 lines · Wentong Wu

Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.

The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system

Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.

Quoted 4 lines

Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware

modules are:

Quoted 3 lines

- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of

camera sensor.

Quoted 8 lines

- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.

IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.

Quoted 6 lines
Quoted 3 lines

In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.

I thought this for a while and did some research, and I can suggest the following:

Thanks a lot

Quoted 3 lines

- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).

Agree, thanks

Quoted 11 lines

- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and
DL_FLAG_RPM_ACTIVE,
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.

That's pretty cool, and the device link will be setup during mei_ace's probe with device_link_add, but we should guarantee there is no i2c transfer during sensor driver's probe. And most of upstream sensor drivers don't initial i2c transfer during probe, but I have no idea what will be like for our upstreaming sensor drivers.

Quoted 2 lines

- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).

IPU driver should make sure start sensor streaming first and then IVSC sub-dev, which will match the command downloading sequence required by firmware.

Quoted 3 lines

- Device links need to be set up via IPU bridge which now is called CIO2 bridge (cio2-bridge.c).

Reviewing the code

Quoted 7 lines

Any questions, comments?

--
Kind regards,

Sakari Ailus